Controllable Generation: Decomposing “what it looks like” into multi-channel conditions of semantics, structure, identity, and constraints

Saying that a generative model is “controllable” is not yet an executable specification, because “controllable” does not specify which degree of freedom of the output is being controlled. The prompt “a person in red on a bridge” cannot fix the person's position and posture precisely because text can only provide discrete semantic constraints: the phrase “on a bridge” only rules out situations that are not on a bridge, but still allows the person to stand on any side of the bridge, face any direction, take any pose, and also allows the bridge itself to have different angles, lengths, and lighting. The information content of text conditions is concentrated in semantics of the “what is it” type, and is very weak in constraining continuous degrees of freedom such as spatial position, orientation, and posture.

To put the control objective into practice, one must first select the degree of freedom to be constrained and choose a condition type that carries the corresponding information. Different conditions differ greatly in the amount of information and uncertainty across different degrees of freedom: an edge map locks the object's contour, that is, the shape boundary; a depth map locks the relative distance from a point to the camera, thereby constraining front-back occlusion and spatial hierarchy; a pose skeleton locks joint positions, directly determining the person's posture; a segmentation map locks region categories, specifying which pixel belongs to which class of object; a reference image may simultaneously carry multiple types of information such as identity, style, and background, and this information is often not explicitly declared. Therefore, the same natural language sentence, when replaced with different condition representations, can constrain completely different degrees of freedom.

From this we can derive an actionable principle: the control objective should be written as measurable constraints, not a vague “more like”. Specifically, it should be possible to answer: whether the target object appears; what the error of keypoints is; and whether the model is allowed to vary freely in areas that have not been explicitly specified. Only when these constraints can be checked and quantified does “controllable” change from a subjective impression into an engineering objective that can be verified and compared. Conversely, if one stops at the expression “more like the reference image”, it is impossible to judge whether the model has met the requirement, nor to locate in which degree of freedom the control failure occurred.

Conditional generative models learn the conditional distribution p(x|c), that is, the distribution of samples x given condition c. Adding conditions narrows the range of candidate outputs, but learning p(x|c) itself does not guarantee that every condition c can be satisfied. An intuitive example is: the training distribution has never contained 'a four-legged chair suspended underwater'; merely feeding this as a control condition at inference time does not enable the model to generate an image out of thin air that both conforms to physical common sense and satisfies all requirements.

The reason is that no matter what conditions the model receives, its output is still constrained by two boundaries: first, the support range of the training data for the conditions, and second, the generative prior learned by the model from the data. Conditions may contradict each other, for example requiring the same object to appear simultaneously in two disjoint locations; control maps may also carry geometric structures absent from the training data, such as angles beyond the normal range of a pose skeleton, or outlines inconsistent with an object's physical structure. When the conditions themselves are infeasible, forcing compliance only produces artifacts such as limb misalignment, texture tearing, or unnatural compositions with floating objects; the model also often chooses to ignore some conditions, prioritizing outputs that look natural. Both paths are not 'strict compliance like a renderer', but a compromise between compliance and naturalness.

The engineering conclusion this leads to is: a controllable generation system cannot assume by default that every input condition is satisfiable. The system needs to have the ability to detect infeasible or mutually conflicting conditions, and after detection, allow three ways of handling them—ask the user for clarification, automatically relax certain constraints, or explicitly declare failure—rather than promising to strictly render any given control map. Only by understanding conditional generation as 'constraints on the candidate distribution' rather than 'pixel-by-pixel instructions for the output' can we truly anticipate the model's behavior when conditions conflict with priors.

Classifier-free guidance is a method for strengthening the influence of conditioning at sampling time. It does not train an additional discriminator; instead, it directly combines two predictions from the same model: the unconditional prediction ε_u and the conditional prediction ε_c. The core idea is that the difference vector between the conditional and unconditional predictions points in the direction where the condition makes the output more consistent with c. By amplifying this difference, the sample can be pushed more strongly toward the conditioning direction during sampling.

The combination formula is ε_guided = ε_u + w×(ε_c − ε_u), where w is the guidance strength. Taking an unconditional noise prediction of 0.8, a conditional prediction of 0.2, and a guidance strength of 3 as an example, substituting gives ε_g = 0.8 + 3×(0.2 − 0.8) = −1.0. This result is more extreme than both original predictions, indicating that the combined value has been extrapolated in the direction away from unconditional.

Different values of w correspond to different behavioral boundaries. When w = 1, the combined result degenerates exactly to the conditional prediction ε_c; at this point the model is ordinary conditional generation. When w > 1, the formula extrapolates in the conditioning direction, and the output's adherence to the condition increases, but at the cost of possible color oversaturation, repetition of local details, and an overall decrease in diversity. The larger w is, the more samples concentrate in the narrow region of high probability of satisfying the condition, deviating from the natural diversity of the data distribution.

It should be noted that ε in the formula can represent different quantities under different parameterizations: under noise-prediction parameterization it represents the predicted noise, while under other parameterizations it may represent velocity, etc. Regardless of the specific object, the concept is the same: use w to amplify the difference between the conditional and unconditional predictions, thereby strengthening the conditioning constraint during sampling. The prerequisite for this mechanism to apply is that the model can produce both conditional and unconditional predictions, so during training it is usually necessary to randomly drop the condition with a certain probability so that the same network learns both behaviors.

Consider a complete pipeline that uses pose, depth, and reference clothing as three conditions to generate a poster. It concretely shows how multi-condition control should be decomposed and debugged in practice. The text part of the task is only responsible for defining semantic content such as the person, poster style, and background, and no longer describes the pose—because the pose is already explicitly given by the pose skeleton; the pose skeleton provides joint keypoints, the depth map provides front-back relationships between objects, and the reference image is limited to providing only clothing texture without transferring the person's identity. In this way, each condition channel has its own role, avoiding vague overlap of information across multiple channels.

When the three controls are given simultaneously and the result fails, to locate which path the failure comes from, the correct order is not to directly tune parameters, but to first verify the compatibility of the conditions, and then add them layer by layer. The first step is to perform single-control generation one by one: generate once using only pose, only depth, and only clothing, establishing for each channel a judgment of 'whether it can work independently' and its own independent metrics. If any single channel cannot meet the standard, it indicates a problem in the injection method or strength of that channel itself; if all single channels are normal, the problem may only appear in multi-condition interaction.

The second step is gradual combination from two-control to three-control. At this stage, focus on checking whether different control maps contradict each other, for example whether the arm position required by the pose skeleton conflicts with the occlusion relationship required by the depth map. If a conflict is found, you need to set explicit priorities for the conditions and use local masks to restrict different conditions to their respective effective regions, rather than letting them compete with each other across the entire image.

The third step is to scan along the strength dimension: start from a lower control strength and gradually increase it, record how the three curves—keypoint error, image quality, and diversity—change with strength, and find the interval where the three are balanced, rather than pulling to maximum strength all at once. At the same time, also check whether information from the reference image has been accidentally leaked—for example, whether the background or face of the reference image has been unconsciously copied into the result.

Throughout the entire process, fixing the random seed for ablation is a key means of distinguishing 'control contribution' from 'random sampling variation': only when the seed is fixed can changes in the output be attributed to changes in the control conditions, rather than sampling noise. Verifying condition compatibility before tuning parameters for multiple controls can avoid misjudging condition conflicts as insufficient model capability.

The information carried by different conditions operates at different scales in the sampling network, which is an easily overlooked structural fact in multi-condition control. Text conditions carry object-level semantics, pose and depth carry geometry and spatial structure, and reference appearance carries texture and color. If all these control signals are simply stitched into a single image and fed to the base generator, information at different granularities will be forcibly compressed into the same pathway and contaminate each other: texture-level conditions may interfere with geometric structure, and geometric constraints may distort semantic content.

The correct approach is to independently encode control signals of different granularities and inject them into the base generator with gating according to scale. That is, each condition first passes through its own encoder to obtain a representation suited to its information granularity, and is then selectively injected at different layers of the sampling network: pose and depth geometric information shapes spatial structure in shallower, pixel-adjacent layers, text object-level semantics handles global understanding in the middle layers, and reference appearance texture and color takes effect in deeper layers. In this way, the relationship expressed in Figure 1 can be summarized in one sentence: control signals with different granularities are best encoded independently and gated by scale.

The direct benefit of this structure is independence among the channels. When each condition exerts influence at the scale it is best at, and the injection strength is determined by gating, adjusting one condition is less likely to accidentally alter constraints already satisfied by another; this also provides the implementation foundation for the layer-by-layer troubleshooting of single-control, dual-control, and triple-control mentioned earlier.

It is not cost-effective to directly retrain an entire large model on edge maps: full retraining is expensive, and it is easy for the model to forget its original generative capabilities while adapting to new conditions. The idea of the ControlNet-style branch is to keep the weights of the base model, build a parallel branch to learn spatial control signals, and add what the branch learns as residuals back to the backbone's intermediate features.

The specific approach is: copy or bypass part of the base network's modules as the control branch. The branch's input consists of spatial conditions such as edge maps and depth maps. The connections between the branch and the backbone are initialized to zero, meaning that initially these connections have zero weights, so that the contribution of the control features to the backbone is strictly zero at the start of training. Thus at the initial moment the entire network's behavior is approximately equivalent to the original model; as training progresses, the branch gradually learns to use edge or depth information to inject non-zero spatial residuals into the backbone. Freezing the backbone and updating only the branch reduces data requirements and lowers the risk of catastrophic forgetting.

Zero initialization makes the training starting point safe, but it does not guarantee that the training process is always safe. The control branch itself can still overfit to edge or depth maps of a specific style, can leak information from the training data into its parameters, or can be incompatible with future new versions of the backbone network. Therefore the control branch needs to be tied to a specific backbone version, and regression validation should be performed for version upgrades; otherwise a ControlNet-style branch trained for an old backbone may fail or produce unexpected outputs after being switched to a new backbone.

Control strength is not always better when higher; there is a three-way trade-off among adherence, quality, and diversity, and strengthening one usually sacrifices the other two. Reducing keypoint error to zero is a typical example: strong spatial control forces the model to obey the pose skeleton pixel by pixel, and the model loses the freedom to correct unnatural skeletons, so the generated person, although keypoints match exactly, may have stiff posture, distorted joints, and strange muscle orientations. Similarly, overly strong text guidance compresses the sample distribution into a high-probability narrow region, causing outputs to lose diversity while matching the description; overly strong reference weights copy the reference image's background and even identity together.

The root of this trade-off is that the control condition provides constraints on the output, while the generative prior provides the ability to make outputs natural and diverse; the two compete over the same batch of intermediate features. Reducing control strength is equivalent to returning more degrees of freedom to the prior; outputs recover naturalness and diversity, but may also deviate from user constraints. Therefore there is no single strength value optimal for all tasks; one can only scan along the strength axis, draw a multi-objective curve, observe how keypoint error, image quality, and diversity change with strength, and then choose the lowest strength that just satisfies the hard constraints—ensuring necessary control while preserving as much of the model's prior freedom as possible.

If a hard geometric constraint must be exact with no deviation, then generative models may not be the right tool; traditional rendering or compositing pipelines are often more reliable in such scenarios. The value of Controllable Generation lies in scenarios where soft constraints and diversity coexist, rather than replacing precise geometric solving.

When multiple conditions act simultaneously and conflict with each other, the model cannot pretend to solve the problem by random compromise. For example, with “the pose requires the hand to be above the head, but depth places the hand behind the body”, if the model averages between two contradictory constraints, the result is often that the hand is neither truly above the head nor truly behind, becoming a mushy structure that fits neither the pose nor the depth. The correct processing order is: first check, then declare, then isolate or report.

The first step is to perform a geometric consistency check in condition space, that is, before injecting conditions into the generation network, compare whether spatial conditions such as skeleton, depth, and segmentation are mutually consistent. This step happens at the condition level rather than the pixel level, so it is low-cost and can intercept most conflicts in advance. The second step is to explicitly declare for each condition its region of effect, effective time, and priority: the region of effect specifies in which part of the image the condition takes effect, and priority specifies whose constraint is harder when a conflict occurs. The third step is to handle residual conflicts—for local conditions, masks can be used for isolation, letting different conditions each manage their own region and avoiding competition over the whole image; if the conflict is too severe to be reconciled with masks, it should be reported to the user rather than silently outputting a compromise result that satisfies neither side.

Condition combinations never seen during training can also cause control entanglement: each condition works when viewed alone, but when combined, the channels interfere with each other on shared intermediate features, causing one side's control to fail. To locate this entanglement, it is necessary to use a stepped evaluation of single-control, pairwise, and multi-control comparisons layer by layer, and always keep an uncontrolled baseline as a reference, in order to judge whether the output deviation comes from the control itself or from new interference introduced by the multi-condition combination.

Reference images are the most likely type of control signal to cross boundaries, because a single image carries far more information than the user declares. The user only wants to copy the color scheme, but the output ends up similar in face and background too. The root cause is that the image encoder entangles multiple attributes—color, texture, identity, composition, background—in the same vector, so the model cannot automatically distinguish “only take the color scheme” from “copy everything wholesale”.

To suppress this kind of boundary crossing, you need to set up safeguards at both the condition injection side and the output side. On the injection side, methods include: selecting only specific regions or specific feature layers of the reference image as conditions, using identity-specific or style-specific encoders to separate the desired attributes from other attributes, using random augmentations to weaken over-reliance on details, and applying negative constraints to explicitly exclude certain attributes. On the output side, you need to perform similarity checks, specifically including nearest-neighbor retrieval comparison with the data source, face similarity checks, background similarity checks, and watermark residue checks, to confirm that the generated result does not unintentionally copy the identity or protected content of the reference image.

From a product perspective, the interface must make clear what is being referenced—pose, layout, color scheme, material, or identity—and record the corresponding authorization, rather than treating “reference style” by default as permission to copy everything in the reference image. Boundary declarations and post-hoc checks for reference image conditioning are a necessary part of copyright and identity compliance in Controllable Generation, not an optional add-on.

The performance ceiling of a control model is not determined solely by network architecture, but is limited by the quality of control maps in the training data. In practice, control maps are rarely manually annotated pixel by pixel; they are mostly generated automatically by pre-trained detectors and are pseudo-labels. These pseudo-labels carry systematic errors: the choice of threshold in edge detection determines the thickness and incompleteness of contours, pose estimation misses occluded joints, and monocular depth estimation has scale inconsistency. When these biased control maps are used as training signals, the control model learns not the true geometry but the artifacts of the extractor itself.

This implies a neglected risk: the model may accurately learn “the edges or depths that the detector considers correct,” rather than “the true edges or depths of the world.” Once the upstream extractor is replaced or upgraded, its output distribution shifts, and the previously trained control model may behave abnormally under the new extractor. Therefore, during training, the generator, the control extractor, and their versions must be saved together; at sampling time, align them to ensure that each image and its control map come from the same extractor version. In terms of data coverage, also supplement rare poses, occlusions, and complex layouts to avoid the model being effective only on common samples. When replacing the extractor, you cannot directly reuse the old model; you must re-evaluate distribution compatibility and decide whether retraining is needed.

The quality of control maps determines the ceiling of controllability: no matter how well the network is designed, if the training signal itself is disconnected from the true geometry, the most the model can learn is only the regularities expressed by that biased label, not the degrees of freedom the user actually wants to control.

Controllable generation cannot be evaluated with a single text similarity score, because a text score reflects semantic conformity, which is not equivalent to the degree to which spatial control is achieved. A high CLIP text score only indicates that the generated image matches the text description at the object and attribute level; it does not prove that spatial constraints such as pose and depth are satisfied. The correct approach is to configure a dedicated metric for each type of control condition: text conditions use object and attribute matching combined with human evaluation, pose conditions use keypoint error, depth conditions use scale-aligned correlation, segmentation conditions use intersection over union (IoU), edge conditions use structural distance, and identity conditions use similarity within an authorized range. Image quality, realism, and diversity are measured as independent dimensions and are not mixed with control metrics.

In addition to testing whether the target constraints are satisfied, it is also necessary to specially check whether “non-target regions” have been inadvertently affected, that is, whether unspecified degrees of freedom have been accidentally fixed. For example, when only pose is controlled, does the background also become abnormally fixed and lose its expected variation; when only the color scheme is changed, does the person’s identity drift? If only target constraints are measured, the model can completely improve its control score by the lazy approach of “locking all unspecified regions as well,” and this side effect is only exposed when unspecified regions are evaluated separately. Therefore, control evaluation must simultaneously cover two directions: “whether what should be controlled is controlled” and “whether what should not be controlled remains free”; otherwise it only rewards side effects.

The control branch cannot be deployed independently of the base generator it depends on, because there is substantial implicit coupling between the two. After the base model is upgraded, an existing ControlNet-style branch may not carry over directly: the control branch depends on the intermediate feature scales of the backbone, the latent space representation of the VAE, and the output of the text encoder. Upgrading any one of these components may change these interfaces, causing the old branch to fail silently—the output no longer matches the control map, but no error is raised because the network can still generate images normally.

Therefore, when deploying, the control model, the extractor, and the base generator should be managed as an indivisible version unit. Specifically, lock down a combination manifest that clearly records the backbone version, the control branch version, the text encoder version, the VAE version, and the control extractor version. After any component changes, revalidate under multiple control strengths, multiple resolutions, and multiple condition combinations rather than running only once under the default configuration. The input control map should be validated before entering the network, including size, value range, coordinate system, and privacy information, to avoid silent bias or compliance problems caused by invalid input.

At runtime, you should also set resource limits and failure fallback mechanisms to prevent abnormal input from overwhelming the service. For reproducibility, the prompt, control map, mask, strength weights, random seed, and full model combination involved in a generation should all be saved, so that any output can be reconstructed or investigated after the fact. Version pinning and complete records are the key to moving controllable generation from "can run" to "reproducible, regression-testable, and auditable."

Stringing together the previous components, we can see a complete causal chain from user intent to verifiable practice.

The starting point is to decompose user intent into mutually non-overlapping degrees of freedom: semantics, geometry, identity, and style, clarifying which dimension the user actually wants to control. Next, select the most appropriate condition representation for each degree of freedom—text carries semantics, pose skeleton and depth carry geometry, and reference images carry identity and style. Conditions of different granularities are encoded independently and injected into different layers of the base generator according to scale, rather than squeezed into the same path and contaminating each other.

Before actual generation, first check whether multiple conditions are feasible and whether they conflict, and declare priorities and regions of effect; when conflicts cannot be reconciled, report to the user rather than making a random compromise. During generation, adjust the control strength to the minimum value that just satisfies the hard constraints, avoiding excessive sacrifice among adherence, quality, and diversity. After completion, use ablation to confirm the true contribution of each control path channel by channel, and at the same time check whether the reference image leaks undeclared identity or background information.

In the evaluation stage, measure adherence, quality, diversity, and preservation of unspecified regions separately, to prevent measuring only the target constraints and rewarding the side effect of “fixing what should not be fixed.” Finally, lock the model, extractor, and base generator as a version unit, and save all source information for each generation to ensure reproducibility.

This chain must be able to withstand verification; it requires clarifying what each layer fixes and what evidence to observe in “Controllable Generation: decomposing ‘what it looks like’ into multi-channel conditions of semantics, structure, identity, and constraints”. The input layer fixes the same batch of samples, the same preprocessing pipeline, and permission boundaries, recording input hashes, slice labels, and rejection reasons; the mechanism layer changes only one core variable while locking the rest of the configuration, observing key intermediate states and the position where the first deviation from expectations occurs; the output layer uses the same acceptance rules and resource budget, comparing stratified differences in quality, cost, latency, and failure rate; the falsification layer retains a control group that does not enable the target mechanism, confirming whether the benefit is stably reproduced across samples and random seeds. When all four layers—input, mechanism, output, and falsification—align, controllable generation is no longer just a coincidence of “looking like,” but a causal path that can be located, regressed, and reproduced.