Diffusion Models

The core problem diffusion models aim to solve is to let AI create an image out of thin air—a picture that "no one has ever taken, yet is plausibly realistic." Here "out of thin air" is the key phrase: the model receives not an existing image to modify, but a training image distribution and a generation condition; what it must produce is a new sample from that same distribution that has never been seen before, and this new image must withstand human intuitive scrutiny.

Where does the difficulty lie? It lies in the fact that "getting it right in one step" is nearly impossible. One image has several million pixels, and these pixels cannot act independently—a corgi's two eyes must be symmetrical, the hat must be worn on the head, the light and shadow must fall on the same side, and the coat color must match the background. The value of any one pixel does not matter; what is truly difficult is making several million pixels coordinate with each other while satisfying all the constraints simultaneously. Asking the model to output so many coordinated pixels in one go is equivalent to requiring it to compute all the joint relationships across the entire image correctly within one step, which is an extremely heavy burden for any generator.

The cleverness of diffusion models is that they do not do this. They do not try to "generate the entire image in one step"; instead, they break generation into a series of small steps, each step handling only a small part of the change, allowing coordination among pixels to be established gradually. The value of this idea becomes fully apparent only after understanding the noise addition and denoising in later sections, but its motivation is already clear here: to avoid the infeasibility of getting it right in one step.

At the same time, it is important to be clear about the boundaries of this matter. The output of a generative model only indicates that "the relationships among pixels are as plausible as in the training data"; it does not prove that the facts, text, or physical processes in the image are true. When a task requires exact reproduction, verifiable measurement, or deterministic answers, a generative model is not a reliable database and should not be used as a source of facts.

Diffusion models turn the hard problem of "generating from nothing" into something learnable by relying on two opposite processes: adding noise and denoising.

The forward process is adding noise. Take a clean real image and sprinkle noise onto it step by step, many steps, until it becomes a patch of pure noise. This process itself is extremely simple, but it also solves a key problem—where training samples come from. Each time noise is added, the "before adding noise" and "after adding noise" are a ready-made question and answer pair: the question is the image that already has a bit of noise, and the answer is its clearer appearance from the previous step. The whole process automatically generates a huge number of such pairs, without manual annotation. This is exactly the idea of Self-supervised Learning (see "Self-supervised Learning" for related content).

The reverse process is denoising. Train a Neural Network to learn to reverse this process: give it a noisy image and have it predict which noise to remove, restoring it to a slightly clearer image. What the network learns is not a complete image generator, but a "denoise one step" operation: input a noisy sample, output a denoising direction.

Once the network learns to denoise, generation follows naturally. Start from a blob of random noise—there is actually nothing in this noise—and simply have the model repeatedly perform denoising. Each time, the image becomes a little clearer, and step by step a completely new image is "developed". This image never appeared in the training data, but each denoising step it goes through conforms to the image structure learned during training.

Here each step has clear input and output boundaries. The forward process takes clean samples and random noise, and outputs training pairs at any noise timestep; the reverse network takes a noisy sample, the current timestep, and optional conditions, and outputs a denoising direction. The output of a single step should be interpreted as "an estimate of the next update", not as the complete final image. Also note its limitations: if the noise schedule is unreasonable, or the training data coverage is insufficient, even if each step is done correctly, accumulating multiple steps can still produce a failed result.

Since the complete noising chain has so many steps, during training do we need to go from start to finish each time? No. The key is: given a clean sample x₀, we can directly sample a noisy sample at any time t without adding noise step by step. The specific formula is that the noisy sample xₜ equals the square root of ᾱₜ times the clean sample x₀, plus the square root of 1 − ᾱₜ times random noise ε. Here ᾱₜ is determined by the noise schedule, which describes how much signal remains and how much noise there is at time t. During training, the network εθ(xₜ, t, c) is usually trained to predict "how much noise was originally added" and minimize ||ε − εθ||²; the condition c can be text. Because noisy samples at any time can be constructed directly, one randomly sampled t can provide an unbiased training signal for the entire timeline, without needing to traverse the whole chain.

Let's make this concrete with a one-dimensional numerical example. Take a clean value x₀ = 2, ᾱₜ = 0.64; then the square root of ᾱₜ is 0.8, and the square root of 1 − ᾱₜ is 0.6. If this time the random noise ε = −1, the noisy value is xₜ = 0.8 × 2 + 0.6 × (−1) = 1.0. Suppose the network predicts εθ = −0.8; then the noise loss on this sample is (−1 + 0.8)² = 0.04. From this, the inferred clean value is approximately (1.0 − 0.6 × (−0.8)) / 0.8 = 1.85. One prediction does not perfectly recover 2, which is exactly the point of multi-step sampling: each step's prediction is only an estimate of the reverse direction, and multi-step sampling will progressively correct along the model-estimated reverse direction, eventually approaching a plausible image.

Here we need to clarify an intuition that can easily become fixed. "Removing a little noise" is only an intuitive description of the training objective, not the only parameterization. The model can also predict the clean sample itself, predict a velocity variable, or predict the score of the data distribution; different samplers can also approximate the reverse trajectory with fewer steps. The core is learning "the direction of the reverse update", not having to erase a fixed noise point pixel by pixel.

The boundary should also be made clear. The training inputs are a clean sample x₀, a random time t, and noise ε; the output is the network's prediction of the noise or an equivalent target. The formula is efficient because it directly constructs any time step, so there is no need to add noise step by step. However, a smaller loss only means that the model is closer to this particular known noise; it alone cannot guarantee that the final sample is on-topic, realistic, or diverse — that depends on the conditioning and guidance discussed in later chapters.

Since the goal is to get an image from noise, why not train a model to do it "in one step" instead of going through dozens of steps? The reason is the same as the "one-step" problem discussed in Section 1: jumping from pure noise to a perfect image in one step is equivalent to requiring the model to satisfy the coordination relationships among all pixels across the entire image in a single prediction—a burden so heavy that it is impractical.

After breaking it into many small steps, the task of each step becomes much simpler. The model only needs to restore a "more noisy image" to a "less noisy image"—a small job the model can learn. The input of each small step is the current noisy state, and the output is the next state slightly closer to the data distribution; the sampler keeps repeating this transformation. Many simple small steps accumulate to complete the originally impossible big jump. This is the core causal chain of "step splitting": breaking the unlearnable large goal into a series of learnable small goals, and then stacking the small goals one by one.

This idea is often compared to Chain of Thought (CoT), and both can indeed be understood through "breaking a big jump into small steps". But the boundary of this analogy must be made clear: diffusion is probabilistic modeling with an explicit forward stochastic process and a reverse generative model, while CoT is a discrete token trajectory. The analogy only explains the "step-by-step" intuition and cannot treat the two sets of mathematical mechanisms as the same.

The number of steps also involves trade-offs. Increasing the number of steps usually reduces discretization error and makes the reverse trajectory closer to the continuous process, but it also increases latency—each additional step means one more forward pass through the network. More importantly, if the model learns the wrong update direction, then no matter how many more steps are taken, it will not automatically correct errors in knowledge and structure; it will only execute the wrong direction more times.

The preceding denoising can only generate "a plausible image" but cannot decide which one to generate. To make the model draw the image the user wants, "a corgi wearing a hat," the approach is to inject text as a condition into every denoising step—so that the model is not just "denoising" but "denoising in a direction that matches this description."

This depends on alignment between text and image. The model must know roughly what the phrase "a corgi wearing a hat" looks like in image space: which part is the hat, which part is the corgi, and how they should be combined. This alignment is exactly what CLIP-style image-text Contrastive Learning provides (see "CLIP," "Multimodal Models," and "Embedding" for related content). The text encoder first turns the prompt into a conditional representation, and the denoising network reads this representation at each step and outputs an update direction influenced by the condition.

Thus the division of roles becomes clear: denoising decides "how to draw a plausible image," and the text condition decides "which image to draw." Together, they form text-to-image generation.

It is also important to understand the correct way to interpret generation results. Generation results should be understood as the model's joint sampling of the condition and the training distribution, rather than verbatim execution of commands. For common combinations well covered by training, the condition can effectively steer generation toward the target content; however, rare combinations, precise counts, complex spatial relationships, and text layout can still go out of control—because the model learns associations at the distribution level, not item-by-item obedience to instructions.

Before diffusion models, the main force in image generation was GAN. The two have a fundamental difference in generation method: GAN makes the generator and discriminator compete against each other, and the generator outputs an image in one step; diffusion models start from noise and generate step by step through multi-step denoising.

This difference directly brings their respective advantages and disadvantages. GAN generates quickly because one forward pass produces an image; but training is unstable and prone to falling into "mode collapse"—that is, the generator, in order to fool the discriminator, learns to generate only a few types of samples, losing diversity. Diffusion models are more stable, have more diverse samples, and are also more easily controlled by conditions such as text, but at the cost of requiring multiple steps and slower generation.

The reason diffusion later took the lead is here: it breaks generation into many learnable small steps, making training more stable, samples more diverse, and also easier to guide with conditions. The speed shortcoming is greatly alleviated by "latent space diffusion"—performing denoising in a compressed small space first, rather than directly in the original pixel space (see "Variational Autoencoder (VAE)" for related content). For the detailed mechanism of GAN, see the "Generative Adversarial Network (GAN)" node.

When comparing the two, maintain fairness: inputs should be the same data, resolution, compute, and evaluation slices, so that differences in output speed, quality, and coverage are interpretable. Also note two easily overlooked boundaries: GAN being fast in one forward pass does not mean training is cheap—the cost of adversarial training can be high; diffusion training being stable does not mean it is better on all tasks. Which one to choose depends on latency requirements, editing capabilities, diversity needs, and deployment budget.

High-resolution pixel space is too expensive, and text conditioning may not be strong enough, so engineering uses two approaches to address this.

The first is latent space diffusion. First, an encoder compresses the image into smaller latent variables; denoising is performed in this smaller space, and finally the result is decoded back to pixels, significantly reducing computational cost. The input to latent space diffusion is the compressed representation, and the output is the decoded image; what compression discards is detail information, which is the price paid for its speedup.

The second is classifier-free guidance. It combines conditional prediction with unconditional prediction to alter the sampling direction: making the sampled result lean more toward the side that "matches the text condition." Increasing the guidance scale usually makes the image adhere more closely to the prompt, but too high a value sacrifices diversity, causes oversaturation, or produces artifacts. This shows that "being more obedient" is not a free one-way gain, but a trade-off—a higher guidance score indicates stronger adherence to the conditional difference, not necessarily higher overall quality.

Since both knobs involve trade-offs, systematic evaluation and failure diagnosis are needed. The approach is to fix a prompt set and random seeds, run grid experiments over sampling steps, scheduler, and guidance scale, and record latency, prompt adherence, sample diversity, and artifact rate. FID or image-text similarity can only serve as proxy metrics; you also need to specifically examine hard slices such as counting, text, hands, and identity consistency. When diagnosing, first determine which category the failure belongs to: if high guidance only improves similarity but makes samples converge, it should be attributed to the adherence–diversity trade-off; if structures are generally blurry, then check the denoiser, VAE decoding, and sampling steps separately. This distinguishes "parameter tuning issues" from "model capability issues," avoiding blindly increasing any single parameter.

Connect the preceding sections into a complete causal chain, starting with "one-step generation is too hard": creating a coordinated, realistic image out of thin air in one step is equivalent to predicting in one go the joint relationships among all pixels of the entire image—a burden so heavy as to be impractical (§1).

Diffusion's response is to split it apart: forward noising, taking a clean real image and gradually adding noise step by step until it becomes pure noise, automatically generating a vast number of "before noising / after noising" training samples along the way; reverse learning to denoise, training the network to restore a noisy image to a slightly cleaner one (§2).

During generation, start from pure noise and repeatedly denoise, letting the image gradually "develop"; the reason for breaking it into many small steps rather than completing it in one step is that each small step—"from a bit more noise to a bit less noise"—is simple and learnable, and many small steps accumulate to accomplish a large jump that was originally impossible. Moreover, training does not need to walk the entire chain step by step; you can directly construct noisy samples at any arbitrary time, and randomly sampling one time can provide training signal for the entire timeline (§2, §3).

Next, text is injected into denoising as a condition, and through image-text alignment the model is steered to generate in a direction that matches the description; thus "denoising decides how to draw a plausible image, and the condition decides which one to draw" (§4).

Compared with GAN, diffusion is more stable, more diverse, and more controllable, so it has become mainstream, at the cost of slower generation (§5); this speed weakness is in turn alleviated by latent-space diffusion, while the guidance scale makes a trade-off between adherence to the prompt and diversity (§6, §7).

The sign of grasping the core is: being able to explain clearly why diffusion splits generation into many small steps of "noising—denoising" and how text directs the denoising direction.

Understanding diffusion models first requires several foundations: neural networks provide the skeleton of the reverse denoising network, self-supervised learning explains why the noising process can automatically produce training samples, and embeddings are the language through which text conditions enter the model (prerequisites).

The core concepts on this page are noising–denoising, multi-step generation, text conditioning injection, and the comparison with GAN. These are the pivot points for understanding the entire mechanism.

Immediately related extension concepts include: Image Generation as the broader task context, CLIP and Multimodal Models (providing alignment between text and images), Generative Adversarial Network (GAN) (the main image generation approach before diffusion, used for comparative understanding), and VAE (the compression and decoding mechanism that latent-space diffusion relies on).

Further directions extend from "generating a single image" to Controllable Generation, Video Generation, Image Editing, and Super-resolution—they are all built on the same foundation of noising–denoising and conditional guidance, but expand the generation target from a single static image to more complex spaces.