Atlas
kb://library/diffusion-models2026-06-16

Diffusion Models

generative-modelsdiffusionddpmddimscore-matchinglatent-diffusionstable-diffusionflow-matchingconsistency-modelsclassifier-free-guidanceimage-generation

Diffusion Models

What it is (intuition first)

Start with a clean image. Add a tiny bit of random noise. Add a little more. Keep going for hundreds of steps until the image is pure static — indistinguishable from TV snow. That destruction is easy and requires no learning at all.

A diffusion model learns to run that movie backwards. It is a neural network trained to look at a noisy image and predict "what noise was added," so it can subtract a little of it and reveal a slightly cleaner image. Apply that denoiser over and over, starting from pure static, and a coherent image emerges from nothing. That is the whole trick: creating noise from data is easy; creating data from noise is generative modeling (Song et al.'s one-line summary of the field).

Two intuitions are worth holding at once:

  1. The thermodynamic picture (DDPM). A drop of ink diffusing into water is irreversible in practice, but if you knew the velocity of every molecule you could in principle reverse it. The forward "add noise" process is the ink spreading; the network learns the reverse drift that un-spreads it.
  2. The hill-climbing picture (score-based). Imagine the data distribution as a landscape where real images sit on high ridges and noise sits in the valleys. The model learns the gradient of log-probability — which direction is "more like real data" — and sampling is just repeatedly stepping uphill while jittering, so you don't collapse onto a single peak.

These two stories turn out to be the same mathematics viewed from different angles, which is why the field consolidated so fast.

This article assumes you're comfortable with neural nets and gradients (see [[neural-network-from-scratch]] and [[math-for-ml-foundations]]); the architectures used inside diffusion models lean heavily on [[attention-and-transformers]].

Why it matters

Diffusion models are the engine behind nearly every modern high-fidelity image, video, and audio generator: Stable Diffusion, Midjourney, DALL·E 2/3, Imagen, Sora-style video, and the FLUX family. They displaced GANs as the dominant image-generation paradigm between 2021 and 2023 for concrete reasons:

  • Stable training. GANs are a minimax game prone to mode collapse and oscillation. Diffusion training is a plain regression loss (predict the noise) — it just works, and it scales predictably.
  • Mode coverage / diversity. GANs tend to drop modes; diffusion models cover the data distribution far more completely.
  • Controllability. Conditioning (text prompts, depth maps, masks) and guidance knobs let you trade fidelity against diversity at sample time, without retraining.
  • State-of-the-art quality. DDPM already matched ProgressiveGAN on 256×256 LSUN in 2020; by 2022 latent diffusion made high-resolution text-to-image cheap enough to run on a consumer GPU.

Beyond pixels, the framework is what matters: it is a general recipe for learning to sample from any complicated distribution by learning a denoiser/score/velocity field. That generality is why it now appears in molecule design, protein structure, robotics policies, and audio.

How it works (real mechanics)

1. The forward process (fixed, no learning)

Pick a data point $x_0$. Define a Markov chain that adds Gaussian noise over $T$ steps according to a variance schedule $\beta_1, \dots, \beta_T$ (small, increasing):

$$q(x_t \mid x_{t-1}) = \mathcal{N}\!\left(x_t;\ \sqrt{1-\beta_t}\,x_{t-1},\ \beta_t \mathbf{I}\right)$$

The key algebraic gift of DDPM: because each step is Gaussian, you can jump to any timestep in closed form without simulating the chain. Let $\alpha_t = 1-\beta_t$ and $\bar\alpha_t = \prod_{s=1}^{t}\alpha_s$. Then

$$q(x_t \mid x_0) = \mathcal{N}\!\left(x_t;\ \sqrt{\bar\alpha_t}\,x_0,\ (1-\bar\alpha_t)\mathbf{I}\right)$$

which means you can sample a noised version directly:

$$x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1-\bar\alpha_t}\,\epsilon, \qquad \epsilon \sim \mathcal{N}(0, \mathbf{I})$$

As $t \to T$, $\bar\alpha_t \to 0$ and $x_T$ becomes (approximately) standard Gaussian noise. This closed form is what makes training cheap: you never run the forward chain step by step.

2. The reverse process (the learned part)

We want $p_\theta(x_{t-1} \mid x_t)$ to undo one noising step. It's parameterized as a Gaussian whose mean is predicted by a network and whose variance is usually fixed to a schedule constant $\sigma_t^2$:

$$p_\theta(x_{t-1}\mid x_t) = \mathcal{N}\!\left(x_{t-1};\ \mu_\theta(x_t, t),\ \sigma_t^2 \mathbf{I}\right)$$

DDPM's crucial reparameterization: instead of predicting the mean directly, predict the noise $\epsilon$ that was added. Train a network $\epsilon_\theta(x_t, t)$ and recover the mean as:

$$\mu_\theta(x_t, t) = \frac{1}{\sqrt{\alpha_t}}\left(x_t - \frac{\beta_t}{\sqrt{1-\bar\alpha_t}}\,\epsilon_\theta(x_t, t)\right)$$

3. The training objective

The principled objective is a variational bound on $-\log p_\theta(x_0)$. DDPM's headline result is that this collapses to a stunningly simple denoising loss when you use the noise parameterization and drop the time-dependent weighting:

$$L_\text{simple} = \mathbb{E}_{t,\,x_0,\,\epsilon}\left[\big\lVert \epsilon - \epsilon_\theta(\underbrace{\sqrt{\bar\alpha_t}\,x_0 + \sqrt{1-\bar\alpha_t}\,\epsilon}_{x_t},\ t)\big\rVert^2\right]$$

In words: noise an image to a random level $t$, hand it to the network, and ask it to predict the exact noise you added. That's it — an MSE regression. Training pseudocode:

# DDPM training step (one minibatch)
x0   = sample_data()                       # clean image(s)
t    = randint(1, T)                       # random timestep per sample
eps  = randn_like(x0)                      # the noise to add
xt   = sqrt(abar[t]) * x0 + sqrt(1 - abar[t]) * eps   # closed-form noising
loss = mse(eps, model(xt, t))              # predict the noise
loss.backward(); opt.step()

4. Sampling (ancestral / DDPM)

Start from $x_T \sim \mathcal{N}(0,\mathbf{I})$ and walk backwards step by step:

# DDPM ancestral sampling
x = randn(shape)                           # pure noise
for t in reversed(range(1, T + 1)):
    eps_hat = model(x, t)
    mean = (x - beta[t]/sqrt(1 - abar[t]) * eps_hat) / sqrt(alpha[t])
    z = randn_like(x) if t > 1 else 0      # no noise on the last step
    x = mean + sigma[t] * z
return x

The network architecture under model was historically a U-Net with residual blocks, self-attention at low resolutions, and a sinusoidal time embedding injected into every block (the same family of positional signal discussed in [[positional-encoding]]). Modern systems replace the U-Net with a diffusion transformer (DiT) — see §8.

5. Score-based view and the SDE unification

Song et al. showed DDPM is one discretization of a stochastic differential equation. The forward noising is a continuous SDE; generation runs the reverse-time SDE, which depends only on the score $\nabla_{x}\log p_t(x)$ — the gradient of the log data density at noise level $t$. The denoiser and the score are equivalent up to scaling: $\epsilon_\theta(x_t,t) \approx -\sqrt{1-\bar\alpha_t}\,\nabla_x \log p_t(x_t)$. The training objective for the score is denoising score matching, which is again just "predict the noise/clean-signal."

This framework unifies the two main noise schedules:

  • VP-SDE (variance preserving) ≙ DDPM: the signal is scaled down as noise is added, total variance stays ~1.
  • VE-SDE (variance exploding) ≙ Song & Ermon's earlier NCSN: signal kept, noise variance grows without bound.

Two more gifts fall out:

  • Probability-flow ODE. Every diffusion SDE has a deterministic ODE with the same marginals. Solve it with a standard ODE solver → deterministic sampling, exact likelihoods, and a meaningful latent space.
  • Predictor–corrector samplers. Alternate an SDE/ODE step (predictor) with Langevin score steps (corrector) to fix discretization error.

6. DDIM — fast deterministic sampling

DDPM needs hundreds-to-thousands of steps because each step is small and stochastic. DDIM (Song, Meng, Ermon) constructs a family of non-Markovian forward processes that share DDPM's exact training objective — so you can train DDPM-style and then sample differently, with no retraining. The DDIM update predicts $x_0$ from the current noise estimate and re-noises toward $t-1$:

$$x_{t-1} = \sqrt{\bar\alpha_{t-1}}\,\underbrace{\left(\frac{x_t - \sqrt{1-\bar\alpha_t}\,\epsilon_\theta(x_t,t)}{\sqrt{\bar\alpha_t}}\right)}_{\hat{x}_0}

  • \sqrt{1-\bar\alpha_{t-1}-\sigma_t^2}\;\epsilon_\theta(x_t,t) + \sigma_t z$$

The knob $\eta$ scales $\sigma_t$: $\eta=1$ recovers stochastic DDPM; $\eta=0$ is fully deterministic (the probability-flow ODE discretization). With $\eta=0$ you can skip timesteps and produce high-quality samples 10×–50× faster in wall-clock time, get reproducible samples from a fixed seed, and do semantically meaningful interpolation in the noise latent.

7. Conditioning and guidance

Classifier guidance (Dhariwal & Nichol) steers samples toward a class by adding the gradient of a separately trained noisy-image classifier to the score. It works but needs an extra classifier trained on noised data.

Classifier-free guidance (CFG) (Ho & Salimans) is the technique everyone uses now. Train one model that is sometimes conditional and sometimes not — randomly drop the conditioning (e.g. 10–20% of the time) so the same network learns both $\epsilon_\theta(x_t, t, c)$ and the unconditional $\epsilon_\theta(x_t, t, \varnothing)$. At sample time, extrapolate away from the unconditional toward the conditional with guidance weight $w$:

$$\hat\epsilon = \epsilon_\theta(x_t, t, \varnothing) + w\,\big(\epsilon_\theta(x_t, t, c) - \epsilon_\theta(x_t, t, \varnothing)\big)$$

$w=1$ is ordinary conditioning; $w>1$ sharpens prompt adherence and fidelity at the cost of diversity (and, pushed too high, saturated/over-baked images). This single knob is why prompt strength sliders exist in every tool. The cost: two forward passes per step.

8. Latent diffusion (Stable Diffusion)

Running diffusion directly on 512×512 pixels is brutally expensive. Latent Diffusion Models (Rombach et al.) split the problem in two:

  1. Perceptual compression. Train a VAE/autoencoder (with a perceptual + patch-adversarial loss) that maps an image into a much smaller latent (e.g. 64×64×4 for a 512×512 image — roughly 48× fewer elements). The encoder throws away imperceptible high-frequency detail; the decoder reconstructs it.
  2. Diffusion in latent space. Run the entire DDPM/score machinery on those latents instead of pixels — a near-optimal trade-off between compression and detail. This slashes compute by an order of magnitude and is what made text-to-image runnable on a single consumer GPU.

Text conditioning enters via cross-attention: text tokens (from a frozen text encoder like CLIP or T5) become keys/values, and the denoiser's spatial features are the queries — the prompt literally attends into the image at every block (same attention mechanism as [[attention-and-transformers]]). This same cross-attention slot accepts depth maps, segmentation masks, and bounding boxes, which is the foundation for ControlNet-style conditioning. The released CompVis/Stability weights built on this became Stable Diffusion.

9. The frontier: consistency, flow matching, rectified flow

The defining weakness of diffusion is many network evaluations per sample. The 2022–2026 frontier is mostly about killing that cost.

Consistency models (Song, Dhariwal, Chen, Sutskever, 2023). Learn a function $f_\theta(x_t, t)$ that maps any point on a probability-flow ODE trajectory directly to its origin $x_0$. The defining self-consistency property: points on the same trajectory map to the same output, with the boundary condition $f_\theta(x_0, 0) = x_0$. Then one network evaluation turns noise into an image. Two training routes: consistency distillation (distill a pretrained diffusion model's ODE) or consistency training (from scratch). Reported SOTA one-step results of FID 3.55 on CIFAR-10 and 6.20 on ImageNet 64×64, with optional multi-step sampling to trade compute for quality, plus zero-shot editing (inpainting, colorization, super-resolution).

Flow Matching (Lipman et al., 2022). Reframe generation as learning a velocity field that continuously transports the noise distribution to the data distribution — a Continuous Normalizing Flow trained without simulating any ODE during training (simulation-free). You pick a conditional probability path between a noise sample and a data sample, then regress the network onto the known instantaneous velocity along that path:

$$L_\text{CFM} = \mathbb{E}_{t,\,x_0,\,x_1}\big[\lVert v_\theta(x_t, t) - u_t(x_t \mid x_1)\rVert^2\big]$$

Diffusion is a special case (curved Gaussian paths). The important new choice is straight-line / optimal-transport paths — rectified flow — where the path from noise to data is a literal straight interpolation $x_t = (1-t)\,x_0 + t\,x_1$ and the target velocity is just $x_1 - x_0$. Straight paths are faster to train, faster to sample (fewer, larger ODE steps), and generalize better.

Rectified-flow transformers are the current SOTA (2024–2026). Stable Diffusion 3 (Esser et al., 2024) trains a rectified-flow objective with an MM-DiT — a multimodal diffusion transformer that gives text and image tokens separate weights but lets information flow bidirectionally between them — plus a logit-normal timestep weighting that biases training toward perceptually relevant noise levels. It follows clean scaling-law trends ([[scaling-laws]]). SD3.5 scaled this to ~8B parameters (with 1–4-step Turbo variants via adversarial diffusion distillation), and FLUX / FLUX 2 (Black Forest Labs; FLUX 2 is a 32B latent flow-matching model released Nov 2025) pushed flow-matching DiTs with RoPE positional encoding to the frontier of prompt adherence and typography. The same recipe now spans audio, video, and even some language modeling.

Key ideas & tradeoffs

  • Training is trivially stable; sampling is the cost center. Diffusion swaps GAN-style adversarial instability for an easy regression loss — but pays with many sequential network evaluations at generation time. Essentially every advance since 2020 (DDIM, distillation, consistency, rectified flow) is an attack on that sampling cost.
  • Steps ↔ quality is a tunable dial. DDPM ~1000 steps → DDIM ~20–50 → distilled/consistency/turbo 1–4. Fewer steps, lower fidelity, but the frontier keeps shrinking the gap.
  • Guidance weight ↔ fidelity vs diversity. CFG w is the single most important sample-time knob; higher = more prompt-faithful but less diverse and eventually over-saturated.
  • Latent vs pixel. Latent diffusion is the standard for images (cheap, high-res). Pixel diffusion survives where a clean autoencoder is hard to get.
  • SDE vs ODE sampling. Stochastic (SDE) sampling can correct its own errors and sometimes reaches higher quality; deterministic (ODE) sampling is faster, reproducible, and gives a usable latent space for editing/interpolation.
  • Architecture: U-Net → DiT. Convolutional U-Nets dominated early; transformer backbones (DiT, MM-DiT) now scale better and dominate the frontier.
  • Noise schedule matters more than it looks. Cosine vs linear schedules, VP vs VE, and logit-normal timestep weighting all materially change sample quality — much of SD3's gain is "where you spend training steps in noise-level space."

Honest caveats & open questions

  • Slow sampling is still the core limitation. Even with distillation, the highest-quality results often still want 20+ steps. One-step models are improving fast but historically traded away some fidelity/diversity; whether one-step ever fully matches many-step quality across all domains is unsettled.
  • Distillation is lossy and finicky. Consistency distillation, adversarial diffusion distillation, and progressive distillation each introduce their own artifacts and training instabilities (ADD reintroduces a GAN-style discriminator, bringing back some adversarial fragility). Distilled few-step models can lose fine detail and prompt nuance versus their teacher.
  • Guidance is a hack that works. CFG has no clean probabilistic justification — it extrapolates outside the model's learned distribution. High guidance distorts color/contrast and can reduce diversity to near-collapse. "Better guidance" remains an active research line.
  • Likelihoods are awkward. The probability-flow ODE gives exact likelihoods in principle, but diffusion models are not primarily likelihood-optimal, and their bits/dim lag autoregressive models; using them for density estimation is niche.
  • Memorization and copyright. Large diffusion models verifiably regurgitate training images for some prompts. This is a live legal and ethical problem, not a solved one.
  • The "is it really diffusion?" boundary is blurring. Flow matching, rectified flow, stochastic interpolants, and diffusion are increasingly recognized as one continuum. The community has not fully settled terminology, the best path/schedule, or whether SDE or ODE formulations are fundamentally preferable — much is still empirical.
  • Evaluation is shaky. FID is the field's default but correlates imperfectly with human judgment and is sensitive to the feature extractor; cross-paper comparisons should be read with caution.
  • Source caveat for this article. The arXiv abstract pages used here confirm authors, claims, and high-level results but do not carry full equations; the formulas above are the standard, widely-reproduced forms from the DDPM/score-SDE/flow-matching papers and should be cross-checked against the full PDFs before citing exact constants. One secondary source (learnopencv SD3.5 writeup) returned HTTP 403 and could not be fetched; SD3 details here come from the SD3 arXiv abstract plus the 2026 web synthesis.

How it connects to OpenAlice

OpenAlice is an LLM-and-agent ecosystem, so most of the knowledge library is text-model centric ([[attention-and-transformers]], [[embeddings]], [[tokenization]], [[scaling-laws]], [[rlhf-and-alignment]]). Diffusion is the multimodal/generative-media counterpart, and several threads connect:

  • Shared backbone — the transformer. The frontier image models (DiT, MM-DiT, FLUX) are transformers. Everything OpenAlice knows about attention, positional encoding, and [[scaling-laws]] transfers directly; image generation is increasingly "a transformer with a different loss and a different denoising loop."
  • Shared inference economics. Diffusion's "many sequential forward passes" problem is structurally analogous to autoregressive decoding latency covered in [[llm-inference-internals]] — and the fixes rhyme (caching, distillation, fewer/larger steps, [[quantization]] for the denoiser). A diffusion serving path would reuse much of the same throughput/latency engineering.
  • Guidance ≈ steering. Classifier-free guidance is conceptually a cousin of the prompt-steering and reward-shaping ideas in [[rlhf-and-alignment]] and process/outcome reward work — a sample-time knob that biases generation toward a target without retraining.
  • The unifying lens. Flow matching / rectified flow position generation as "learn a velocity field that transports one distribution to another." That framing is general enough to matter if OpenAlice ever generates images, audio, avatars, or VRM/streaming assets for the live products — the same denoiser-as-sampler recipe applies across modalities.

If OpenAlice grows a media-generation capability (avatars, thumbnails, stream backdrops, voice), latent diffusion + a rectified-flow DiT + CFG + a few-step distilled sampler is the current default stack to reach for, and this article is the entry point for that surface.