Definition
Diffusion model (also denoising diffusion model or score-based generative model) is a generative model defined by two processes. A fixed forward process turns a data sample into nearly pure Gaussian noise through small noising steps. A learned reverse process removes the noise again, one step at a time. The only trained component is a network that predicts the noise in . Equivalently it estimates the score of the noised data distribution. A sample is generated by drawing and running the reverse process down to .
The idea comes from non-equilibrium thermodynamics. Sohl-Dickstein, Weiss, Maheswaranathan and Ganguli, "Deep Unsupervised Learning using Nonequilibrium Thermodynamics" (ICML 2015), observed that a slow diffusion destroys structure in a way that is easy to describe, and that if every step is small enough its reversal is also approximately Gaussian. A network only has to learn the mean (and possibly the variance) of each small reverse step. They trained it by maximising a variational lower bound on the log-likelihood, as in a Variational autoencoder (VAE) with a fixed encoder. The samples were not competitive, and the approach stayed niche for five years.
Two lines of work then met. Song and Ermon (NeurIPS 2019) trained a noise-conditional score network with denoising score matching at many noise levels and sampled with annealed Langevin dynamics. Ho, Jain and Abbeel (NeurIPS 2020) re-parameterised the 2015 model so that the network predicts the added noise, and trained it with a plain mean-squared error (see Denoising diffusion probabilistic models (DDPM)). The two were the same model seen from different sides. Song et al. (ICLR 2021) then unified both as discretisations of a stochastic differential equation. Within two years diffusion models overtook GANs on ImageNet (Dhariwal and Nichol, NeurIPS 2021) and became the standard for text-to-image, video and audio generation.
The forward process: data to noise
- variance added at step (the noise schedule); DDPM uses a linear ramp from to over steps
- shrinks the signal each step so that the total variance stays at 1 for unit-variance data ("variance preserving")
- fraction of the original signal variance left after steps; for the DDPM schedule
- closed form for any : with , so training never has to run the chain
The forward process has no parameters. Its only design choice is the schedule, often summarised by the signal-to-noise ratio . With the linear schedule the SNR falls below 1 at , so roughly the last three quarters of the chain are dominated by noise. The cosine schedule of Nichol and Dhariwal (2021) keeps more signal in the middle steps. Because barely depends on , the end of the chain is close to for any data, and that is the prior we sample from.
The reverse process: noise to data
Reversing a single step needs , which depends on the unknown data distribution. Conditioned on , though, the reverse step is a known Gaussian, . The model therefore estimates , or equivalently the noise , from and plugs the estimate in. Everything is driven by one network:
- uniform on ; one network handles every noise level, receiving through an embedding
- noise predictor, usually a U-Net or a transformer over image patches or latents
- the implied denoised estimate; at the optimum it is the posterior mean
The loss, its weighting and its link to the variational bound are covered in Diffusion noise-prediction loss. Two facts matter here. The optimal network is a conditional mean, so no single step has to represent the multimodal data distribution: the chain of small steps does that. And because is a mean, a one-step jump from pure noise lands on the average of the data (for example, the centre of a ring of modes). Sharp samples need many small steps or a good ODE solver.
Three views of the same model
| view | what the network outputs | sampler | sibling article |
|---|---|---|---|
| Latent-variable model (DDPM) | noise ε_θ(x_t, t), i.e. the mean of p_θ(x_{t−1}|x_t) | ancestral sampling, T stochastic steps | Denoising diffusion probabilistic models (DDPM) |
| Score matching (NCSN) | score s_θ(x, t) ≈ ∇log q_t(x) = −ε_θ/√(1−ᾱ_t) | annealed Langevin dynamics | Score-based generative models |
| SDE / ODE | drift term of a reverse-time SDE or of the probability-flow ODE | any SDE or ODE solver; DDIM is a first-order ODE step | Denoising diffusion implicit models (DDIM) |
- the variance-preserving forward SDE, the limit of DDPM's chain (Song et al., ICLR 2021)
- the reverse-time SDE (Anderson, 1982): it needs only the score , which the network provides; is Brownian motion running backward in time
- the probability-flow ODE: a deterministic path with the same marginals at every , so maps to by a bijection
The ODE view explains most of what came after. Deterministic samplers such as DDIM integrate it with a few large steps. Encoding real data into noise ("inversion") runs it backward. Exact likelihoods follow from the instantaneous change-of-variables formula, as in Normalizing flows. Flow matching and Rectified flow train the velocity field of such an ODE directly, and Consistency models learn to jump along it in one step.
Guidance: trading diversity for fidelity
A conditional model (class label, text embedding via Cross-attention) samples from , but in practice samples look better when they are pushed further toward the condition. Classifier guidance (Dhariwal and Nichol, 2021) adds the gradient of a classifier trained on noisy images. Classifier-free guidance (Ho and Salimans, NeurIPS 2021 workshop) needs no classifier. One network is trained both with and without the condition: the label is replaced by a null token with a small probability (0.1 or 0.2 worked best in the paper). At sampling time the two predictions are extrapolated:
- classifier-guidance scale; with an unconditional the sampler targets up to normalisation (Dhariwal and Nichol also guide conditional models)
- classifier-free guidance weight; is the plain conditional model. Many libraries write the same rule as with
- the null condition; the difference acts as an implicit classifier gradient,
Guidance raises sample quality and agreement with the prompt but lowers diversity, and at large it oversaturates images. Classifier-free guidance doubles the number of network evaluations per step (conditional and unconditional), usually done as one batch of two. Almost every text-to-image system, including Latent diffusion models and Stable Diffusion, uses it.
Milestones
- 2015
- Sohl-Dickstein et al. (ICML): diffusion probabilistic models trained by a variational bound
- 2019
- Song & Ermon (NeurIPS): noise-conditional score networks, annealed Langevin sampling
- 2020
- Ho, Jain & Abbeel (NeurIPS): DDPM, ε-prediction and ; CIFAR-10 FID 3.17
- 2021
- Song et al. (ICLR): SDE framework and probability-flow ODE. Song, Meng & Ermon (ICLR): DDIM, 10–50× faster sampling. Nichol & Dhariwal (ICML): cosine schedule, learned variances
- 2021
- Dhariwal & Nichol (NeurIPS), "Diffusion Models Beat GANs on Image Synthesis": classifier guidance, ImageNet 256×256 FID 4.59 against 6.95 for BigGAN-deep
- 2022
- Classifier-free guidance; latent diffusion (Rombach et al., CVPR), which runs the process in an autoencoder's latent space
Reverse: drag t from 1000 down to 0 to watch noise become data. The slider shows the chain once it has reached step t.
Worked example: the schedule and one guided prediction
| t | β_t | ᾱ_t | √ᾱ_t | SNR | mode std at t |
|---|---|---|---|---|---|
| 1 | 0.00010 | 0.9999 | 0.9999 | 9999 | 0.081 |
| 100 | 0.00207 | 0.8970 | 0.9471 | 8.71 | 0.330 |
| 250 | 0.00506 | 0.5241 | 0.7239 | 1.10 | 0.692 |
| 500 | 0.01004 | 0.0786 | 0.2803 | 0.0853 | 0.960 |
| 750 | 0.01502 | 0.00335 | 0.0579 | 0.00336 | 0.998 |
| 1000 | 0.02000 | 4.04 × 10⁻⁵ | 0.00635 | 4.04 × 10⁻⁵ | 1.000 |
Classifier-free guidance at x_t = (0.6, 0.4), t = 150, on the ring data (exact ε*, class A)
- 1At , , so and . The point sits between a class-A mode at and a class-B mode at .
- 2Unconditional prediction: , so , about halfway between the two modes.
- 3Class-A prediction: , giving . The condition has moved the estimate toward the class-A mode at .
- 4Guidance with : and . With : and , which overshoots past the mode's . Large extrapolates beyond anything the conditional model predicts.
1import math, torch, torch.nn as nn23T = 10004betas = torch.linspace(1e-4, 0.02, T) # DDPM linear schedule5abar = torch.cumprod(1 - betas, 0) # abar[t-1] = ᾱ_t67def data(n): # ring of 8 Gaussians; class = component parity8 k = torch.randint(0, 8, (n,))9 ang = 2 * math.pi * k / 810 x0 = torch.stack([ang.cos(), ang.sin()], 1) + 0.08 * torch.randn(n, 2)11 return x0, k % 21213class EpsNet(nn.Module): # ε_θ(x_t, t, c); c = 2 means "no condition"14 def __init__(self, h=128):15 super().__init__()16 self.c = nn.Embedding(3, h)17 self.inp = nn.Linear(2 + 1, h)18 self.mlp = nn.Sequential(nn.SiLU(), nn.Linear(h, h), nn.SiLU(), nn.Linear(h, h), nn.SiLU(), nn.Linear(h, 2))19 def forward(self, x, t, c):20 return self.mlp(self.inp(torch.cat([x, (t.float() / T).unsqueeze(1)], 1)) + self.c(c))2122net = EpsNet()23opt = torch.optim.Adam(net.parameters(), lr=1e-3)24for it in range(3000):25 x0, c = data(512)26 t = torch.randint(1, T + 1, (512,))27 eps = torch.randn_like(x0)28 a = abar[t - 1].unsqueeze(1)29 xt = a.sqrt() * x0 + (1 - a).sqrt() * eps # forward process in one step30 c = torch.where(torch.rand(512) < 0.1, torch.full_like(c, 2), c) # drop the label 10% of the time31 loss = ((net(xt, t, c) - eps) ** 2).mean() # L_simple32 opt.zero_grad(); loss.backward(); opt.step()3334@torch.no_grad()35def sample(n, cls, w=0.0, S=50, eta=0.0):36 """Generalised DDIM sampler over S evenly spaced steps; eta = 1 is DDPM-like, eta = 0 the ODE."""37 taus = torch.linspace(0, T, S + 1).round().long() # τ_0 = 0, ..., τ_S = T38 x = torch.randn(n, 2)39 c = torch.full((n,), cls)40 u = torch.full((n,), 2)41 for i in range(S, 0, -1):42 t, s = taus[i].item(), taus[i - 1].item()43 at = abar[t - 1]44 as_ = abar[s - 1] if s > 0 else torch.tensor(1.0)45 tt = torch.full((n,), t)46 e = net(x, tt, c)47 if w > 0: # classifier-free guidance48 e = (1 + w) * e - w * net(x, tt, u)49 x0 = (x - (1 - at).sqrt() * e) / at.sqrt()50 sig = eta * ((1 - as_) / (1 - at) * (1 - at / as_)).sqrt()51 x = as_.sqrt() * x0 + (1 - as_ - sig**2).sqrt() * e + sig * torch.randn_like(x)52 return x5354torch.manual_seed(0)55xs = sample(1000, cls=0, w=2.0, S=50)56ang = torch.atan2(xs[:, 1], xs[:, 0])57k = torch.round(ang / (2 * math.pi / 8)).long() % 858print(xs.shape) # torch.Size([1000, 2])59print(f"{abar[-1].item():.2e}") # 4.04e-05 (ᾱ_T: x_T is almost pure noise)60print("share in class 0:", (k % 2 == 0).float().mean().item()) # close to 1.0 after trainingUse when
- +Stable training: a mean-squared-error regression with no adversarial game
- +Covers all modes of the data; diversity is high and controllable with guidance
- +One network supports stochastic and deterministic samplers, inversion, inpainting and likelihood evaluation
- +Scales to text-to-image, video and audio, usually in an autoencoder latent space
Avoid when
- −Sampling is iterative: tens to a thousand network calls instead of one
- −Guidance improves fidelity at the cost of diversity, and oversaturates at large weights
- −Training loss is a poor progress signal; quality is judged with samples or FID
- −Pixel-space models at high resolution are expensive; latent diffusion is the usual fix
Check yourself
A trained diffusion model maps x_T straight to its predicted clean sample x̂₀ in a single step. On data made of eight separated modes, what does this produce?
Check yourself
With classifier-free guidance weight w, which prediction does the sampler use?
Remember
- ✓Forward: with ; DDPM uses linear from to , .
- ✓One network predicts the noise; it is also the score, , and the drift of the probability-flow ODE.
- ✓Three views: DDPM (ancestral sampling), score matching (Langevin), SDE/ODE (any solver; DDIM is a first-order ODE step).
- ✓Classifier-free guidance: ; better fidelity, less diversity, twice the NFE.
- ✓Cost: 1000 NFE for DDPM, 10–50 for DDIM, fewer with higher-order solvers or distillation, against 1 for a GAN.