Diffusion models

A diffusion model adds Gaussian noise to data over many small steps until only noise remains, and trains one network to predict that noise. Sampling starts from pure noise and denoises step by step; the network is at once a noise predictor, a score function and the drift of an ODE.

14 min read1139 wordscore0 viewsupdated 2026-10-12

Definition

Diffusion model (also denoising diffusion model or score-based generative model) is a generative model defined by two processes. A fixed forward process turns a data sample x0x_0 into nearly pure Gaussian noise xTx_T through TT small noising steps. A learned reverse process pθ(xt−1∣xt)p_\theta(x_{t-1}\mid x_t) removes the noise again, one step at a time. The only trained component is a network εθ(xt,t)\varepsilon_\theta(x_t,t) that predicts the noise in xtx_t. Equivalently it estimates the score ∇xlog⁡qt(x)\nabla_x\log q_t(x) of the noised data distribution. A sample is generated by drawing xT∼N(0,I)x_T\sim\mathcal N(0,I) and running the reverse process down to x0x_0.

The idea comes from non-equilibrium thermodynamics. Sohl-Dickstein, Weiss, Maheswaranathan and Ganguli, "Deep Unsupervised Learning using Nonequilibrium Thermodynamics" (ICML 2015), observed that a slow diffusion destroys structure in a way that is easy to describe, and that if every step is small enough its reversal is also approximately Gaussian. A network only has to learn the mean (and possibly the variance) of each small reverse step. They trained it by maximising a variational lower bound on the log-likelihood, as in a Variational autoencoder (VAE) with a fixed encoder. The samples were not competitive, and the approach stayed niche for five years.

Two lines of work then met. Song and Ermon (NeurIPS 2019) trained a noise-conditional score network with denoising score matching at many noise levels and sampled with annealed Langevin dynamics. Ho, Jain and Abbeel (NeurIPS 2020) re-parameterised the 2015 model so that the network predicts the added noise, and trained it with a plain mean-squared error (see Denoising diffusion probabilistic models (DDPM)). The two were the same model seen from different sides. Song et al. (ICLR 2021) then unified both as discretisations of a stochastic differential equation. Within two years diffusion models overtook GANs on ImageNet (Dhariwal and Nichol, NeurIPS 2021) and became the standard for text-to-image, video and audio generation.

The forward process: data to noise

q(xt∣xt−1)=N ⁣(1−βt xt−1, βtI),q(xt∣x0)=N ⁣(αˉt x0, (1−αˉt)I),αˉt=∏s=1t(1−βs)q(x_t\mid x_{t-1})=\mathcal N\!\big(\sqrt{1-\beta_t}\,x_{t-1},\ \beta_tI\big),\qquad q(x_t\mid x_0)=\mathcal N\!\big(\sqrt{\bar\alpha_t}\,x_0,\ (1-\bar\alpha_t)I\big),\qquad \bar\alpha_t=\prod_{s=1}^{t}(1-\beta_s)
βt\beta_t
variance added at step tt (the noise schedule); DDPM uses a linear ramp from 10−410^{-4} to 0.020.02 over T=1000T=1000 steps
1−βt\sqrt{1-\beta_t}
shrinks the signal each step so that the total variance stays at 1 for unit-variance data ("variance preserving")
αˉt\bar\alpha_t
fraction of the original signal variance left after tt steps; αˉT=4.04×10−5\bar\alpha_T=4.04\times10^{-5} for the DDPM schedule
q(xt∣x0)q(x_t\mid x_0)
closed form for any tt: xt=αˉt x0+1−αˉt εx_t=\sqrt{\bar\alpha_t}\,x_0+\sqrt{1-\bar\alpha_t}\,\varepsilon with ε∼N(0,I)\varepsilon\sim\mathcal N(0,I), so training never has to run the chain

The forward process has no parameters. Its only design choice is the schedule, often summarised by the signal-to-noise ratio SNR(t)=αˉt/(1−αˉt)\mathrm{SNR}(t)=\bar\alpha_t/(1-\bar\alpha_t). With the linear schedule the SNR falls below 1 at t=260t=260, so roughly the last three quarters of the chain are dominated by noise. The cosine schedule of Nichol and Dhariwal (2021) keeps more signal in the middle steps. Because q(xT∣x0)q(x_T\mid x_0) barely depends on x0x_0, the end of the chain is close to N(0,I)\mathcal N(0,I) for any data, and that is the prior we sample from.

The reverse process: noise to data

Reversing a single step needs q(xt−1∣xt)q(x_{t-1}\mid x_t), which depends on the unknown data distribution. Conditioned on x0x_0, though, the reverse step is a known Gaussian, q(xt−1∣xt,x0)q(x_{t-1}\mid x_t,x_0). The model therefore estimates x0x_0, or equivalently the noise ε\varepsilon, from xtx_t and plugs the estimate in. Everything is driven by one network:

min⁡θ Et,x0,ε∥ε−εθ(αˉt x0+1−αˉt ε, t)∥2,x^0(xt)=xt−1−αˉt εθ(xt,t)αˉt\min_\theta\ \mathbb E_{t,x_0,\varepsilon}\big\|\varepsilon-\varepsilon_\theta(\sqrt{\bar\alpha_t}\,x_0+\sqrt{1-\bar\alpha_t}\,\varepsilon,\ t)\big\|^2,\qquad \hat x_0(x_t)=\frac{x_t-\sqrt{1-\bar\alpha_t}\,\varepsilon_\theta(x_t,t)}{\sqrt{\bar\alpha_t}}
tt
uniform on {1,…,T}\{1,\dots,T\}; one network handles every noise level, receiving tt through an embedding
εθ\varepsilon_\theta
noise predictor, usually a U-Net or a transformer over image patches or latents
x^0\hat x_0
the implied denoised estimate; at the optimum it is the posterior mean E[x0∣xt]\mathbb E[x_0\mid x_t]

The loss, its weighting and its link to the variational bound are covered in Diffusion noise-prediction loss. Two facts matter here. The optimal network is a conditional mean, so no single step has to represent the multimodal data distribution: the chain of small steps does that. And because x^0\hat x_0 is a mean, a one-step jump from pure noise lands on the average of the data (for example, the centre of a ring of modes). Sharp samples need many small steps or a good ODE solver.

Three views of the same model

viewwhat the network outputssamplersibling article
Latent-variable model (DDPM)noise ε_θ(x_t, t), i.e. the mean of p_θ(x_{t−1}|x_t)ancestral sampling, T stochastic stepsDenoising diffusion probabilistic models (DDPM)
Score matching (NCSN)score s_θ(x, t) ≈ ∇log q_t(x) = −ε_θ/√(1−ᾱ_t)annealed Langevin dynamicsScore-based generative models
SDE / ODEdrift term of a reverse-time SDE or of the probability-flow ODEany SDE or ODE solver; DDIM is a first-order ODE stepDenoising diffusion implicit models (DDIM)
The trained network is the same object in all three; only the sampler and the bookkeeping differ
dx=−12β(t) x dt+β(t) dw,dx=[−12β(t) x−β(t)∇xlog⁡qt(x)]dt+β(t) dwˉ,dxdt=−12β(t)[x+∇xlog⁡qt(x)]dx=-\tfrac12\beta(t)\,x\,dt+\sqrt{\beta(t)}\,dw,\qquad dx=\Big[-\tfrac12\beta(t)\,x-\beta(t)\nabla_x\log q_t(x)\Big]dt+\sqrt{\beta(t)}\,d\bar w,\qquad \frac{dx}{dt}=-\tfrac12\beta(t)\Big[x+\nabla_x\log q_t(x)\Big]
first\text{first}
the variance-preserving forward SDE, the T→∞T\to\infty limit of DDPM's chain (Song et al., ICLR 2021)
second\text{second}
the reverse-time SDE (Anderson, 1982): it needs only the score ∇log⁡qt\nabla\log q_t, which the network provides; dwˉd\bar w is Brownian motion running backward in time
third\text{third}
the probability-flow ODE: a deterministic path with the same marginals qtq_t at every tt, so xTx_T maps to x0x_0 by a bijection

The ODE view explains most of what came after. Deterministic samplers such as DDIM integrate it with a few large steps. Encoding real data into noise ("inversion") runs it backward. Exact likelihoods follow from the instantaneous change-of-variables formula, as in Normalizing flows. Flow matching and Rectified flow train the velocity field of such an ODE directly, and Consistency models learn to jump along it in one step.

Guidance: trading diversity for fidelity

A conditional model εθ(xt,t,c)\varepsilon_\theta(x_t,t,c) (class label, text embedding via Cross-attention) samples from p(x∣c)p(x\mid c), but in practice samples look better when they are pushed further toward the condition. Classifier guidance (Dhariwal and Nichol, 2021) adds the gradient of a classifier pϕ(c∣xt)p_\phi(c\mid x_t) trained on noisy images. Classifier-free guidance (Ho and Salimans, NeurIPS 2021 workshop) needs no classifier. One network is trained both with and without the condition: the label is replaced by a null token with a small probability (0.1 or 0.2 worked best in the paper). At sampling time the two predictions are extrapolated:

ε^=εθ(xt,t)−s1−αˉt ∇xtlog⁡pϕ(c∣xt),ε~=(1+w) εθ(xt,t,c)−w εθ(xt,t,∅)\hat\varepsilon=\varepsilon_\theta(x_t,t)-s\sqrt{1-\bar\alpha_t}\,\nabla_{x_t}\log p_\phi(c\mid x_t),\qquad \tilde\varepsilon=(1+w)\,\varepsilon_\theta(x_t,t,c)-w\,\varepsilon_\theta(x_t,t,\varnothing)
ss
classifier-guidance scale; with an unconditional εθ\varepsilon_\theta the sampler targets p(x) pϕ(c∣x)sp(x)\,p_\phi(c\mid x)^{s} up to normalisation (Dhariwal and Nichol also guide conditional models)
ww
classifier-free guidance weight; w=0w=0 is the plain conditional model. Many libraries write the same rule as ε∅+g(εc−ε∅)\varepsilon_\varnothing+g(\varepsilon_c-\varepsilon_\varnothing) with g=1+wg=1+w
∅\varnothing
the null condition; the difference εc−ε∅\varepsilon_c-\varepsilon_\varnothing acts as an implicit classifier gradient, −1−αˉt ∇log⁡p(c∣xt)-\sqrt{1-\bar\alpha_t}\,\nabla\log p(c\mid x_t)

Guidance raises sample quality and agreement with the prompt but lowers diversity, and at large ww it oversaturates images. Classifier-free guidance doubles the number of network evaluations per step (conditional and unconditional), usually done as one batch of two. Almost every text-to-image system, including Latent diffusion models and Stable Diffusion, uses it.

Milestones

2015
Sohl-Dickstein et al. (ICML): diffusion probabilistic models trained by a variational bound
2019
Song & Ermon (NeurIPS): noise-conditional score networks, annealed Langevin sampling
2020
Ho, Jain & Abbeel (NeurIPS): DDPM, ε-prediction and LsimpleL_{\text{simple}}; CIFAR-10 FID 3.17
2021
Song et al. (ICLR): SDE framework and probability-flow ODE. Song, Meng & Ermon (ICLR): DDIM, 10–50× faster sampling. Nichol & Dhariwal (ICML): cosine schedule, learned variances
2021
Dhariwal & Nichol (NeurIPS), "Diffusion Models Beat GANs on Image Synthesis": classifier guidance, ImageNet 256×256 FID 4.59 against 6.95 for BigGAN-deep
2022
Classifier-free guidance; latent diffusion (Rombach et al., CVPR), which runs the process in an autoencoder's latent space
x_T ~ N(0, I) → denoise with ε*(x_t, t) → x₀┅ 2-std circle of each noised modet0 of 1000ᾱ_t1.000SNR = ᾱ/(1−ᾱ)∞mode std at t0.080NFE100final samples (S = 100):within 3σ of a mode 99.2%mean dist / σ 1.01 (data ≈ 1.25)modes 24 39 28 27 31 32 25 34ᾱ_t, linear β 1e-4 → 0.02t = 01000ε*(x, t) = −√(1−ᾱ_t)·∇log q_t(x) is exact here: it plays a perfectly trained networkcolour = class of the mode each sample ends in · fresh noise z at every step
process
steps S
condition

Reverse: drag t from 1000 down to 0 to watch noise become data. The slider shows the chain once it has reached step t.

fig.Eight Gaussian modes (std 0.08) on a unit ring, alternating between class A (blue) and class B (amber). Because the data is a Gaussian mixture, every noised marginal is one too, so the optimal noise predictor ε*(x, t) is computed exactly: it plays a perfectly trained network. Forward: drag t and the 240 points blur into N(0, I). At t = 250, ᾱ = 0.524 and SNR = 1.10. At t = 400 the per-mode std is already 0.898, and at t = 1000 ᾱ = 4.0 × 10⁻⁵. Reverse: the same 240 noise vectors x_T are denoised with S evenly spaced steps. The default (stochastic, η = 1, S = 100) puts 99.2% of samples within 3σ of a mode, spread over all eight modes (24 to 39 per mode). With S = 10 only 63.3% land on a mode, with S = 20 90.0%, and with S = 50 100%. The deterministic ODE sampler (η = 0) sends every x_T to the same mode whatever S is (62.1% on-mode at S = 10, 98.8% at S = 1000). Choose class A and all samples land on class-A modes (100%). Raising the classifier-free weight w pulls them tighter: under the ODE, the mean distance to the nearest mode falls from 1.03σ at w = 0 to 0.75σ at w = 2 (about 1.25σ for the true distribution). Guidance doubles the network calls per step (NFE 200 at S = 100).

Worked example: the schedule and one guided prediction

tβ_tᾱ_t√ᾱ_tSNRmode std at t
10.000100.99990.999999990.081
1000.002070.89700.94718.710.330
2500.005060.52410.72391.100.692
5000.010040.07860.28030.08530.960
7500.015020.003350.05790.003360.998
10000.020004.04 × 10⁻⁵0.006354.04 × 10⁻⁵1.000
DDPM linear schedule (β from 10⁻⁴ to 0.02, T = 1000), computed in Node; the last column is the std of one ring mode (data std 0.08) after noising

Classifier-free guidance at x_t = (0.6, 0.4), t = 150, on the ring data (exact ε*, class A)

  1. 1At t=150t=150, αˉt=0.7883\bar\alpha_t=0.7883, so αˉt=0.8879\sqrt{\bar\alpha_t}=0.8879 and 1−αˉt=0.4601\sqrt{1-\bar\alpha_t}=0.4601. The point sits between a class-A mode at (1,0)(1,0) and a class-B mode at (0.707,0.707)(0.707,0.707).
  2. 2Unconditional prediction: ε∅=(0.0097, 0.0046)\varepsilon_\varnothing=(0.0097,\,0.0046), so x^0=(xt−0.4601 ε)/0.8879=(0.671, 0.448)\hat x_0=(x_t-0.4601\,\varepsilon)/0.8879=(0.671,\,0.448), about halfway between the two modes.
  3. 3Class-A prediction: εc=(−0.0037, 0.3034)\varepsilon_c=(-0.0037,\,0.3034), giving x^0=(0.678, 0.293)\hat x_0=(0.678,\,0.293). The condition has moved the estimate toward the class-A mode at (1,0)(1,0).
  4. 4Guidance with w=1w=1: ε~=2εc−ε∅=(−0.0171, 0.6022)\tilde\varepsilon=2\varepsilon_c-\varepsilon_\varnothing=(-0.0171,\,0.6022) and x^0=(0.685, 0.139)\hat x_0=(0.685,\,0.139). With w=3w=3: ε~=4εc−3ε∅=(−0.0440, 1.1998)\tilde\varepsilon=4\varepsilon_c-3\varepsilon_\varnothing=(-0.0440,\,1.1998) and x^0=(0.699, −0.171)\hat x_0=(0.699,\,-0.171), which overshoots past the mode's y=0y=0. Large ww extrapolates beyond anything the conditional model predicts.
python
1import math, torch, torch.nn as nn
2
3T = 1000
4betas = torch.linspace(1e-4, 0.02, T) # DDPM linear schedule
5abar = torch.cumprod(1 - betas, 0) # abar[t-1] = ᾱ_t
6
7def data(n): # ring of 8 Gaussians; class = component parity
8 k = torch.randint(0, 8, (n,))
9 ang = 2 * math.pi * k / 8
10 x0 = torch.stack([ang.cos(), ang.sin()], 1) + 0.08 * torch.randn(n, 2)
11 return x0, k % 2
12
13class EpsNet(nn.Module): # ε_θ(x_t, t, c); c = 2 means "no condition"
14 def __init__(self, h=128):
15 super().__init__()
16 self.c = nn.Embedding(3, h)
17 self.inp = nn.Linear(2 + 1, h)
18 self.mlp = nn.Sequential(nn.SiLU(), nn.Linear(h, h), nn.SiLU(), nn.Linear(h, h), nn.SiLU(), nn.Linear(h, 2))
19 def forward(self, x, t, c):
20 return self.mlp(self.inp(torch.cat([x, (t.float() / T).unsqueeze(1)], 1)) + self.c(c))
21
22net = EpsNet()
23opt = torch.optim.Adam(net.parameters(), lr=1e-3)
24for it in range(3000):
25 x0, c = data(512)
26 t = torch.randint(1, T + 1, (512,))
27 eps = torch.randn_like(x0)
28 a = abar[t - 1].unsqueeze(1)
29 xt = a.sqrt() * x0 + (1 - a).sqrt() * eps # forward process in one step
30 c = torch.where(torch.rand(512) < 0.1, torch.full_like(c, 2), c) # drop the label 10% of the time
31 loss = ((net(xt, t, c) - eps) ** 2).mean() # L_simple
32 opt.zero_grad(); loss.backward(); opt.step()
33
34@torch.no_grad()
35def sample(n, cls, w=0.0, S=50, eta=0.0):
36 """Generalised DDIM sampler over S evenly spaced steps; eta = 1 is DDPM-like, eta = 0 the ODE."""
37 taus = torch.linspace(0, T, S + 1).round().long() # τ_0 = 0, ..., τ_S = T
38 x = torch.randn(n, 2)
39 c = torch.full((n,), cls)
40 u = torch.full((n,), 2)
41 for i in range(S, 0, -1):
42 t, s = taus[i].item(), taus[i - 1].item()
43 at = abar[t - 1]
44 as_ = abar[s - 1] if s > 0 else torch.tensor(1.0)
45 tt = torch.full((n,), t)
46 e = net(x, tt, c)
47 if w > 0: # classifier-free guidance
48 e = (1 + w) * e - w * net(x, tt, u)
49 x0 = (x - (1 - at).sqrt() * e) / at.sqrt()
50 sig = eta * ((1 - as_) / (1 - at) * (1 - at / as_)).sqrt()
51 x = as_.sqrt() * x0 + (1 - as_ - sig**2).sqrt() * e + sig * torch.randn_like(x)
52 return x
53
54torch.manual_seed(0)
55xs = sample(1000, cls=0, w=2.0, S=50)
56ang = torch.atan2(xs[:, 1], xs[:, 0])
57k = torch.round(ang / (2 * math.pi / 8)).long() % 8
58print(xs.shape) # torch.Size([1000, 2])
59print(f"{abar[-1].item():.2e}") # 4.04e-05 (ᾱ_T: x_T is almost pure noise)
60print("share in class 0:", (k % 2 == 0).float().mean().item()) # close to 1.0 after training
A complete class-conditional diffusion model on 2-D data: L_simple training with label dropout, then a generalised DDIM/DDPM sampler with classifier-free guidance (runs on CPU in well under a minute)

Use when

  • +Stable training: a mean-squared-error regression with no adversarial game
  • +Covers all modes of the data; diversity is high and controllable with guidance
  • +One network supports stochastic and deterministic samplers, inversion, inpainting and likelihood evaluation
  • +Scales to text-to-image, video and audio, usually in an autoencoder latent space

Avoid when

  • −Sampling is iterative: tens to a thousand network calls instead of one
  • −Guidance improves fidelity at the cost of diversity, and oversaturates at large weights
  • −Training loss is a poor progress signal; quality is judged with samples or FID
  • −Pixel-space models at high resolution are expensive; latent diffusion is the usual fix

Check yourself

A trained diffusion model maps x_T straight to its predicted clean sample x̂₀ in a single step. On data made of eight separated modes, what does this produce?

Check yourself

With classifier-free guidance weight w, which prediction does the sampler use?

Remember

  • ✓Forward: q(xt∣x0)=N(αˉt x0,(1−αˉt)I)q(x_t\mid x_0)=\mathcal N(\sqrt{\bar\alpha_t}\,x_0,(1-\bar\alpha_t)I) with αˉt=∏s(1−βs)\bar\alpha_t=\prod_s(1-\beta_s); DDPM uses linear β\beta from 10−410^{-4} to 0.020.02, T=1000T=1000.
  • ✓One network εθ(xt,t)\varepsilon_\theta(x_t,t) predicts the noise; it is also the score, ∇log⁡qt(x)=−εθ/1−αˉt\nabla\log q_t(x)=-\varepsilon_\theta/\sqrt{1-\bar\alpha_t}, and the drift of the probability-flow ODE.
  • ✓Three views: DDPM (ancestral sampling), score matching (Langevin), SDE/ODE (any solver; DDIM is a first-order ODE step).
  • ✓Classifier-free guidance: ε~=(1+w)εθ(xt,t,c)−wεθ(xt,t,∅)\tilde\varepsilon=(1+w)\varepsilon_\theta(x_t,t,c)-w\varepsilon_\theta(x_t,t,\varnothing); better fidelity, less diversity, twice the NFE.
  • ✓Cost: 1000 NFE for DDPM, 10–50 for DDIM, fewer with higher-order solvers or distillation, against 1 for a GAN.
← back to Deep Learning roadmap