# Diffusion models

> A diffusion model adds Gaussian noise to data over many small steps until only noise remains, and trains one network to predict that noise. Sampling starts from pure noise and denoises step by step; the network is at once a noise predictor, a score function and the drift of an ODE.

Source: https://www.mlfoundry.in/learn/diffusion-models  
Section: Deep Learning › Generative Deep Learning Algorithms  
Level: core · 14 min read · updated 2026-10-12

**Diffusion model:** (also **denoising diffusion model** or **score-based generative model**) is a generative model defined by two processes. A fixed **forward process** turns a data sample $x_0$ into nearly pure Gaussian noise $x_T$ through $T$ small noising steps. A learned **reverse process** $p_\theta(x_{t-1}\mid x_t)$ removes the noise again, one step at a time. The only trained component is a network $\varepsilon_\theta(x_t,t)$ that predicts the noise in $x_t$. Equivalently it estimates the score $\nabla_x\log q_t(x)$ of the noised data distribution. A sample is generated by drawing $x_T\sim\mathcal N(0,I)$ and running the reverse process down to $x_0$.

The idea comes from non-equilibrium thermodynamics. Sohl-Dickstein, Weiss, Maheswaranathan and Ganguli, "Deep Unsupervised Learning using Nonequilibrium Thermodynamics" (ICML 2015), observed that a slow diffusion destroys structure in a way that is easy to describe, and that if every step is small enough its reversal is also approximately Gaussian. A network only has to learn the mean (and possibly the variance) of each small reverse step. They trained it by maximising a variational lower bound on the log-likelihood, as in a **Variational autoencoder (VAE)** with a fixed encoder. The samples were not competitive, and the approach stayed niche for five years.

Two lines of work then met. Song and Ermon (NeurIPS 2019) trained a noise-conditional **score network** with denoising score matching at many noise levels and sampled with annealed Langevin dynamics. Ho, Jain and Abbeel (NeurIPS 2020) re-parameterised the 2015 model so that the network predicts the added noise, and trained it with a plain mean-squared error (see **Denoising diffusion probabilistic models (DDPM)**). The two were the same model seen from different sides. Song et al. (ICLR 2021) then unified both as discretisations of a stochastic differential equation. Within two years diffusion models overtook GANs on ImageNet (Dhariwal and Nichol, NeurIPS 2021) and became the standard for text-to-image, video and audio generation.

## The forward process: data to noise

$$
q(x_t\mid x_{t-1})=\mathcal N\!\big(\sqrt{1-\beta_t}\,x_{t-1},\ \beta_tI\big),\qquad q(x_t\mid x_0)=\mathcal N\!\big(\sqrt{\bar\alpha_t}\,x_0,\ (1-\bar\alpha_t)I\big),\qquad \bar\alpha_t=\prod_{s=1}^{t}(1-\beta_s)
$$

- $\beta_t$: variance added at step $t$ (the **noise schedule**); DDPM uses a linear ramp from $10^{-4}$ to $0.02$ over $T=1000$ steps
- $\sqrt{1-\beta_t}$: shrinks the signal each step so that the total variance stays at 1 for unit-variance data ("variance preserving")
- $\bar\alpha_t$: fraction of the original signal variance left after $t$ steps; $\bar\alpha_T=4.04\times10^{-5}$ for the DDPM schedule
- $q(x_t\mid x_0)$: closed form for any $t$: $x_t=\sqrt{\bar\alpha_t}\,x_0+\sqrt{1-\bar\alpha_t}\,\varepsilon$ with $\varepsilon\sim\mathcal N(0,I)$, so training never has to run the chain

The forward process has no parameters. Its only design choice is the schedule, often summarised by the signal-to-noise ratio $\mathrm{SNR}(t)=\bar\alpha_t/(1-\bar\alpha_t)$. With the linear schedule the SNR falls below 1 at $t=260$, so roughly the last three quarters of the chain are dominated by noise. The cosine schedule of Nichol and Dhariwal (2021) keeps more signal in the middle steps. Because $q(x_T\mid x_0)$ barely depends on $x_0$, the end of the chain is close to $\mathcal N(0,I)$ for any data, and that is the prior we sample from.

## The reverse process: noise to data

Reversing a single step needs $q(x_{t-1}\mid x_t)$, which depends on the unknown data distribution. Conditioned on $x_0$, though, the reverse step is a known Gaussian, $q(x_{t-1}\mid x_t,x_0)$. The model therefore estimates $x_0$, or equivalently the noise $\varepsilon$, from $x_t$ and plugs the estimate in. Everything is driven by one network:

$$
\min_\theta\ \mathbb E_{t,x_0,\varepsilon}\big\|\varepsilon-\varepsilon_\theta(\sqrt{\bar\alpha_t}\,x_0+\sqrt{1-\bar\alpha_t}\,\varepsilon,\ t)\big\|^2,\qquad \hat x_0(x_t)=\frac{x_t-\sqrt{1-\bar\alpha_t}\,\varepsilon_\theta(x_t,t)}{\sqrt{\bar\alpha_t}}
$$

- $t$: uniform on $\{1,\dots,T\}$; one network handles every noise level, receiving $t$ through an embedding
- $\varepsilon_\theta$: noise predictor, usually a U-Net or a transformer over image patches or latents
- $\hat x_0$: the implied denoised estimate; at the optimum it is the posterior mean $\mathbb E[x_0\mid x_t]$

The loss, its weighting and its link to the variational bound are covered in **Diffusion noise-prediction loss**. Two facts matter here. The optimal network is a conditional mean, so no single step has to represent the multimodal data distribution: the chain of small steps does that. And because $\hat x_0$ is a mean, a one-step jump from pure noise lands on the average of the data (for example, the centre of a ring of modes). Sharp samples need many small steps or a good ODE solver.

## Three views of the same model

The trained network is the same object in all three; only the sampler and the bookkeeping differ

| view | what the network outputs | sampler | sibling article |
| --- | --- | --- | --- |
| Latent-variable model (DDPM) | noise ε_θ(x_t, t), i.e. the mean of p_θ(x_{t−1}\|x_t) | ancestral sampling, T stochastic steps | Denoising diffusion probabilistic models (DDPM) |
| Score matching (NCSN) | score s_θ(x, t) ≈ ∇log q_t(x) = −ε_θ/√(1−ᾱ_t) | annealed Langevin dynamics | Score-based generative models |
| SDE / ODE | drift term of a reverse-time SDE or of the probability-flow ODE | any SDE or ODE solver; DDIM is a first-order ODE step | Denoising diffusion implicit models (DDIM) |

$$
dx=-\tfrac12\beta(t)\,x\,dt+\sqrt{\beta(t)}\,dw,\qquad dx=\Big[-\tfrac12\beta(t)\,x-\beta(t)\nabla_x\log q_t(x)\Big]dt+\sqrt{\beta(t)}\,d\bar w,\qquad \frac{dx}{dt}=-\tfrac12\beta(t)\Big[x+\nabla_x\log q_t(x)\Big]
$$

- $\text{first}$: the variance-preserving forward SDE, the $T\to\infty$ limit of DDPM's chain (Song et al., ICLR 2021)
- $\text{second}$: the reverse-time SDE (Anderson, 1982): it needs only the score $\nabla\log q_t$, which the network provides; $d\bar w$ is Brownian motion running backward in time
- $\text{third}$: the **probability-flow ODE**: a deterministic path with the same marginals $q_t$ at every $t$, so $x_T$ maps to $x_0$ by a bijection

The ODE view explains most of what came after. Deterministic samplers such as DDIM integrate it with a few large steps. Encoding real data into noise ("inversion") runs it backward. Exact likelihoods follow from the instantaneous change-of-variables formula, as in **Normalizing flows**. **Flow matching** and **Rectified flow** train the velocity field of such an ODE directly, and **Consistency models** learn to jump along it in one step.

## Guidance: trading diversity for fidelity

A conditional model $\varepsilon_\theta(x_t,t,c)$ (class label, text embedding via **Cross-attention**) samples from $p(x\mid c)$, but in practice samples look better when they are pushed further toward the condition. **Classifier guidance** (Dhariwal and Nichol, 2021) adds the gradient of a classifier $p_\phi(c\mid x_t)$ trained on noisy images. **Classifier-free guidance** (Ho and Salimans, NeurIPS 2021 workshop) needs no classifier. One network is trained both with and without the condition: the label is replaced by a null token with a small probability (0.1 or 0.2 worked best in the paper). At sampling time the two predictions are extrapolated:

$$
\hat\varepsilon=\varepsilon_\theta(x_t,t)-s\sqrt{1-\bar\alpha_t}\,\nabla_{x_t}\log p_\phi(c\mid x_t),\qquad \tilde\varepsilon=(1+w)\,\varepsilon_\theta(x_t,t,c)-w\,\varepsilon_\theta(x_t,t,\varnothing)
$$

- $s$: classifier-guidance scale; with an unconditional $\varepsilon_\theta$ the sampler targets $p(x)\,p_\phi(c\mid x)^{s}$ up to normalisation (Dhariwal and Nichol also guide conditional models)
- $w$: classifier-free guidance weight; $w=0$ is the plain conditional model. Many libraries write the same rule as $\varepsilon_\varnothing+g(\varepsilon_c-\varepsilon_\varnothing)$ with $g=1+w$
- $\varnothing$: the null condition; the difference $\varepsilon_c-\varepsilon_\varnothing$ acts as an implicit classifier gradient, $-\sqrt{1-\bar\alpha_t}\,\nabla\log p(c\mid x_t)$

Guidance raises sample quality and agreement with the prompt but lowers diversity, and at large $w$ it oversaturates images. Classifier-free guidance doubles the number of network evaluations per step (conditional and unconditional), usually done as one batch of two. Almost every text-to-image system, including **Latent diffusion models** and **Stable Diffusion**, uses it.

**Milestones**

- **2015:** Sohl-Dickstein et al. (ICML): diffusion probabilistic models trained by a variational bound
- **2019:** Song & Ermon (NeurIPS): noise-conditional score networks, annealed Langevin sampling
- **2020:** Ho, Jain & Abbeel (NeurIPS): DDPM, ε-prediction and $L_{\text{simple}}$; CIFAR-10 FID 3.17
- **2021:** Song et al. (ICLR): SDE framework and probability-flow ODE. Song, Meng & Ermon (ICLR): DDIM, 10–50× faster sampling. Nichol & Dhariwal (ICML): cosine schedule, learned variances
- **2021:** Dhariwal & Nichol (NeurIPS), "Diffusion Models Beat GANs on Image Synthesis": classifier guidance, ImageNet 256×256 FID 4.59 against 6.95 for BigGAN-deep
- **2022:** Classifier-free guidance; latent diffusion (Rombach et al., CVPR), which runs the process in an autoencoder's latent space

> Figure: Eight Gaussian modes (std 0.08) on a unit ring, alternating between class A (blue) and class B (amber). Because the data is a Gaussian mixture, every noised marginal is one too, so the optimal noise predictor ε*(x, t) is computed exactly: it plays a perfectly trained network. Forward: drag t and the 240 points blur into N(0, I). At t = 250, ᾱ = 0.524 and SNR = 1.10. At t = 400 the per-mode std is already 0.898, and at t = 1000 ᾱ = 4.0 × 10⁻⁵. Reverse: the same 240 noise vectors x_T are denoised with S evenly spaced steps. The default (stochastic, η = 1, S = 100) puts 99.2% of samples within 3σ of a mode, spread over all eight modes (24 to 39 per mode). With S = 10 only 63.3% land on a mode, with S = 20 90.0%, and with S = 50 100%. The deterministic ODE sampler (η = 0) sends every x_T to the same mode whatever S is (62.1% on-mode at S = 10, 98.8% at S = 1000). Choose class A and all samples land on class-A modes (100%). Raising the classifier-free weight w pulls them tighter: under the ODE, the mean distance to the nearest mode falls from 1.03σ at w = 0 to 0.75σ at w = 2 (about 1.25σ for the true distribution). Guidance doubles the network calls per step (NFE 200 at S = 100).

## Worked example: the schedule and one guided prediction

DDPM linear schedule (β from 10⁻⁴ to 0.02, T = 1000), computed in Node; the last column is the std of one ring mode (data std 0.08) after noising

| t | β_t | ᾱ_t | √ᾱ_t | SNR | mode std at t |
| --- | --- | --- | --- | --- | --- |
| 1 | 0.00010 | 0.9999 | 0.9999 | 9999 | 0.081 |
| 100 | 0.00207 | 0.8970 | 0.9471 | 8.71 | 0.330 |
| 250 | 0.00506 | 0.5241 | 0.7239 | 1.10 | 0.692 |
| 500 | 0.01004 | 0.0786 | 0.2803 | 0.0853 | 0.960 |
| 750 | 0.01502 | 0.00335 | 0.0579 | 0.00336 | 0.998 |
| 1000 | 0.02000 | 4.04 × 10⁻⁵ | 0.00635 | 4.04 × 10⁻⁵ | 1.000 |

**Classifier-free guidance at x_t = (0.6, 0.4), t = 150, on the ring data (exact ε*, class A)**

1. At $t=150$, $\bar\alpha_t=0.7883$, so $\sqrt{\bar\alpha_t}=0.8879$ and $\sqrt{1-\bar\alpha_t}=0.4601$. The point sits between a class-A mode at $(1,0)$ and a class-B mode at $(0.707,0.707)$.
2. Unconditional prediction: $\varepsilon_\varnothing=(0.0097,\,0.0046)$, so $\hat x_0=(x_t-0.4601\,\varepsilon)/0.8879=(0.671,\,0.448)$, about halfway between the two modes.
3. Class-A prediction: $\varepsilon_c=(-0.0037,\,0.3034)$, giving $\hat x_0=(0.678,\,0.293)$. The condition has moved the estimate toward the class-A mode at $(1,0)$.
4. Guidance with $w=1$: $\tilde\varepsilon=2\varepsilon_c-\varepsilon_\varnothing=(-0.0171,\,0.6022)$ and $\hat x_0=(0.685,\,0.139)$. With $w=3$: $\tilde\varepsilon=4\varepsilon_c-3\varepsilon_\varnothing=(-0.0440,\,1.1998)$ and $\hat x_0=(0.699,\,-0.171)$, which overshoots past the mode's $y=0$. Large $w$ extrapolates beyond anything the conditional model predicts.

```python
import math, torch, torch.nn as nn

T = 1000
betas = torch.linspace(1e-4, 0.02, T)                  # DDPM linear schedule
abar = torch.cumprod(1 - betas, 0)                     # abar[t-1] = ᾱ_t

def data(n):                                           # ring of 8 Gaussians; class = component parity
    k = torch.randint(0, 8, (n,))
    ang = 2 * math.pi * k / 8
    x0 = torch.stack([ang.cos(), ang.sin()], 1) + 0.08 * torch.randn(n, 2)
    return x0, k % 2

class EpsNet(nn.Module):                               # ε_θ(x_t, t, c); c = 2 means "no condition"
    def __init__(self, h=128):
        super().__init__()
        self.c = nn.Embedding(3, h)
        self.inp = nn.Linear(2 + 1, h)
        self.mlp = nn.Sequential(nn.SiLU(), nn.Linear(h, h), nn.SiLU(), nn.Linear(h, h), nn.SiLU(), nn.Linear(h, 2))
    def forward(self, x, t, c):
        return self.mlp(self.inp(torch.cat([x, (t.float() / T).unsqueeze(1)], 1)) + self.c(c))

net = EpsNet()
opt = torch.optim.Adam(net.parameters(), lr=1e-3)
for it in range(3000):
    x0, c = data(512)
    t = torch.randint(1, T + 1, (512,))
    eps = torch.randn_like(x0)
    a = abar[t - 1].unsqueeze(1)
    xt = a.sqrt() * x0 + (1 - a).sqrt() * eps          # forward process in one step
    c = torch.where(torch.rand(512) < 0.1, torch.full_like(c, 2), c)   # drop the label 10% of the time
    loss = ((net(xt, t, c) - eps) ** 2).mean()         # L_simple
    opt.zero_grad(); loss.backward(); opt.step()

@torch.no_grad()
def sample(n, cls, w=0.0, S=50, eta=0.0):
    """Generalised DDIM sampler over S evenly spaced steps; eta = 1 is DDPM-like, eta = 0 the ODE."""
    taus = torch.linspace(0, T, S + 1).round().long()  # τ_0 = 0, ..., τ_S = T
    x = torch.randn(n, 2)
    c = torch.full((n,), cls)
    u = torch.full((n,), 2)
    for i in range(S, 0, -1):
        t, s = taus[i].item(), taus[i - 1].item()
        at = abar[t - 1]
        as_ = abar[s - 1] if s > 0 else torch.tensor(1.0)
        tt = torch.full((n,), t)
        e = net(x, tt, c)
        if w > 0:                                      # classifier-free guidance
            e = (1 + w) * e - w * net(x, tt, u)
        x0 = (x - (1 - at).sqrt() * e) / at.sqrt()
        sig = eta * ((1 - as_) / (1 - at) * (1 - at / as_)).sqrt()
        x = as_.sqrt() * x0 + (1 - as_ - sig**2).sqrt() * e + sig * torch.randn_like(x)
    return x

torch.manual_seed(0)
xs = sample(1000, cls=0, w=2.0, S=50)
ang = torch.atan2(xs[:, 1], xs[:, 0])
k = torch.round(ang / (2 * math.pi / 8)).long() % 8
print(xs.shape)                                         # torch.Size([1000, 2])
print(f"{abar[-1].item():.2e}")                         # 4.04e-05  (ᾱ_T: x_T is almost pure noise)
print("share in class 0:", (k % 2 == 0).float().mean().item())   # close to 1.0 after training
```

A complete class-conditional diffusion model on 2-D data: L_simple training with label dropout, then a generalised DDIM/DDPM sampler with classifier-free guidance (runs on CPU in well under a minute)

> **Counting the cost of a sample:** Compare samplers by **network function evaluations (NFE)**, not by "steps". DDPM's ancestral sampler uses $T=1000$ evaluations per sample, where a GAN generator uses one. Classifier-free guidance doubles the count, and higher-order ODE solvers can use several evaluations per step. DDIM brings the count to 10–50, DPM-Solver (Lu et al., NeurIPS 2022) to about 10–20, progressive distillation (Salimans and Ho, ICLR 2022) to as few as 4, and **Consistency models** to 1–2. Speed and quality also depend on the schedule: in the figure, 10 uniform steps leave over a third of the samples between modes.

> **Why many small Gaussian steps work:** Any single reverse step is close to Gaussian, so the network only regresses a mean, a stable least-squares problem with no adversary and no mode collapse. The hard part, the multimodal shape of the data, is split across the steps. Early steps decide which mode a sample goes to; late steps add fine detail. That is why the deterministic sampler in the figure picks the same mode for a given $x_T$ at every step count, and only the precision changes.

> **Diffusion versus the other generative families:** Like a **Variational autoencoder (VAE)**, a diffusion model is trained on a variational bound, but its encoder is fixed and its latents have the same size as the data. Like **Normalizing flows**, its ODE view gives a bijection and exact likelihoods, without any architectural constraint on the network. Unlike a **Generative adversarial network (GAN)**, training is plain regression. The price is iterative sampling.

**Strengths**

- Stable training: a mean-squared-error regression with no adversarial game
- Covers all modes of the data; diversity is high and controllable with guidance
- One network supports stochastic and deterministic samplers, inversion, inpainting and likelihood evaluation
- Scales to text-to-image, video and audio, usually in an autoencoder latent space

**Limitations**

- Sampling is iterative: tens to a thousand network calls instead of one
- Guidance improves fidelity at the cost of diversity, and oversaturates at large weights
- Training loss is a poor progress signal; quality is judged with samples or FID
- Pixel-space models at high resolution are expensive; latent diffusion is the usual fix

**Question:** A trained diffusion model maps x_T straight to its predicted clean sample x̂₀ in a single step. On data made of eight separated modes, what does this produce?

- A. A sample on one of the eight modes, chosen at random
- B. Points near the overall mean of the data, between the modes
- C. Exact samples, but only from the most likely mode
- D. Pure noise, because the network has never seen t = T

**Answer:** B. Points near the overall mean of the data, between the modes — The optimal network gives $\hat x_0=\mathbb E[x_0\mid x_t]$, a posterior **mean**. At $t=T$ the noise carries almost no information about $x_0$ ($\bar\alpha_T=4\times10^{-5}$), so the mean is close to the mean of the whole data distribution: the centre of the ring. The chain of small steps, or an accurate ODE solver, is what turns means into sharp samples.

**Question:** With classifier-free guidance weight w, which prediction does the sampler use?

- A. $\varepsilon_\theta(x_t,t,c)+w\,\varepsilon_\theta(x_t,t,\varnothing)$
- B. $w\,\varepsilon_\theta(x_t,t,c)$
- C. $(1+w)\,\varepsilon_\theta(x_t,t,c)-w\,\varepsilon_\theta(x_t,t,\varnothing)$
- D. $\varepsilon_\theta(x_t,t,c)-w\nabla_{x}\log p_\phi(c\mid x_t)$, using a separate classifier

**Answer:** C. $(1+w)\,\varepsilon_\theta(x_t,t,c)-w\,\varepsilon_\theta(x_t,t,\varnothing)$ — It extrapolates from the unconditional toward the conditional prediction: $\varepsilon_\varnothing+(1+w)(\varepsilon_c-\varepsilon_\varnothing)$. Both predictions come from the same network, trained with the condition dropped at random. The last option is classifier guidance, which needs a separate noise-aware classifier (and a $\sqrt{1-\bar\alpha_t}$ factor).

**Key points**

- Forward: $q(x_t\mid x_0)=\mathcal N(\sqrt{\bar\alpha_t}\,x_0,(1-\bar\alpha_t)I)$ with $\bar\alpha_t=\prod_s(1-\beta_s)$; DDPM uses linear $\beta$ from $10^{-4}$ to $0.02$, $T=1000$.
- One network $\varepsilon_\theta(x_t,t)$ predicts the noise; it is also the score, $\nabla\log q_t(x)=-\varepsilon_\theta/\sqrt{1-\bar\alpha_t}$, and the drift of the probability-flow ODE.
- Three views: DDPM (ancestral sampling), score matching (Langevin), SDE/ODE (any solver; DDIM is a first-order ODE step).
- Classifier-free guidance: $\tilde\varepsilon=(1+w)\varepsilon_\theta(x_t,t,c)-w\varepsilon_\theta(x_t,t,\varnothing)$; better fidelity, less diversity, twice the NFE.
- Cost: 1000 NFE for DDPM, 10–50 for DDIM, fewer with higher-order solvers or distillation, against 1 for a GAN.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
