Definition
Denoising diffusion probabilistic model (DDPM) is the diffusion model of Ho, Jain and Abbeel, "Denoising Diffusion Probabilistic Models" (NeurIPS 2020). It is a latent-variable model with a fixed Gaussian forward chain . Each reverse step is a Gaussian whose mean is computed from a network that predicts the noise in , and whose variance is fixed. The network is trained with the unweighted noise-prediction loss . Samples are drawn by ancestral sampling: start at and draw for .
Diffusion probabilistic models had been proposed by Sohl-Dickstein et al. in 2015 (see Diffusion models) but produced weak samples. Jonathan Ho, Ajay Jain and Pieter Abbeel made three choices that turned them into a state-of-the-art image generator. They fixed the reverse variances instead of learning them. They parameterised the reverse mean through a noise prediction , which turns each term of the variational bound into a weighted denoising score-matching loss. And they dropped the weights. On CIFAR-10 this gave an Inception score of 9.46 and FID 3.17, better than the GANs of the time, and LSUN 256×256 samples comparable to ProgressiveGAN. The same paper pointed out the connection to Song and Ermon's score networks and Langevin dynamics.
Forward chain and its posterior
- linear from to with ; small enough that each reverse step is close to Gaussian, large enough that
- an image with pixel values scaled linearly to
- marginal in closed form: ,
The true reverse step is intractable. Conditioned on it is not: by Bayes' rule and the Gaussian forward kernels it is a Gaussian with known mean and variance.
- a weighted average of the clean sample and the current noisy one; for large almost all weight is on (0.9941 against 0.0031 at )
- posterior variance; slightly below , and much smaller only for the first few steps ()
The ε-parameterisation and L_simple
The model is . The variational bound splits into a KL term per step, . Between two Gaussians with the same fixed variance, this KL is a squared distance between the means. Instead of predicting directly, write in terms of and the noise, and let the network predict the noise:
- exactly with substituted for the unknown
- fixed reverse variance, or ; the paper found both gave similar sample quality
- a constant independent of
- the bound with the per-step weight set to 1; this down-weights the small- terms, where denoising is easy, relative to the true bound
- the discrete decoder term of the bound is approximated by the same MSE at
Why the weights matter, and how ε-, - and v-prediction relate, is covered in Diffusion noise-prediction loss. The paper also tried predicting directly. That works only with the true bound and gives worse samples. ε-prediction resembles Langevin dynamics, because is a scaled score: (see Score-based generative models).
The paper's two algorithms
- 1Training (Algorithm 1). Repeat: take from the data, , ; take a gradient step on .
- 2Sampling (Algorithm 2). Draw . For : if , else , and . Return .
- 3Sampling needs sequential network evaluations per batch. Denoising diffusion implicit models (DDIM) reuses the same trained with far fewer steps.
Architecture and setup
Setup in the paper (Ho et al. 2020)
- Schedule
- , linear from to ; or (similar results)
- Network
- U-Net based on an unmasked PixelCNN++: Wide ResNet blocks, group normalisation, self-attention at the 16×16 resolution, and the Transformer's sinusoidal embedding of added in every residual block; parameters shared across
- Size
- 35.7M parameters for CIFAR-10, 114M for LSUN and CelebA-HQ (256×256), about 256M for a larger LSUN Bedroom model
- Optimisation
- Adam, learning rate ( at 256×256), EMA of weights with decay 0.9999; dropout 0.1 and random horizontal flips on CIFAR-10; 800K steps at batch 128 for CIFAR-10 on a TPU v3-8
- Loss
- ; the true variational bound gives better likelihoods but worse samples
| parameterisation and objective | IS | FID | NLL (bits/dim) |
|---|---|---|---|
| predict μ̃, true bound L, fixed isotropic Σ | 8.06 | 13.22 | |
| predict ε, true bound L, fixed isotropic Σ | 7.67 | 13.51 | ≤ 3.70 |
| predict ε, L_simple | 9.46 | 3.17 | ≤ 3.75 |
On LSUN 256×256 the paper reports FID 7.89 for Church and 6.36 for Bedroom (4.90 with the larger model). The likelihoods are worse than those of autoregressive models. The authors explain this through a rate–distortion analysis: most of the lossless codelength describes imperceptible image details, which deliberately down-weights. Running the chain as progressive decoding shows large-scale features appearing first and fine details last.
The cosine schedule (Nichol & Dhariwal, 2021)
- makes fall slowly at both ends and roughly linearly in the middle
- small offset so that is not too small near
- clip that prevents a singularity at , where
In "Improved Denoising Diffusion Probabilistic Models" (ICML 2021), Alex Nichol and Prafulla Dhariwal noticed that with the linear schedule the last part of the forward chain is almost pure noise and contributes little. The SNR drops below 1 at for the linear schedule and at for the cosine one. They also learned the reverse variance as an interpolation , trained with a hybrid loss with . That improved log-likelihoods and allowed sampling with far fewer steps.
Worked example: one reverse step
| t | β_t | ᾱ_t | β̃_t | 1/√α_t | β_t/√(1−ᾱ_t) | cx₀ | cx_t |
|---|---|---|---|---|---|---|---|
| 2 | 1.199 × 10⁻⁴ | 0.9998 | 5.45 × 10⁻⁵ | 1.0001 | 0.00809 | 0.5453 | 0.4547 |
| 10 | 2.793 × 10⁻⁴ | 0.9981 | 2.382 × 10⁻⁴ | 1.0001 | 0.00642 | 0.1473 | 0.8527 |
| 100 | 0.002072 | 0.8970 | 0.002035 | 1.0010 | 0.00646 | 0.01908 | 0.9809 |
| 500 | 0.01004 | 0.07859 | 0.01003 | 1.0051 | 0.01046 | 0.003070 | 0.9941 |
| 1000 | 0.02000 | 4.04 × 10⁻⁵ | 0.02000 | 1.0102 | 0.02000 | 1.28 × 10⁻⁴ | 0.9899 |
x_t = 0.5 at t = 100, on the figure's mixture, with the exact ε*
- 1, so and . The exact noise prediction is .
- 2Denoised estimate: , pulled toward the larger mode at .
- 3Posterior mean at : .
- 4Algorithm 2's mean: . The two agree, as the derivation says.
- 5Add noise: or , so . The step moves the mean by only 0.0058; the noise term is eight times larger. That is why 1000 steps are needed.
1import math, torch, torch.nn as nn, torch.nn.functional as F23class DDPM:4 def __init__(self, T=1000, schedule="linear"):5 self.T = T6 if schedule == "linear":7 beta = torch.linspace(1e-4, 0.02, T, dtype=torch.float64)8 else: # cosine (Nichol & Dhariwal 2021), s = 0.0089 s = 0.00810 f = lambda t: torch.cos((t / T + s) / (1 + s) * math.pi / 2) ** 211 t = torch.arange(T + 1, dtype=torch.float64)12 beta = (1 - f(t[1:]) / f(t[:-1])).clamp(max=0.999)13 self.beta = beta.float() # index i ↔ step t = i + 114 self.alpha = 1 - self.beta15 self.abar = torch.cumprod(beta.new_ones(T) - beta, 0).float()16 self.abar_prev = F.pad(self.abar[:-1], (1, 0), value=1.0)17 self.beta_tilde = self.beta * (1 - self.abar_prev) / (1 - self.abar)1819 def q_sample(self, x0, t, eps): # x_t = √ᾱ_t x0 + √(1-ᾱ_t) ε20 a = self.abar[t - 1].view(-1, 1, 1, 1)21 return a.sqrt() * x0 + (1 - a).sqrt() * eps2223 def loss(self, model, x0): # L_simple24 t = torch.randint(1, self.T + 1, (x0.size(0),))25 eps = torch.randn_like(x0)26 return F.mse_loss(model(self.q_sample(x0, t, eps), t), eps)2728 def posterior_mean(self, x0, xt, t): # μ̃_t(x_t, x0) of q(x_{t-1} | x_t, x0)29 i = t - 130 c0 = self.abar_prev[i].sqrt() * self.beta[i] / (1 - self.abar[i])31 ct = self.alpha[i].sqrt() * (1 - self.abar_prev[i]) / (1 - self.abar[i])32 return c0 * x0 + ct * xt3334 @torch.no_grad()35 def sample(self, model, shape, var="beta"): # Algorithm 2 (ancestral sampling)36 x = torch.randn(shape)37 for t in range(self.T, 0, -1):38 i = t - 139 eps = model(x, torch.full((shape[0],), t))40 mean = (x - self.beta[i] / (1 - self.abar[i]).sqrt() * eps) / self.alpha[i].sqrt()41 sigma = (self.beta[i] if var == "beta" else self.beta_tilde[i]).sqrt()42 x = mean + sigma * torch.randn_like(x) if t > 1 else mean43 return x4445def timestep_embedding(t, dim): # sinusoidal, as in the Transformer46 half = dim // 247 freqs = torch.exp(-math.log(10000) * torch.arange(half) / half)48 a = t.float()[:, None] * freqs[None]49 return torch.cat([a.sin(), a.cos()], 1)5051class TinyDenoiser(nn.Module): # stand-in for the U-Net: conv blocks + t embedding52 def __init__(self, ch=3, h=32):53 super().__init__()54 self.temb = nn.Sequential(nn.Linear(h, h), nn.SiLU(), nn.Linear(h, h))55 self.inp = nn.Conv2d(ch, h, 3, padding=1)56 self.mid = nn.Sequential(nn.GroupNorm(8, h), nn.SiLU(), nn.Conv2d(h, h, 3, padding=1))57 self.out = nn.Conv2d(h, ch, 3, padding=1)58 self.h = h59 def forward(self, x, t):60 e = self.temb(timestep_embedding(t, self.h))[:, :, None, None]61 return self.out(self.mid(self.inp(x) + e))6263d = DDPM()64model = TinyDenoiser()65x0 = torch.rand(8, 3, 32, 32) * 2 - 1 # images scaled to [-1, 1]66loss = d.loss(model, x0); loss.backward()67print(loss.shape, f"{d.abar[-1].item():.2e}") # torch.Size([]) 4.04e-056869# Algorithm 2's mean is the posterior mean evaluated at x̂0 = (x_t - √(1-ᾱ_t) ε)/√ᾱ_t70t = torch.tensor([100]); xt = torch.randn(1, 3, 32, 32); eps = torch.randn_like(xt)71a = d.abar[t - 1]72x0_hat = (xt - (1 - a).sqrt() * eps) / a.sqrt()73alg2 = (xt - d.beta[t - 1] / (1 - a).sqrt() * eps) / d.alpha[t - 1].sqrt()74print(torch.allclose(alg2, d.posterior_mean(x0_hat, xt, t), atol=1e-5)) # True7576x = d.sample(model.eval(), (2, 3, 32, 32)) # 1000 network calls77print(x.shape) # torch.Size([2, 3, 32, 32])78c = DDPM(schedule="cosine")79print(f"{c.abar[499].item():.4f} {d.abar[499].item():.4f}") # 0.4938 0.0786 (ᾱ_500: cosine keeps more signal)Use when
- +Simple, stable training: one MSE, no adversary, no learned variances needed
- +High sample quality and full mode coverage (CIFAR-10 FID 3.17 in 2020)
- +Principled: a variational bound gives log-likelihood estimates and connects to score matching
- +The trained ε_θ is reused by faster samplers (DDIM, ODE solvers) without retraining
Avoid when
- −Sampling is slow: 1000 sequential network evaluations per sample
- −Likelihoods are worse than those of autoregressive models when trained with L_simple
- −Fixed variances and the linear schedule are suboptimal for few-step sampling
- −Pixel-space U-Nets are expensive at high resolution
Check yourself
In DDPM, the network output ε_θ(x_t, t) enters the reverse step through which mean?
Check yourself
Ho et al. trained with L_simple rather than the true variational bound. What did they observe?
Remember
- ✓DDPM: , , linear from to ; train with .
- ✓Sampling: with or ; 1000 network calls.
- ✓The reverse mean equals the posterior mean at the network's denoised estimate .
- ✓CIFAR-10: FID 3.17 and IS 9.46 with L_simple, against FID 13.51 with the true bound; U-Net with 35.7M parameters.
- ✓Cosine schedule (Nichol & Dhariwal 2021): , , , keeps SNR ≥ 1 for about half the chain.