Definition
The diffusion noise-prediction loss (DDPM's ; Ho, Jain & Abbeel, 2020) trains a network to recover the Gaussian noise that was mixed into a clean sample: , with uniform on and . Once trained, the network generates data by removing predicted noise step by step, starting from pure noise.
Diffusion models are the dominant image, video and audio generators, and this is the loss almost all of them are trained with. It is just mean squared error. There is no discriminator and no adversarial game, the target is always a known Gaussian sample, and every training example is a fresh random pair . The subtlety is in the choices around it: how noise levels are scheduled, whether the network predicts the noise, the clean sample or a "velocity", and how the noise levels are weighted.
Sohl-Dickstein et al. (2015) introduced diffusion probabilistic models, trained with a variational bound. Song & Ermon (2019) independently trained noise-conditional score networks with denoising score matching. Ho, Jain & Abbeel (2020) connected the two. They showed that a particular parameterization of the variational bound is a weighted noise-prediction MSE, and that dropping the weights () gave better samples. Nichol & Dhariwal (2021) introduced the cosine schedule, Salimans & Ho (2022) the v-prediction target, Kingma et al. (2021) the SNR view of the loss, and Hang et al. (2023) Min-SNR-γ weighting.
Forward process and the loss
- noise schedule. DDPM: linear from to over steps
- fraction of signal variance left at step ; decreases from to
- signal-to-noise ratio of ; the one number that describes a noise level
Because the noised sample has a closed form, training never runs the chain. Pick a random , draw , form in one step, and regress:
- the network (usually a U-Net or transformer), conditioned on through an embedding
- the exact noise used to build x_t: the regression target
At a glance
- Loss
- MSE between true and predicted noise, averaged over random noise levels
- Optimal model
- , equivalently the posterior mean
- Relation to ELBO
- the ELBO is the same MSE with weight per step; sets it to 1
- Relation to score
- Targets
- (DDPM), , or ; all are rescalings of each other
- Floor
- never 0 even for a perfect model: the noise cannot be fully recovered from x_t
Why predicting noise works: denoising and score matching
The MSE regression target is , so the best possible network is the conditional mean . Since is fixed, knowing the expected noise is the same as knowing the expected clean sample. Predicting noise is denoising.
- the network's implied estimate of the clean sample, at the optimum
- the score of the noising kernel. Regressing it is denoising score matching (Vincent, 2011), whose optimum is the true score
So a trained noise predictor is a scaled score function: . It points from back toward regions of high data density. Tweedie's formula, , is the same statement in denoising form. Sampling (DDPM ancestral steps, DDIM, or an ODE/SDE solver) repeatedly uses this score to move pure noise toward data.
One loss, three targets
The network can output any quantity from which and follow, given . The three standard targets differ only by -dependent factors. Write for the clean-sample error:
- from
- "velocity" (Salimans & Ho, 2022): the rate of change of along the circle
So choosing a target is choosing a weighting over noise levels, measured in units of clean-sample error. ε-prediction weights by SNR: at low noise SNR is huge, and a tiny error in is a big error in . At very high noise SNR goes to 0, ε-prediction becomes trivial (), and derived from it is unstable, because it divides by . x₀-prediction is the opposite. v-prediction behaves like ε at low noise and like x₀ at high noise, staying well conditioned at both ends. That is why it is used for distillation and zero-terminal-SNR schedules.
Noise schedules and SNR weighting
The schedule decides how much training time each SNR gets, since is sampled uniformly. The linear schedule drops the SNR fast: from on, and . Nichol & Dhariwal's cosine schedule, with and , keeps through , spending more steps at the informative mid-range noise levels. Kingma et al. (2021) showed that the continuous-time ELBO depends on the schedule only through its endpoints. The schedule acts on what is learned only through the weighting it induces.
- weight on the clean-sample error at noise level
- Min-SNR cap, in Hang et al. (2023). In ε units the loss weight is
Different noise levels are competing tasks: low noise teaches fine detail, high noise teaches global layout. Under plain ε-prediction the very low-noise steps have large, mostly irreducible loss (0.92 at in the figure) and take a large share of the gradient. Min-SNR-γ caps their weight. Hang et al. report about 3.4× faster convergence on ImageNet 256 with it, and the figure shows why it is target-agnostic: after the cap, ε, x₀ and v give the same weighted objective. Other weightings in use include EDM's (Karras et al., 2022) log-normal sampling of noise levels with a preconditioned target, and the ELBO weighting itself, which is preferred when likelihood matters.
Worked example
One training example, ᾱ_t = 0.64 (√ᾱ = 0.8, √(1−ᾱ) = 0.6, SNR = 1.778)
- 1Clean sample , noise : .
- 2The network predicts . Loss , gradient .
- 3Implied clean sample , error . Check: .
- 4Velocity target , implied : .
- 5Min-SNR-5 weight: SNR , so the weight is 1. At a low-noise step with (SNR ) the ε-loss would be multiplied by .
| t | ᾱ_t | SNR | ε floor | x₀ floor | v floor |
|---|---|---|---|---|---|
| 10 | 0.9981 | 526.8 | 0.922 | 0.0018 | 0.924 |
| 100 | 0.8970 | 8.71 | 0.229 | 0.0263 | 0.255 |
| 300 | 0.3964 | 0.657 | 0.385 | 0.586 | 0.971 |
| 500 | 0.0786 | 0.0853 | 0.080 | 0.940 | 1.020 |
| 1000 | 4.0e-5 | 4.0e-5 | 0.000 | 1.022 | 1.022 |
1import torch, torch.nn.functional as F23T = 10004betas = torch.linspace(1e-4, 0.02, T) # DDPM linear schedule5alpha_bar = torch.cumprod(1.0 - betas, dim=0) # alpha_bar[0] = 1 - 1e-4, alpha_bar[-1] = 4.0e-567def diffusion_loss(model, x0, target="eps", min_snr_gamma=None):8 B = x0.shape[0]9 t = torch.randint(0, T, (B,), device=x0.device) # index 0..T-1 is step 1..T10 ab = alpha_bar.to(x0.device)[t].view(B, *[1] * (x0.dim() - 1))11 eps = torch.randn_like(x0)12 xt = ab.sqrt() * x0 + (1 - ab).sqrt() * eps # closed-form q(x_t | x_0)13 y = {"eps": eps, "x0": x0, "v": ab.sqrt() * eps - (1 - ab).sqrt() * x0}[target]14 per_example = F.mse_loss(model(xt, t), y, reduction="none").flatten(1).mean(1)15 if min_snr_gamma is not None:16 snr = (ab / (1 - ab)).flatten()17 scale = {"eps": snr, "x0": torch.ones_like(snr), "v": snr + 1}[target]18 per_example = per_example * snr.clamp(max=min_snr_gamma) / scale19 return per_example.mean() # gamma=None: L_simpleUse when
- +Plain MSE regression: stable, no adversarial game, easy to scale
- +A reweighted ELBO and a denoising score-matching objective, so it has a clear probabilistic meaning
- +Covers all modes: the model learns the full distribution, not a single answer
- +Target and weighting are flexible (ε, x₀, v; uniform, Min-SNR, EDM)
Avoid when
- −Large irreducible floor and high variance make the loss value a poor progress signal
- −Plain ε-prediction over-weights low-noise steps and is unstable near zero SNR
- −The weighting that gives the best samples is not the ELBO, so likelihoods and samples favour different settings
- −Sampling needs many network evaluations (tens to thousands of steps) unless distilled
Check yourself
A network trained with L_simple outputs ε̂ = ε_θ(x_t, t). What is the optimal ε̂ in terms of the data?
Check yourself
Switching the target from ε to x₀ with an unweighted MSE is equivalent to what change in the ε-loss?
Remember
- ✓ with uniform : plain MSE on the noise.
- ✓The optimal noise predictor is : a denoiser and, up to , the score .
- ✓It is the ELBO with its per-step weights dropped; the weights matter for likelihood, not much for samples.
- ✓ε, x₀ and v targets differ by factors SNR and 1+SNR on the clean-sample error, so the target is a weighting choice.
- ✓Schedules (linear, cosine) set how training time is spread over SNR; Min-SNR-γ caps the weight of easy low-noise steps.