Diffusion noise-prediction loss

Corrupt data with Gaussian noise at a random level and train a network to predict the noise with plain MSE. This simple loss is a reweighted ELBO and a form of denoising score matching. The noise schedule, the prediction target and the SNR weighting decide what it emphasizes.

15 min read1020 wordsadvanced0 viewsupdated 2026-10-11

Definition

The diffusion noise-prediction loss (DDPM's LsimpleL_{\text{simple}}; Ho, Jain & Abbeel, 2020) trains a network εθ(xt,t)\varepsilon_\theta(x_t,t) to recover the Gaussian noise that was mixed into a clean sample: Lsimple=Et,x0,ε∥ε−εθ(αˉt x0+1−αˉt ε,  t)∥2L_{\text{simple}}=\mathbb E_{t,x_0,\varepsilon}\big\|\varepsilon-\varepsilon_\theta\big(\sqrt{\bar\alpha_t}\,x_0+\sqrt{1-\bar\alpha_t}\,\varepsilon,\;t\big)\big\|^2, with tt uniform on {1,…,T}\{1,\dots,T\} and ε∼N(0,I)\varepsilon\sim\mathcal N(0,I). Once trained, the network generates data by removing predicted noise step by step, starting from pure noise.

Diffusion models are the dominant image, video and audio generators, and this is the loss almost all of them are trained with. It is just mean squared error. There is no discriminator and no adversarial game, the target is always a known Gaussian sample, and every training example is a fresh random pair (t,ε)(t,\varepsilon). The subtlety is in the choices around it: how noise levels are scheduled, whether the network predicts the noise, the clean sample or a "velocity", and how the noise levels are weighted.

Sohl-Dickstein et al. (2015) introduced diffusion probabilistic models, trained with a variational bound. Song & Ermon (2019) independently trained noise-conditional score networks with denoising score matching. Ho, Jain & Abbeel (2020) connected the two. They showed that a particular parameterization of the variational bound is a weighted noise-prediction MSE, and that dropping the weights (LsimpleL_{\text{simple}}) gave better samples. Nichol & Dhariwal (2021) introduced the cosine schedule, Salimans & Ho (2022) the v-prediction target, Kingma et al. (2021) the SNR view of the loss, and Hang et al. (2023) Min-SNR-γ weighting.

Forward process and the loss

q(xt∣x0)=N(αˉt x0,  (1−αˉt)I),αˉt=∏s=1t(1−βs),SNR(t)=αˉt1−αˉtq(x_t\mid x_0)=\mathcal N\big(\sqrt{\bar\alpha_t}\,x_0,\;(1-\bar\alpha_t)I\big),\qquad \bar\alpha_t=\prod_{s=1}^{t}(1-\beta_s),\qquad \mathrm{SNR}(t)=\frac{\bar\alpha_t}{1-\bar\alpha_t}
βt\beta_t
noise schedule. DDPM: linear from 10−410^{-4} to 0.020.02 over T=1000T=1000 steps
αˉt\bar\alpha_t
fraction of signal variance left at step tt; decreases from ≈1\approx1 to ≈0\approx0
SNR(t)\mathrm{SNR}(t)
signal-to-noise ratio of xtx_t; the one number that describes a noise level

Because the noised sample has a closed form, training never runs the chain. Pick a random tt, draw ε\varepsilon, form xtx_t in one step, and regress:

Lsimple(θ)=Et∼U{1..T}, x0, ε∼N(0,I)[∥ε−εθ(αˉt x0+1−αˉt ε,  t)∥22],∇εθℓ=2 (εθ−ε)L_{\text{simple}}(\theta)=\mathbb E_{t\sim\mathcal U\{1..T\},\,x_0,\,\varepsilon\sim\mathcal N(0,I)}\Big[\big\|\varepsilon-\varepsilon_\theta\big(\sqrt{\bar\alpha_t}\,x_0+\sqrt{1-\bar\alpha_t}\,\varepsilon,\;t\big)\big\|_2^2\Big],\qquad \nabla_{\varepsilon_\theta}\ell=2\,(\varepsilon_\theta-\varepsilon)
εθ(xt,t)\varepsilon_\theta(x_t,t)
the network (usually a U-Net or transformer), conditioned on tt through an embedding
ε\varepsilon
the exact noise used to build x_t: the regression target

At a glance

Loss
MSE between true and predicted noise, averaged over random noise levels
Optimal model
εθ∗(xt,t)=E[ε∣xt]\varepsilon_\theta^*(x_t,t)=\mathbb E[\varepsilon\mid x_t], equivalently the posterior mean E[x0∣xt]\mathbb E[x_0\mid x_t]
Relation to ELBO
the ELBO is the same MSE with weight ∝βt2/(σt2αt(1−αˉt))\propto\beta_t^2/(\sigma_t^2\alpha_t(1-\bar\alpha_t)) per step; LsimpleL_{\text{simple}} sets it to 1
Relation to score
εθ(xt,t)≈−1−αˉt ∇xtlog⁡qt(xt)\varepsilon_\theta(x_t,t)\approx-\sqrt{1-\bar\alpha_t}\,\nabla_{x_t}\log q_t(x_t)
Targets
ε\varepsilon (DDPM), x0x_0, or v=αˉt ε−1−αˉt x0v=\sqrt{\bar\alpha_t}\,\varepsilon-\sqrt{1-\bar\alpha_t}\,x_0; all are rescalings of each other
Floor
never 0 even for a perfect model: the noise cannot be fully recovered from x_t

Why predicting noise works: denoising and score matching

The MSE regression target is ε\varepsilon, so the best possible network is the conditional mean E[ε∣xt]\mathbb E[\varepsilon\mid x_t]. Since xt=αˉtx0+1−αˉtεx_t=\sqrt{\bar\alpha_t}x_0+\sqrt{1-\bar\alpha_t}\varepsilon is fixed, knowing the expected noise is the same as knowing the expected clean sample. Predicting noise is denoising.

x^0(xt)=xt−1−αˉt εθ(xt,t)αˉt,∇xtlog⁡q(xt∣x0)=−ε1−αˉt\hat x_0(x_t)=\frac{x_t-\sqrt{1-\bar\alpha_t}\,\varepsilon_\theta(x_t,t)}{\sqrt{\bar\alpha_t}},\qquad \nabla_{x_t}\log q(x_t\mid x_0)=-\frac{\varepsilon}{\sqrt{1-\bar\alpha_t}}
x^0\hat x_0
the network's implied estimate of the clean sample, E[x0∣xt]\mathbb E[x_0\mid x_t] at the optimum
∇log⁡q(xt∣x0)\nabla\log q(x_t|x_0)
the score of the noising kernel. Regressing it is denoising score matching (Vincent, 2011), whose optimum is the true score ∇log⁡qt(xt)\nabla\log q_t(x_t)

So a trained noise predictor is a scaled score function: εθ(xt,t)≈−1−αˉt ∇xtlog⁡qt(xt)\varepsilon_\theta(x_t,t)\approx-\sqrt{1-\bar\alpha_t}\,\nabla_{x_t}\log q_t(x_t). It points from xtx_t back toward regions of high data density. Tweedie's formula, E[x0∣xt]=(xt+(1−αˉt)∇log⁡qt(xt))/αˉt\mathbb E[x_0\mid x_t]=\big(x_t+(1-\bar\alpha_t)\nabla\log q_t(x_t)\big)/\sqrt{\bar\alpha_t}, is the same statement in denoising form. Sampling (DDPM ancestral steps, DDIM, or an ODE/SDE solver) repeatedly uses this score to move pure noise toward data.

t = 300: x_t = √ᾱ·x₀ + √(1−ᾱ)·ε, ᾱ = 0.3964▲−3−2−10123−303■ q(x_t) ━ optimal ε_θ(x_t) ● target ▲ x₀ │ errorx₀ = −0.864 · ε = −1.211 → x_t = −1.485target ε = −1.211 · best prediction −1.157(ε_θ − ε)² = 0.0029 · ∂ℓ/∂ε_θ = 0.108weight w(t) = 1.00 → weighted loss 0.0029optimal x₀_θ = E[x₀|x_t] = −0.931log₁₀ SNR(t), SNR = ᾱ/(1−ᾱ)−404━ linear ━ cosineperfect-model floor w(t)·mmse0.92002505007501000SNR = 0.657 · floor here 0.3850mean floor over t ~ U{1…1000}: 0.1778uniform weight (L_simple)ε: hard at low noise, easy at high
predictschedule
fig.1-D data with two clusters, ½N(−1, 0.15²) + ½N(1, 0.15²), so the Bayes-optimal denoiser E[x₀|x_t] is exact. Left: the noised density q(x_t) at step t and the best possible prediction of ε, x₀ or v as a function of x_t, with one training example. At the default t = 300 (linear schedule, ᾱ = 0.3964, SNR = 0.657), x₀ = −0.864 and ε = −1.211 give x_t = −1.485. The optimal ε prediction is −1.157, so a perfect network still has loss 0.0029 on this example and gradient 0.108. Right: log SNR for both schedules, and the loss floor a perfect model cannot go below at each t (0.385 at t = 300). Averaged over uniform t, the ε-loss floor is 0.178 for the linear schedule and 0.309 for cosine. The floor for x₀ is 0.724, for v 0.902. With Min-SNR-5 weighting all three become the same objective, with floor 0.136.

One loss, three targets

The network can output any quantity from which x0x_0 and ε\varepsilon follow, given xtx_t. The three standard targets differ only by tt-dependent factors. Write e=∥x0−x^0∥2e=\|x_0-\hat x_0\|^2 for the clean-sample error:

∥ε−ε^∥2=SNR(t)  e,∥x0−x^0∥2=e,∥v−v^∥2=(1+SNR(t)) e,v=αˉt ε−1−αˉt x0\|\varepsilon-\hat\varepsilon\|^2=\mathrm{SNR}(t)\;e,\qquad \|x_0-\hat x_0\|^2=e,\qquad \|v-\hat v\|^2=\big(1+\mathrm{SNR}(t)\big)\,e,\qquad v=\sqrt{\bar\alpha_t}\,\varepsilon-\sqrt{1-\bar\alpha_t}\,x_0
SNR(t) e\mathrm{SNR}(t)\,e
from ε−ε^=−αˉt/(1−αˉt) (x0−x^0)\varepsilon-\hat\varepsilon=-\sqrt{\bar\alpha_t/(1-\bar\alpha_t)}\,(x_0-\hat x_0)
vv
"velocity" (Salimans & Ho, 2022): the rate of change of xtx_t along the circle (αˉ,1−αˉ)(\sqrt{\bar\alpha},\sqrt{1-\bar\alpha})

So choosing a target is choosing a weighting over noise levels, measured in units of clean-sample error. ε-prediction weights ee by SNR: at low noise SNR is huge, and a tiny error in x^0\hat x_0 is a big error in ε^\hat\varepsilon. At very high noise SNR goes to 0, ε-prediction becomes trivial (ε^≈xt/1−αˉt\hat\varepsilon\approx x_t/\sqrt{1-\bar\alpha_t}), and x^0\hat x_0 derived from it is unstable, because it divides by αˉt→0\sqrt{\bar\alpha_t}\to0. x₀-prediction is the opposite. v-prediction behaves like ε at low noise and like x₀ at high noise, staying well conditioned at both ends. That is why it is used for distillation and zero-terminal-SNR schedules.

Noise schedules and SNR weighting

The schedule decides how much training time each SNR gets, since tt is sampled uniformly. The linear schedule drops the SNR fast: SNR<1\mathrm{SNR}<1 from t=260t=260 on, and αˉT=4.0×10−5\bar\alpha_T=4.0\times10^{-5}. Nichol & Dhariwal's cosine schedule, αˉt=f(t)/f(0)\bar\alpha_t=f(t)/f(0) with f(t)=cos⁡2 ⁣(t/T+s1+s⋅π2)f(t)=\cos^2\!\big(\tfrac{t/T+s}{1+s}\cdot\tfrac\pi2\big) and s=0.008s=0.008, keeps SNR≥1\mathrm{SNR}\ge1 through t=496t=496, spending more steps at the informative mid-range noise levels. Kingma et al. (2021) showed that the continuous-time ELBO depends on the schedule only through its endpoints. The schedule acts on what is learned only through the weighting it induces.

Lw=Et,x0,ε[w(t) ∥x0−x^0∥2],wε=SNR,wv=SNR+1,wMin-SNR-γ=min⁡(SNR,γ)\mathcal L_w=\mathbb E_{t,x_0,\varepsilon}\big[w(t)\,\|x_0-\hat x_0\|^2\big],\qquad w_{\varepsilon}=\mathrm{SNR},\quad w_{v}=\mathrm{SNR}+1,\quad w_{\text{Min-SNR-}\gamma}=\min(\mathrm{SNR},\gamma)
w(t)w(t)
weight on the clean-sample error at noise level tt
γ\gamma
Min-SNR cap, γ=5\gamma=5 in Hang et al. (2023). In ε units the loss weight is min⁡(SNR,γ)/SNR\min(\mathrm{SNR},\gamma)/\mathrm{SNR}

Different noise levels are competing tasks: low noise teaches fine detail, high noise teaches global layout. Under plain ε-prediction the very low-noise steps have large, mostly irreducible loss (0.92 at t=10t=10 in the figure) and take a large share of the gradient. Min-SNR-γ caps their weight. Hang et al. report about 3.4× faster convergence on ImageNet 256 with it, and the figure shows why it is target-agnostic: after the cap, ε, x₀ and v give the same weighted objective. Other weightings in use include EDM's (Karras et al., 2022) log-normal sampling of noise levels with a preconditioned target, and the ELBO weighting itself, which is preferred when likelihood matters.

Worked example

One training example, ᾱ_t = 0.64 (√ᾱ = 0.8, √(1−ᾱ) = 0.6, SNR = 1.778)

  1. 1Clean sample x0=0.8x_0=0.8, noise ε=−0.5\varepsilon=-0.5: xt=0.8⋅0.8+0.6⋅(−0.5)=0.34x_t=0.8\cdot0.8+0.6\cdot(-0.5)=0.34.
  2. 2The network predicts ε^=−0.3\hat\varepsilon=-0.3. Loss (ε^−ε)2=0.22=0.04(\hat\varepsilon-\varepsilon)^2=0.2^2=0.04, gradient 2(ε^−ε)=0.42(\hat\varepsilon-\varepsilon)=0.4.
  3. 3Implied clean sample x^0=(0.34−0.6⋅(−0.3))/0.8=0.65\hat x_0=(0.34-0.6\cdot(-0.3))/0.8=0.65, error e=0.152=0.0225e=0.15^2=0.0225. Check: SNR⋅e=1.778⋅0.0225=0.04\mathrm{SNR}\cdot e=1.778\cdot0.0225=0.04.
  4. 4Velocity target v=0.8(−0.5)−0.6(0.8)=−0.88v=0.8(-0.5)-0.6(0.8)=-0.88, implied v^=0.8(−0.3)−0.6(0.65)=−0.63\hat v=0.8(-0.3)-0.6(0.65)=-0.63: (v^−v)2=0.0625=(1+1.778)⋅0.0225(\hat v-v)^2=0.0625=(1+1.778)\cdot0.0225.
  5. 5Min-SNR-5 weight: SNR 1.778<51.778<5, so the weight is 1. At a low-noise step with αˉt=0.96\bar\alpha_t=0.96 (SNR =24=24) the ε-loss would be multiplied by 5/24=0.2085/24=0.208.
tᾱ_tSNRε floorx₀ floorv floor
100.9981526.80.9220.00180.924
1000.89708.710.2290.02630.255
3000.39640.6570.3850.5860.971
5000.07860.08530.0800.9401.020
10004.0e-54.0e-50.0001.0221.022
The figure's loss floor E‖target − optimal prediction‖² per step, linear schedule (computed from the lib by quadrature)
python
1import torch, torch.nn.functional as F
2
3T = 1000
4betas = torch.linspace(1e-4, 0.02, T) # DDPM linear schedule
5alpha_bar = torch.cumprod(1.0 - betas, dim=0) # alpha_bar[0] = 1 - 1e-4, alpha_bar[-1] = 4.0e-5
6
7def diffusion_loss(model, x0, target="eps", min_snr_gamma=None):
8 B = x0.shape[0]
9 t = torch.randint(0, T, (B,), device=x0.device) # index 0..T-1 is step 1..T
10 ab = alpha_bar.to(x0.device)[t].view(B, *[1] * (x0.dim() - 1))
11 eps = torch.randn_like(x0)
12 xt = ab.sqrt() * x0 + (1 - ab).sqrt() * eps # closed-form q(x_t | x_0)
13 y = {"eps": eps, "x0": x0, "v": ab.sqrt() * eps - (1 - ab).sqrt() * x0}[target]
14 per_example = F.mse_loss(model(xt, t), y, reduction="none").flatten(1).mean(1)
15 if min_snr_gamma is not None:
16 snr = (ab / (1 - ab)).flatten()
17 scale = {"eps": snr, "x0": torch.ones_like(snr), "v": snr + 1}[target]
18 per_example = per_example * snr.clamp(max=min_snr_gamma) / scale
19 return per_example.mean() # gamma=None: L_simple
DDPM training loss with ε, x₀ or v targets and optional Min-SNR-γ weighting

Use when

  • +Plain MSE regression: stable, no adversarial game, easy to scale
  • +A reweighted ELBO and a denoising score-matching objective, so it has a clear probabilistic meaning
  • +Covers all modes: the model learns the full distribution, not a single answer
  • +Target and weighting are flexible (ε, x₀, v; uniform, Min-SNR, EDM)

Avoid when

  • −Large irreducible floor and high variance make the loss value a poor progress signal
  • −Plain ε-prediction over-weights low-noise steps and is unstable near zero SNR
  • −The weighting that gives the best samples is not the ELBO, so likelihoods and samples favour different settings
  • −Sampling needs many network evaluations (tens to thousands of steps) unless distilled

Check yourself

A network trained with L_simple outputs ε̂ = ε_θ(x_t, t). What is the optimal ε̂ in terms of the data?

Check yourself

Switching the target from ε to x₀ with an unweighted MSE is equivalent to what change in the ε-loss?

Remember

  • ✓Lsimple=E∥ε−εθ(αˉtx0+1−αˉtε,t)∥2L_{\text{simple}}=\mathbb E\|\varepsilon-\varepsilon_\theta(\sqrt{\bar\alpha_t}x_0+\sqrt{1-\bar\alpha_t}\varepsilon,t)\|^2 with uniform tt: plain MSE on the noise.
  • ✓The optimal noise predictor is E[ε∣xt]\mathbb E[\varepsilon|x_t]: a denoiser and, up to −1−αˉt-\sqrt{1-\bar\alpha_t}, the score ∇log⁡qt\nabla\log q_t.
  • ✓It is the ELBO with its per-step weights dropped; the weights matter for likelihood, not much for samples.
  • ✓ε, x₀ and v targets differ by factors SNR and 1+SNR on the clean-sample error, so the target is a weighting choice.
  • ✓Schedules (linear, cosine) set how training time is spread over SNR; Min-SNR-γ caps the weight of easy low-noise steps.
← back to Deep Learning roadmap