# Diffusion noise-prediction loss

> Corrupt data with Gaussian noise at a random level and train a network to predict the noise with plain MSE. This simple loss is a reweighted ELBO and a form of denoising score matching. The noise schedule, the prediction target and the SNR weighting decide what it emphasizes.

Source: https://www.mlfoundry.in/learn/diffusion-noise-prediction-loss  
Section: Deep Learning › Loss Functions  
Level: advanced · 15 min read · updated 2026-10-11

**The diffusion noise-prediction loss:** (DDPM's $L_{\text{simple}}$; Ho, Jain & Abbeel, 2020) trains a network $\varepsilon_\theta(x_t,t)$ to recover the Gaussian noise that was mixed into a clean sample: $L_{\text{simple}}=\mathbb E_{t,x_0,\varepsilon}\big\|\varepsilon-\varepsilon_\theta\big(\sqrt{\bar\alpha_t}\,x_0+\sqrt{1-\bar\alpha_t}\,\varepsilon,\;t\big)\big\|^2$, with $t$ uniform on $\{1,\dots,T\}$ and $\varepsilon\sim\mathcal N(0,I)$. Once trained, the network generates data by removing predicted noise step by step, starting from pure noise.

Diffusion models are the dominant image, video and audio generators, and this is the loss almost all of them are trained with. It is just mean squared error. There is no discriminator and no adversarial game, the target is always a known Gaussian sample, and every training example is a fresh random pair $(t,\varepsilon)$. The subtlety is in the choices around it: how noise levels are scheduled, whether the network predicts the noise, the clean sample or a "velocity", and how the noise levels are weighted.

Sohl-Dickstein et al. (2015) introduced diffusion probabilistic models, trained with a variational bound. Song & Ermon (2019) independently trained noise-conditional score networks with denoising score matching. Ho, Jain & Abbeel (2020) connected the two. They showed that a particular parameterization of the variational bound is a weighted noise-prediction MSE, and that **dropping the weights** ($L_{\text{simple}}$) gave better samples. Nichol & Dhariwal (2021) introduced the cosine schedule, Salimans & Ho (2022) the v-prediction target, Kingma et al. (2021) the SNR view of the loss, and Hang et al. (2023) Min-SNR-γ weighting.

## Forward process and the loss

$$
q(x_t\mid x_0)=\mathcal N\big(\sqrt{\bar\alpha_t}\,x_0,\;(1-\bar\alpha_t)I\big),\qquad \bar\alpha_t=\prod_{s=1}^{t}(1-\beta_s),\qquad \mathrm{SNR}(t)=\frac{\bar\alpha_t}{1-\bar\alpha_t}
$$

- $\beta_t$: noise schedule. DDPM: linear from $10^{-4}$ to $0.02$ over $T=1000$ steps
- $\bar\alpha_t$: fraction of signal variance left at step $t$; decreases from $\approx1$ to $\approx0$
- $\mathrm{SNR}(t)$: signal-to-noise ratio of $x_t$; the one number that describes a noise level

Because the noised sample has a closed form, training never runs the chain. Pick a random $t$, draw $\varepsilon$, form $x_t$ in one step, and regress:

$$
L_{\text{simple}}(\theta)=\mathbb E_{t\sim\mathcal U\{1..T\},\,x_0,\,\varepsilon\sim\mathcal N(0,I)}\Big[\big\|\varepsilon-\varepsilon_\theta\big(\sqrt{\bar\alpha_t}\,x_0+\sqrt{1-\bar\alpha_t}\,\varepsilon,\;t\big)\big\|_2^2\Big],\qquad \nabla_{\varepsilon_\theta}\ell=2\,(\varepsilon_\theta-\varepsilon)
$$

- $\varepsilon_\theta(x_t,t)$: the network (usually a U-Net or transformer), conditioned on $t$ through an embedding
- $\varepsilon$: the exact noise used to build x_t: the regression target

**At a glance**

- **Loss:** MSE between true and predicted noise, averaged over random noise levels
- **Optimal model:** $\varepsilon_\theta^*(x_t,t)=\mathbb E[\varepsilon\mid x_t]$, equivalently the posterior mean $\mathbb E[x_0\mid x_t]$
- **Relation to ELBO:** the ELBO is the same MSE with weight $\propto\beta_t^2/(\sigma_t^2\alpha_t(1-\bar\alpha_t))$ per step; $L_{\text{simple}}$ sets it to 1
- **Relation to score:** $\varepsilon_\theta(x_t,t)\approx-\sqrt{1-\bar\alpha_t}\,\nabla_{x_t}\log q_t(x_t)$
- **Targets:** $\varepsilon$ (DDPM), $x_0$, or $v=\sqrt{\bar\alpha_t}\,\varepsilon-\sqrt{1-\bar\alpha_t}\,x_0$; all are rescalings of each other
- **Floor:** never 0 even for a perfect model: the noise cannot be fully recovered from x_t

## Why predicting noise works: denoising and score matching

The MSE regression target is $\varepsilon$, so the best possible network is the conditional mean $\mathbb E[\varepsilon\mid x_t]$. Since $x_t=\sqrt{\bar\alpha_t}x_0+\sqrt{1-\bar\alpha_t}\varepsilon$ is fixed, knowing the expected noise is the same as knowing the expected clean sample. **Predicting noise is denoising.**

$$
\hat x_0(x_t)=\frac{x_t-\sqrt{1-\bar\alpha_t}\,\varepsilon_\theta(x_t,t)}{\sqrt{\bar\alpha_t}},\qquad \nabla_{x_t}\log q(x_t\mid x_0)=-\frac{\varepsilon}{\sqrt{1-\bar\alpha_t}}
$$

- $\hat x_0$: the network's implied estimate of the clean sample, $\mathbb E[x_0\mid x_t]$ at the optimum
- $\nabla\log q(x_t|x_0)$: the score of the noising kernel. Regressing it is denoising score matching (Vincent, 2011), whose optimum is the true score $\nabla\log q_t(x_t)$

So a trained noise predictor is a scaled **score function**: $\varepsilon_\theta(x_t,t)\approx-\sqrt{1-\bar\alpha_t}\,\nabla_{x_t}\log q_t(x_t)$. It points from $x_t$ back toward regions of high data density. Tweedie's formula, $\mathbb E[x_0\mid x_t]=\big(x_t+(1-\bar\alpha_t)\nabla\log q_t(x_t)\big)/\sqrt{\bar\alpha_t}$, is the same statement in denoising form. Sampling (DDPM ancestral steps, DDIM, or an ODE/SDE solver) repeatedly uses this score to move pure noise toward data.

> Figure: 1-D data with two clusters, ½N(−1, 0.15²) + ½N(1, 0.15²), so the Bayes-optimal denoiser E[x₀|x_t] is exact. Left: the noised density q(x_t) at step t and the best possible prediction of ε, x₀ or v as a function of x_t, with one training example. At the default t = 300 (linear schedule, ᾱ = 0.3964, SNR = 0.657), x₀ = −0.864 and ε = −1.211 give x_t = −1.485. The optimal ε prediction is −1.157, so a perfect network still has loss 0.0029 on this example and gradient 0.108. Right: log SNR for both schedules, and the loss floor a perfect model cannot go below at each t (0.385 at t = 300). Averaged over uniform t, the ε-loss floor is 0.178 for the linear schedule and 0.309 for cosine. The floor for x₀ is 0.724, for v 0.902. With Min-SNR-5 weighting all three become the same objective, with floor 0.136.

## One loss, three targets

The network can output any quantity from which $x_0$ and $\varepsilon$ follow, given $x_t$. The three standard targets differ only by $t$-dependent factors. Write $e=\|x_0-\hat x_0\|^2$ for the clean-sample error:

$$
\|\varepsilon-\hat\varepsilon\|^2=\mathrm{SNR}(t)\;e,\qquad \|x_0-\hat x_0\|^2=e,\qquad \|v-\hat v\|^2=\big(1+\mathrm{SNR}(t)\big)\,e,\qquad v=\sqrt{\bar\alpha_t}\,\varepsilon-\sqrt{1-\bar\alpha_t}\,x_0
$$

- $\mathrm{SNR}(t)\,e$: from $\varepsilon-\hat\varepsilon=-\sqrt{\bar\alpha_t/(1-\bar\alpha_t)}\,(x_0-\hat x_0)$
- $v$: "velocity" (Salimans & Ho, 2022): the rate of change of $x_t$ along the circle $(\sqrt{\bar\alpha},\sqrt{1-\bar\alpha})$

So choosing a target **is** choosing a weighting over noise levels, measured in units of clean-sample error. ε-prediction weights $e$ by SNR: at low noise SNR is huge, and a tiny error in $\hat x_0$ is a big error in $\hat\varepsilon$. At very high noise SNR goes to 0, ε-prediction becomes trivial ($\hat\varepsilon\approx x_t/\sqrt{1-\bar\alpha_t}$), and $\hat x_0$ derived from it is unstable, because it divides by $\sqrt{\bar\alpha_t}\to0$. x₀-prediction is the opposite. v-prediction behaves like ε at low noise and like x₀ at high noise, staying well conditioned at both ends. That is why it is used for distillation and zero-terminal-SNR schedules.

## Noise schedules and SNR weighting

The schedule decides how much training time each SNR gets, since $t$ is sampled uniformly. The **linear** schedule drops the SNR fast: $\mathrm{SNR}<1$ from $t=260$ on, and $\bar\alpha_T=4.0\times10^{-5}$. Nichol & Dhariwal's **cosine** schedule, $\bar\alpha_t=f(t)/f(0)$ with $f(t)=\cos^2\!\big(\tfrac{t/T+s}{1+s}\cdot\tfrac\pi2\big)$ and $s=0.008$, keeps $\mathrm{SNR}\ge1$ through $t=496$, spending more steps at the informative mid-range noise levels. Kingma et al. (2021) showed that the continuous-time ELBO depends on the schedule only through its endpoints. The schedule acts on what is learned only through the weighting it induces.

$$
\mathcal L_w=\mathbb E_{t,x_0,\varepsilon}\big[w(t)\,\|x_0-\hat x_0\|^2\big],\qquad w_{\varepsilon}=\mathrm{SNR},\quad w_{v}=\mathrm{SNR}+1,\quad w_{\text{Min-SNR-}\gamma}=\min(\mathrm{SNR},\gamma)
$$

- $w(t)$: weight on the clean-sample error at noise level $t$
- $\gamma$: Min-SNR cap, $\gamma=5$ in Hang et al. (2023). In ε units the loss weight is $\min(\mathrm{SNR},\gamma)/\mathrm{SNR}$

Different noise levels are competing tasks: low noise teaches fine detail, high noise teaches global layout. Under plain ε-prediction the very low-noise steps have large, mostly irreducible loss (0.92 at $t=10$ in the figure) and take a large share of the gradient. **Min-SNR-γ** caps their weight. Hang et al. report about 3.4× faster convergence on ImageNet 256 with it, and the figure shows why it is target-agnostic: after the cap, ε, x₀ and v give the same weighted objective. Other weightings in use include EDM's (Karras et al., 2022) log-normal sampling of noise levels with a preconditioned target, and the ELBO weighting itself, which is preferred when likelihood matters.

## Worked example

**One training example, ᾱ_t = 0.64 (√ᾱ = 0.8, √(1−ᾱ) = 0.6, SNR = 1.778)**

1. Clean sample $x_0=0.8$, noise $\varepsilon=-0.5$: $x_t=0.8\cdot0.8+0.6\cdot(-0.5)=0.34$.
2. The network predicts $\hat\varepsilon=-0.3$. Loss $(\hat\varepsilon-\varepsilon)^2=0.2^2=0.04$, gradient $2(\hat\varepsilon-\varepsilon)=0.4$.
3. Implied clean sample $\hat x_0=(0.34-0.6\cdot(-0.3))/0.8=0.65$, error $e=0.15^2=0.0225$. Check: $\mathrm{SNR}\cdot e=1.778\cdot0.0225=0.04$.
4. Velocity target $v=0.8(-0.5)-0.6(0.8)=-0.88$, implied $\hat v=0.8(-0.3)-0.6(0.65)=-0.63$: $(\hat v-v)^2=0.0625=(1+1.778)\cdot0.0225$.
5. Min-SNR-5 weight: SNR $1.778<5$, so the weight is 1. At a low-noise step with $\bar\alpha_t=0.96$ (SNR $=24$) the ε-loss would be multiplied by $5/24=0.208$.

The figure's loss floor E‖target − optimal prediction‖² per step, linear schedule (computed from the lib by quadrature)

| t | ᾱ_t | SNR | ε floor | x₀ floor | v floor |
| --- | --- | --- | --- | --- | --- |
| 10 | 0.9981 | 526.8 | 0.922 | 0.0018 | 0.924 |
| 100 | 0.8970 | 8.71 | 0.229 | 0.0263 | 0.255 |
| 300 | 0.3964 | 0.657 | 0.385 | 0.586 | 0.971 |
| 500 | 0.0786 | 0.0853 | 0.080 | 0.940 | 1.020 |
| 1000 | 4.0e-5 | 4.0e-5 | 0.000 | 1.022 | 1.022 |

```python
import torch, torch.nn.functional as F

T = 1000
betas = torch.linspace(1e-4, 0.02, T)                  # DDPM linear schedule
alpha_bar = torch.cumprod(1.0 - betas, dim=0)          # alpha_bar[0] = 1 - 1e-4, alpha_bar[-1] = 4.0e-5

def diffusion_loss(model, x0, target="eps", min_snr_gamma=None):
    B = x0.shape[0]
    t = torch.randint(0, T, (B,), device=x0.device)    # index 0..T-1 is step 1..T
    ab = alpha_bar.to(x0.device)[t].view(B, *[1] * (x0.dim() - 1))
    eps = torch.randn_like(x0)
    xt = ab.sqrt() * x0 + (1 - ab).sqrt() * eps        # closed-form q(x_t | x_0)
    y = {"eps": eps, "x0": x0, "v": ab.sqrt() * eps - (1 - ab).sqrt() * x0}[target]
    per_example = F.mse_loss(model(xt, t), y, reduction="none").flatten(1).mean(1)
    if min_snr_gamma is not None:
        snr = (ab / (1 - ab)).flatten()
        scale = {"eps": snr, "x0": torch.ones_like(snr), "v": snr + 1}[target]
        per_example = per_example * snr.clamp(max=min_snr_gamma) / scale
    return per_example.mean()                          # gamma=None: L_simple
```

DDPM training loss with ε, x₀ or v targets and optional Min-SNR-γ weighting

> **Loss curves that do not go down:** The diffusion loss is noisy and has a large irreducible floor, so it falls quickly and then looks flat for most of training while samples keep improving. Each batch also has different random $t$, so batch-to-batch variance is large. Track the loss per $t$-bucket or with fixed evaluation noise, and judge with samples or FID. Also make sure the network actually receives $t$ (or $\bar\alpha_t$): without it, the optimal prediction has to average over noise levels.

> **Why a simple MSE beats a GAN's game:** At every noise level the target is a single, fixed, Gaussian quantity, so training is ordinary regression with a stable gradient, no mode-seeking, and no opponent. The multimodality that makes per-pixel reconstruction blurry is handled by the sampler instead of the loss. Each step only needs the posterior **mean** at that noise level, and the chain of small denoising steps assembles a sharp sample from those means.

> **Terminal SNR and other details:** The linear and Stable Diffusion schedules do not reach $\mathrm{SNR}=0$ at $t=T$ ($\bar\alpha_T=4.0\times10^{-5}$ for DDPM's linear schedule), so training never sees pure noise while sampling starts from it. Lin et al. (2024) rescale the schedule to zero terminal SNR, which requires v-prediction because ε-prediction is degenerate there. DDPM also fixes the reverse variances; Nichol & Dhariwal learn them with a small ELBO term added to $L_{\text{simple}}$ ("hybrid" loss).

**Strengths**

- Plain MSE regression: stable, no adversarial game, easy to scale
- A reweighted ELBO and a denoising score-matching objective, so it has a clear probabilistic meaning
- Covers all modes: the model learns the full distribution, not a single answer
- Target and weighting are flexible (ε, x₀, v; uniform, Min-SNR, EDM)

**Limitations**

- Large irreducible floor and high variance make the loss value a poor progress signal
- Plain ε-prediction over-weights low-noise steps and is unstable near zero SNR
- The weighting that gives the best samples is not the ELBO, so likelihoods and samples favour different settings
- Sampling needs many network evaluations (tens to thousands of steps) unless distilled

**Question:** A network trained with L_simple outputs ε̂ = ε_θ(x_t, t). What is the optimal ε̂ in terms of the data?

- A. $\varepsilon$ itself, exactly
- B. $\mathbb E[\varepsilon\mid x_t]$, equivalently $(x_t-\sqrt{\bar\alpha_t}\,\mathbb E[x_0\mid x_t])/\sqrt{1-\bar\alpha_t}$
- C. $0$, the mean of the noise
- D. $x_t$

**Answer:** B. $\mathbb E[\varepsilon\mid x_t]$, equivalently $(x_t-\sqrt{\bar\alpha_t}\,\mathbb E[x_0\mid x_t])/\sqrt{1-\bar\alpha_t}$ — MSE is minimized by the conditional mean of the target. Given $x_t$, the noise is not determined, so the best prediction is $\mathbb E[\varepsilon\mid x_t]$, which follows from the posterior mean of $x_0$. That is why even a perfect model has a positive loss: 0.385 at $t=300$ in the figure.

**Question:** Switching the target from ε to x₀ with an unweighted MSE is equivalent to what change in the ε-loss?

- A. No change at all
- B. Multiplying the ε-loss at step $t$ by $1/\mathrm{SNR}(t)$
- C. Multiplying the ε-loss by $\mathrm{SNR}(t)$
- D. Changing the noise schedule to cosine

**Answer:** B. Multiplying the ε-loss at step $t$ by $1/\mathrm{SNR}(t)$ — $\|\varepsilon-\hat\varepsilon\|^2=\mathrm{SNR}(t)\,\|x_0-\hat x_0\|^2$, so $\|x_0-\hat x_0\|^2=\|\varepsilon-\hat\varepsilon\|^2/\mathrm{SNR}(t)$. x₀-prediction down-weights low-noise steps (high SNR) and up-weights high-noise ones. Min-SNR-γ sits in between by capping the weight at $\gamma$.

**Key points**

- $L_{\text{simple}}=\mathbb E\|\varepsilon-\varepsilon_\theta(\sqrt{\bar\alpha_t}x_0+\sqrt{1-\bar\alpha_t}\varepsilon,t)\|^2$ with uniform $t$: plain MSE on the noise.
- The optimal noise predictor is $\mathbb E[\varepsilon|x_t]$: a denoiser and, up to $-\sqrt{1-\bar\alpha_t}$, the score $\nabla\log q_t$.
- It is the ELBO with its per-step weights dropped; the weights matter for likelihood, not much for samples.
- ε, x₀ and v targets differ by factors SNR and 1+SNR on the clean-sample error, so the target is a weighting choice.
- Schedules (linear, cosine) set how training time is spread over SNR; Min-SNR-γ caps the weight of easy low-noise steps.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
