# Score-based generative models

> A score-based model learns s_θ(x, σ) ≈ ∇ₓ log p_σ(x), the gradient of the log density of data blurred by Gaussian noise of size σ, by denoising score matching. It samples by following that field from large to small σ: annealed Langevin dynamics, a reverse-time SDE, or its probability-flow ODE.

Source: https://www.mlfoundry.in/learn/score-based-generative-models  
Section: Deep Learning › Generative Deep Learning Algorithms  
Level: advanced · 15 min read · updated 2026-10-12

**Score-based generative model:** A generative model that represents a distribution by its **score**, the vector field $\nabla_x\log p(x)$, instead of by $p(x)$ itself. A network $s_\theta(x,\sigma)$ is trained to match the score of the data distribution **perturbed** by Gaussian noise of scale $\sigma$, for many scales at once. Samples are produced by Langevin dynamics run from large to small $\sigma$ (Song and Ermon, NeurIPS 2019), or, in the continuous-time view, by solving a **reverse-time stochastic differential equation** or the equivalent deterministic **probability-flow ODE** (Song, Sohl-Dickstein, Kingma, Kumar, Ermon and Poole, ICLR 2021).

A density model $p_\theta(x)=e^{-E_\theta(x)}/Z_\theta$ needs the normalising constant $Z_\theta=\int e^{-E_\theta}$, which is intractable for images. The score does not: $\nabla_x\log p_\theta(x)=-\nabla_xE_\theta(x)$, and $Z_\theta$ drops out. Hyvärinen (JMLR 2005) showed how to fit a score without knowing the true one (**score matching**). Vincent ("A Connection Between Score Matching and Denoising Autoencoders", Neural Computation 2011) showed that matching the score of *noise-perturbed* data is the same as training a denoiser. Yang Song and Stefano Ermon ("Generative Modeling by Estimating Gradients of the Data Distribution", NeurIPS 2019) turned this into a generator, the **noise conditional score network** (NCSN), sampled with **annealed Langevin dynamics**. In parallel, Ho, Jain and Abbeel's DDPM (2020) reached state-of-the-art image quality with a noise-prediction loss (see **Denoising diffusion probabilistic models (DDPM)**). Song et al. (2021) showed that NCSN and DDPM are two discretisations of one stochastic differential equation, and that view now underlies most of **Diffusion models**.

## The score and Langevin dynamics

$$
s(x)=\nabla_x\log p(x),\qquad x_{k+1}=x_k+\frac{\alpha}{2}\,s(x_k)+\sqrt{\alpha}\,z_k,\quad z_k\sim\mathcal N(0,I)
$$

- $s(x)$: points uphill in log density; its length grows with the distance from the nearest high-density region
- $\alpha$: step size; as $\alpha\to0$ and $K\to\infty$ steps, $x_K$ is distributed as $p$ (under regularity conditions)
- $\sqrt\alpha\,z_k$: injected noise; without it the iteration is gradient ascent and collapses onto the modes

Two problems make plain Langevin with a learned score fail on real data. **First, inaccurate scores in low-density regions.** The score is learned from samples, so it is only reliable where data lives. Images lie near a low-dimensional manifold, where the score of the clean distribution is not even defined off the manifold. A sampler starting from noise begins exactly where the estimate is worst. **Second, slow mixing between modes.** Song and Ermon's example is $p=\tfrac15\mathcal N((-5,-5),I)+\tfrac45\mathcal N((5,5),I)$. Far from both modes the score is almost the score of whichever component is nearer, so it carries almost no information about the mixing weights. Langevin chains started uniformly split roughly in half instead of 1:4, and the barrier between the modes is too wide to cross in practice. Both problems have the same fix: **add noise to the data**. A wide Gaussian blur fills the empty space with density, so the score is defined and learnable everywhere, and it connects the modes so their relative mass becomes visible to the sampler.

## Denoising score matching

$$
\mathcal J_{\mathrm{ESM}}=\tfrac12\,\mathbb E_{p}\big\|s_\theta(x)-\nabla_x\log p(x)\big\|^2\;=\;\mathbb E_{p}\Big[\mathrm{tr}\big(\nabla_xs_\theta(x)\big)+\tfrac12\|s_\theta(x)\|^2\Big]+\text{const}
$$

- $\mathcal J_{\mathrm{ESM}}$: explicit score matching: the target $\nabla\log p$ is unknown
- $\mathrm{tr}(\nabla_xs_\theta)$: Hyvärinen's integration-by-parts form needs no target, but the trace of the Jacobian costs $D$ backward passes for $D$-dimensional data

$$
\mathcal J_{\mathrm{DSM}}(\sigma)=\tfrac12\,\mathbb E_{x\sim p,\ \tilde x\sim\mathcal N(x,\sigma^2I)}\Big\|s_\theta(\tilde x,\sigma)+\frac{\tilde x-x}{\sigma^2}\Big\|^2,\qquad s^\ast(\tilde x,\sigma)=\nabla_{\tilde x}\log p_\sigma(\tilde x),\quad p_\sigma=p*\mathcal N(0,\sigma^2I)
$$

- $-(\tilde x-x)/\sigma^2$: the score of the noising kernel $q_\sigma(\tilde x\mid x)$, known in closed form; with $\tilde x=x+\sigma z$ it equals $-z/\sigma$
- $s^\ast$: the minimiser (Vincent 2011): the score of the **perturbed** density $p_\sigma$, not of $p$
- $\text{residual}$: the minimum is not 0; it is $\tfrac12\mathbb E\|z-\mathbb E[z\mid\tilde x]\|^2/\sigma^2$, which only measures how ambiguous the noise is

The optimal score is also an optimal denoiser. **Tweedie's formula** gives $\mathbb E[x\mid\tilde x]=\tilde x+\sigma^2\nabla_{\tilde x}\log p_\sigma(\tilde x)$, so a score network and a network that predicts the noise $z$ are the same object up to scaling: $s_\theta(\tilde x,\sigma)=-\hat z_\theta(\tilde x,\sigma)/\sigma$. This is why DDPM's $\epsilon$-prediction loss (see **Diffusion noise-prediction loss**) is denoising score matching with a particular weighting.

## NCSN: many noise levels, annealed Langevin

$$
\mathcal L(\theta)=\frac1L\sum_{i=1}^{L}\lambda(\sigma_i)\,\mathcal J_{\mathrm{DSM}}(\sigma_i),\qquad \lambda(\sigma)=\sigma^2,\qquad \sigma_1>\sigma_2>\dots>\sigma_L,\ \ \frac{\sigma_{i}}{\sigma_{i+1}}=\text{const}
$$

- $\sigma_1$: large enough to connect all modes; NCSNv2 recommends about the largest pairwise distance between training points
- $\sigma_L$: small enough that $p_{\sigma_L}\approx p$
- $\lambda(\sigma)=\sigma^2$: because $\|\nabla\log p_\sigma\|\propto1/\sigma$, this makes every level's term $\tfrac12\mathbb E\|\sigma s_\theta+z\|^2$ of the same order

**Annealed Langevin dynamics (Song and Ermon 2019, Algorithm 1)**

1. Initialise $\tilde x_0$ from a fixed prior (uniform noise in the paper). Choose a base step $\epsilon$ and $T$ steps per level.
2. For $i=1,\dots,L$: set $\alpha_i=\epsilon\,\sigma_i^2/\sigma_L^2$, so the "signal-to-noise" of a step, $\|\tfrac{\alpha_i}{2}s\|/\|\sqrt{\alpha_i}z\|$, does not depend on $\sigma_i$.
3. Run $T$ Langevin steps $\tilde x_t=\tilde x_{t-1}+\tfrac{\alpha_i}{2}s_\theta(\tilde x_{t-1},\sigma_i)+\sqrt{\alpha_i}\,z_t$, then start the next level from the last sample.
4. Return $\tilde x_T$ of the last level. Song and Ermon used $L=10$ geometric levels from $\sigma_1=1$ to $\sigma_{10}=0.01$ (pixels in $[0,1]$), $T=100$, $\epsilon=2\times10^{-5}$.

**NCSN and NCSNv2 in practice**

- **Network:** NCSN: a RefineNet-style U-Net with dilated convolutions and conditional instance normalisation (CondInstanceNorm++) that injects $\sigma_i$
- **CIFAR-10 (2019):** Inception score 8.87, FID 25.32, unconditional
- **NCSNv2 (Song & Ermon, NeurIPS 2020):** $\sigma_1$ about the maximum pairwise data distance; more, geometrically spaced levels; noise conditioning by output scaling $s_\theta(x,\sigma)=s_\theta(x)/\sigma$; exponential moving average of weights. CIFAR-10 FID 10.87, and samples at up to 256×256

## Score SDEs: infinitely many noise levels

Let the noise scale grow continuously with a time $t\in[0,T]$. The perturbed densities $p_t$ are then the marginals of a forward **SDE**, and Anderson ("Reverse-time diffusion equation models", Stochastic Processes and their Applications 1982) gives the SDE that runs it backward. Its only unknown is the score of every marginal, which is what a time-conditional $s_\theta(x,t)$ learns by denoising score matching.

$$
\mathrm dx=f(x,t)\,\mathrm dt+g(t)\,\mathrm dw,\qquad \mathrm dx=\big[f(x,t)-g(t)^2\,\nabla_x\log p_t(x)\big]\mathrm dt+g(t)\,\mathrm d\bar w,\qquad \frac{\mathrm dx}{\mathrm dt}=f(x,t)-\tfrac12g(t)^2\,\nabla_x\log p_t(x)
$$

- $f,\ g$: drift and diffusion coefficients of the forward (noising) SDE; $w$ is a Wiener process
- $\mathrm d\bar w$: a Wiener process running backward in time, and $\mathrm dt$ is negative: integrate from $t=T$ to $0$
- $\text{third}$: the **probability-flow ODE**: deterministic, with the same marginals $p_t$ as the SDE for every $t$

The two main SDEs of Song et al. (2021) and the discrete models they generalise

| property | VE SDE | VP SDE |
| --- | --- | --- |
| forward SDE | $\mathrm dx=\sqrt{\mathrm d[\sigma^2(t)]/\mathrm dt}\;\mathrm dw$ | $\mathrm dx=-\tfrac12\beta(t)\,x\,\mathrm dt+\sqrt{\beta(t)}\,\mathrm dw$ |
| perturbation kernel | $\mathcal N\big(x_0,\ [\sigma^2(t)-\sigma^2(0)]I\big)$ | $\mathcal N\big(x_0e^{-\frac12\int_0^t\beta},\ (1-e^{-\int_0^t\beta})I\big)$ |
| variance as t grows | explodes (σ_max = 50 for CIFAR-10) | stays near 1 if the data has unit variance |
| discrete version | NCSN, σ(t) geometric from 0.01 to 50 | DDPM, β(t) linear from 0.1 to 20 (β₁ = 10⁻⁴ … β₁₀₀₀ = 0.02) |
| prior at t = T | $\mathcal N(0,\sigma_{\max}^2I)$ | $\mathcal N(0,I)$ |

Any numerical solver of the reverse SDE is a sampler. The paper's **reverse diffusion** predictor discretises it the same way as the forward process. For VE it is $x_{i-1}=x_i+(\sigma_i^2-\sigma_{i-1}^2)\,s_\theta(x_i,\sigma_i)+\sqrt{\sigma_i^2-\sigma_{i-1}^2}\;z_i$. A **predictor–corrector** (PC) sampler follows each predictor step with one or more Langevin **corrector** steps at the new noise level. These are score-based MCMC moves that pull the sample back to $p_{t}$, with a step size set from a target signal-to-noise ratio. Annealed Langevin is the corrector alone; ancestral DDPM sampling is the predictor alone. At equal compute, PC beat both in their CIFAR-10 experiments. The probability-flow ODE gives three further things. It is a deterministic map from noise to data, so latents can be interpolated and encoded. Adaptive black-box ODE solvers trade accuracy for steps. And the model has an **exact likelihood** through the instantaneous change of variables, $\log p_0(x_0)=\log p_T(x_T)+\int_0^T\nabla\!\cdot\tilde f(x_t,t)\,\mathrm dt$, with the divergence estimated by Hutchinson–Skilling. Later fast samplers such as **Denoising diffusion implicit models (DDIM)** are solvers of this ODE.

**Score SDE results (Song et al., ICLR 2021)**

- **CIFAR-10 samples:** NCSN++ cont. (deep, VE SDE): FID 2.20, Inception score 9.89, unconditional, then a record
- **CIFAR-10 likelihood:** DDPM++ cont. (deep, sub-VP SDE) with the probability-flow ODE: 2.99 bits/dim
- **High resolution:** 1024×1024 CelebA-HQ samples, the first from a score-based model
- **Controllable generation:** class-conditional sampling, inpainting and colourisation with an unconditional model, by adding $\nabla_x\log p_t(y\mid x)$ to the score

> Figure: Song and Ermon's mixture p = 0.2·N((−5,−5), I) + 0.8·N((5,5), I). Every sampler uses the exact score of the perturbed mixture (component variance 1 + σ²), 300 samples and the same 1000 score evaluations, so only the sampler differs. Plain Langevin at σ = 0.3 from a uniform start on [−8, 8]² keeps the split it started with: 0.480 of the samples end at (5, 5) (0.515 averaged over 9 seeds), not 0.8. Annealed Langevin with 10 geometric levels from σ_max = 20 to 0.3 and 100 steps each ends at 0.807 (0.783 over 9 seeds). Its first step size is α₁ = 0.05·(20/0.3)² = 222. With σ_max = 2, smaller than the 14.1 gap between the modes, it reaches only 0.583, and with σ_max = 1 only 0.493. The reverse VE SDE (reverse-diffusion steps) starts from N(0, σ_max² I) and ends at 0.763 for σ_max = 20 and 0.783 for σ_max = 50. The probability-flow ODE ends at 0.760 for σ_max = 20 and changes no further after step 600, since it never injects noise. The within-mode standard deviation after annealing is 1.08 (target 1). Arrows show σ·∇ₓlog p_σ, the quantity a noise-predicting network outputs.

## Worked example: one score at three noise levels

Take the same mixture and the point $\tilde x=(1,0)$, slightly on the heavy side of the boundary $x+y=0$. Each component of $p_\sigma$ is $\mathcal N(\mu_k,(1+\sigma^2)I)$. With responsibilities $\gamma_k$, the score is $\sum_k\gamma_k(\mu_k-\tilde x)/(1+\sigma^2)$.

Score of p_σ at (1, 0) (computed in Node with the figure's code)

| σ | variance 1 + σ² | γ of the (5, 5) mode | ∇log p_σ(1, 0) | σ·‖∇log p_σ‖ | Tweedie E[x \| x̃] |
| --- | --- | --- | --- | --- | --- |
| 0.3 | 1.09 | 0.999974 | (3.670, 4.587) | 1.762 | (1.330, 0.413) |
| 2 | 5 | 0.967273 | (0.735, 0.935) | 2.377 | (3.938, 3.738) |
| 20 | 401 | 0.803960 | (0.0051, 0.0076) | 0.183 | (3.035, 3.032) |

**Reading the table**

1. At $\sigma=0.3$ the light mode has responsibility $2.6\times10^{-5}$. The score is that of the heavy component alone: it says nothing about the 0.2 weight.
2. At $\sigma=20$ the responsibility of the heavy mode is $0.804$, almost the prior weight $0.8$. Tweedie's denoised estimate $\tilde x+\sigma^2\nabla\log p_\sigma=(3.035,3.032)$ is close to the overall mean $0.8\cdot5+0.2\cdot(-5)=3$. The weights are encoded in the field.
3. Annealed Langevin at $\sigma=2$ (level step $\alpha=0.05\cdot2^2/0.3^2=2.222$): the drift is $\tfrac\alpha2\nabla\log p_2=(0.816,\,1.038)$ and the noise has standard deviation $\sqrt{2.222}=1.491$ per coordinate.
4. At $\sigma=20$ the step is $\alpha_1=222.2$, noise standard deviation $14.9$: large enough to jump between the modes, which is how the sampler sets their proportions before the small levels refine the shape.

```python
import math, torch, torch.nn as nn

torch.manual_seed(0)
def sample_data(n):                                   # 0.2·N((−5,−5), I) + 0.8·N((5,5), I)
    heavy = (torch.rand(n) < 0.8).float().unsqueeze(1)
    return torch.randn(n, 2) + (10 * heavy - 5) * torch.ones(n, 2)

sigmas = torch.logspace(math.log10(20), math.log10(0.3), 10)   # σ_1 = 20 > … > σ_L = 0.3

class NCSN(nn.Module):                                 # s_θ(x, σ) = f_θ(x, log σ) / σ
    def __init__(self, h=128):
        super().__init__()
        self.f = nn.Sequential(nn.Linear(3, h), nn.SiLU(), nn.Linear(h, h), nn.SiLU(), nn.Linear(h, 2))
    def forward(self, x, sigma):                       # x (N, 2), sigma (N, 1)
        return self.f(torch.cat([x / (1 + sigma**2).sqrt(), sigma.log()], 1)) / sigma

def dsm_loss(model, x):
    idx = torch.randint(0, len(sigmas), (x.size(0),))
    sigma = sigmas[idx].unsqueeze(1)                   # (N, 1)
    z = torch.randn_like(x)
    x_tilde = x + sigma * z
    target = -z / sigma                                # ∇ log q_σ(x̃ | x) = −(x̃ − x)/σ²
    return 0.5 * ((sigma * (model(x_tilde, sigma) - target)) ** 2).sum(1).mean()   # λ(σ) = σ²

model = NCSN()
opt = torch.optim.Adam(model.parameters(), lr=1e-3)
for step in range(3000):
    loss = dsm_loss(model, sample_data(512))
    opt.zero_grad(); loss.backward(); opt.step()
print(f"final DSM loss {loss.item():.3f}")            # ≈ 0.39, mostly the irreducible residual, not 0

@torch.no_grad()
def annealed_langevin(model, n=2000, T=100, eps=0.05):
    x = torch.rand(n, 2) * 16 - 8                      # uniform on [−8, 8]²
    for sigma in sigmas:
        alpha = eps * (sigma / sigmas[-1]) ** 2        # α_i = ε σ_i² / σ_L²
        s = torch.full((n, 1), sigma.item())
        for _ in range(T):
            x = x + 0.5 * alpha * model(x, s) + alpha.sqrt() * torch.randn_like(x)
    return x

x = annealed_langevin(model)
print(x.shape)                                         # torch.Size([2000, 2])
print(f"share in the (5, 5) mode: {(x.sum(1) > 0).float().mean():.3f}")   # 0.801 in our run (target 0.8)
```

NCSN on the 2-D mixture: denoising score matching over 10 noise levels, then annealed Langevin sampling (runs on CPU in about 30 s)

> **A low loss does not mean a good score where you sample:** Denoising score matching only constrains $s_\theta$ near noisy training points. With a single small $\sigma$ the network is never trained in the empty space between modes, yet that is where a sampler started from noise spends its first steps. And even with the exact score, plain Langevin cannot recover mixing weights in a reasonable time (0.480 instead of 0.8 in the figure). The DSM loss also never goes to zero: its floor depends on $\sigma$ and on the data, so compare runs at matched noise levels, not by the raw number.

> **Score, noise prediction and denoising are one quantity:** For $\tilde x=x+\sigma z$: $\nabla\log p_\sigma(\tilde x)=-\mathbb E[z\mid\tilde x]/\sigma$, and $\mathbb E[x\mid\tilde x]=\tilde x+\sigma^2\nabla\log p_\sigma(\tilde x)$. A DDPM $\epsilon_\theta(x_t,t)$ is therefore a score model, $s_\theta=-\epsilon_\theta/\sqrt{1-\bar\alpha_t}$, and NCSN's $\lambda(\sigma)=\sigma^2$ weighting is DDPM's unweighted $\epsilon$ loss. Guidance methods (classifier guidance, classifier-free guidance in **Stable Diffusion**) are written most naturally as sums of scores.

> **Choosing the noise range:** The largest $\sigma$ must make $p_{\sigma_1}$ look like the prior you start from, and must be comparable to the largest distance between data points, or modes stay disconnected (σ_max = 2 versus the 14.1 gap in the figure gives 0.583). The smallest $\sigma$ sets the residual blur of the samples. In between, geometric spacing keeps neighbouring $p_{\sigma_i}$ overlapping, so each level's sampler starts close to its target. VE models also often add one final denoising step $x+\sigma_L^2s_\theta(x,\sigma_L)$ (Tweedie) to remove the last bit of noise.

**Strengths**

- No normalising constant, no adversarial game, no architectural invertibility constraint: any network that outputs a vector of the input's shape
- Stable regression-style training (denoising score matching) that scales to high-resolution images
- One trained score model supports many samplers: annealed Langevin, reverse SDE, predictor–corrector, probability-flow ODE
- Exact likelihoods and a deterministic encoder through the probability-flow ODE; conditional generation by adding a likelihood score

**Limitations**

- Sampling needs hundreds to thousands of network evaluations unless distilled or solved with a fast solver
- Quality depends on the noise schedule, $\sigma$ range, step sizes and corrector settings
- Plain (unannealed) Langevin with a learned score fails on multimodal or manifold data
- Likelihood via the ODE needs a full ODE solve plus a trace estimator per data point

**Question:** Why does annealed Langevin start at a large σ instead of running plain Langevin at the smallest σ?

- A. Large σ makes the network smaller
- B. The perturbed density at large σ has accurate, well-defined scores everywhere and connects the modes, so the sampler can set the mode proportions before refining at small σ
- C. Large σ steps are deterministic
- D. Langevin dynamics only converges for σ greater than 1

**Answer:** B. The perturbed density at large σ has accurate, well-defined scores everywhere and connects the modes, so the sampler can set the mode proportions before refining at small σ — At small $\sigma$ the score is poorly estimated off the data and nearly blind to mixing weights (responsibility $2.6\times10^{-5}$ for the far mode in the worked example). At $\sigma=20$ the score encodes the 0.8/0.2 weights and steps of size $\alpha_1\approx222$ can jump between the modes.

**Question:** What does the probability-flow ODE share with the reverse-time SDE?

- A. Nothing: it samples a different distribution
- B. The same sample paths
- C. The same marginal densities p_t at every time t, but with deterministic trajectories
- D. The same amount of injected noise

**Answer:** C. The same marginal densities p_t at every time t, but with deterministic trajectories — $\mathrm dx/\mathrm dt=f-\tfrac12g^2\nabla\log p_t$ moves probability mass exactly as the SDE does on average, so $p_t$ matches for all $t$, but each starting point maps to one output. That enables exact likelihoods and encoding. The cost: errors in the score or the prior are not corrected by fresh noise.

**Key points**

- Score $s(x)=\nabla_x\log p(x)$ needs no normalising constant; Langevin dynamics $x\leftarrow x+\tfrac{\alpha}{2}s(x)+\sqrt{\alpha}\,z$ samples $p$ as $\alpha\to0$.
- Denoising score matching: minimise $\mathbb E\,\lambda(\sigma)\|s_\theta(x+\sigma z,\sigma)+z/\sigma\|^2$ with $\lambda=\sigma^2$; the optimum is $\nabla\log p_\sigma$, and $s_\theta=-\epsilon_\theta/\sigma$ links it to noise prediction.
- NCSN (Song & Ermon 2019): $L$ geometric noise levels and annealed Langevin with $\alpha_i=\epsilon\sigma_i^2/\sigma_L^2$; $\sigma_{\max}$ must span the distance between modes.
- Score SDE (Song et al. 2021): reverse SDE $\mathrm dx=[f-g^2\nabla\log p_t]\mathrm dt+g\,\mathrm d\bar w$ (Anderson 1982) and probability-flow ODE $\mathrm dx/\mathrm dt=f-\tfrac12g^2\nabla\log p_t$; NCSN = VE SDE, DDPM = VP SDE.
- Predictor–corrector sampling, exact ODE likelihoods; CIFAR-10 FID 2.20 and 2.99 bits/dim, and 1024×1024 samples.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
