Score-based generative models

A score-based model learns s_θ(x, σ) ≈ ∇ₓ log p_σ(x), the gradient of the log density of data blurred by Gaussian noise of size σ, by denoising score matching. It samples by following that field from large to small σ: annealed Langevin dynamics, a reverse-time SDE, or its probability-flow ODE.

15 min read1243 wordsadvanced0 viewsupdated 2026-10-12

Definition

Score-based generative model A generative model that represents a distribution by its score, the vector field ∇xlog⁡p(x)\nabla_x\log p(x), instead of by p(x)p(x) itself. A network sθ(x,σ)s_\theta(x,\sigma) is trained to match the score of the data distribution perturbed by Gaussian noise of scale σ\sigma, for many scales at once. Samples are produced by Langevin dynamics run from large to small σ\sigma (Song and Ermon, NeurIPS 2019), or, in the continuous-time view, by solving a reverse-time stochastic differential equation or the equivalent deterministic probability-flow ODE (Song, Sohl-Dickstein, Kingma, Kumar, Ermon and Poole, ICLR 2021).

A density model pθ(x)=e−Eθ(x)/Zθp_\theta(x)=e^{-E_\theta(x)}/Z_\theta needs the normalising constant Zθ=∫e−EθZ_\theta=\int e^{-E_\theta}, which is intractable for images. The score does not: ∇xlog⁡pθ(x)=−∇xEθ(x)\nabla_x\log p_\theta(x)=-\nabla_xE_\theta(x), and ZθZ_\theta drops out. Hyvärinen (JMLR 2005) showed how to fit a score without knowing the true one (score matching). Vincent ("A Connection Between Score Matching and Denoising Autoencoders", Neural Computation 2011) showed that matching the score of noise-perturbed data is the same as training a denoiser. Yang Song and Stefano Ermon ("Generative Modeling by Estimating Gradients of the Data Distribution", NeurIPS 2019) turned this into a generator, the noise conditional score network (NCSN), sampled with annealed Langevin dynamics. In parallel, Ho, Jain and Abbeel's DDPM (2020) reached state-of-the-art image quality with a noise-prediction loss (see Denoising diffusion probabilistic models (DDPM)). Song et al. (2021) showed that NCSN and DDPM are two discretisations of one stochastic differential equation, and that view now underlies most of Diffusion models.

The score and Langevin dynamics

s(x)=∇xlog⁡p(x),xk+1=xk+α2 s(xk)+α zk,zk∼N(0,I)s(x)=\nabla_x\log p(x),\qquad x_{k+1}=x_k+\frac{\alpha}{2}\,s(x_k)+\sqrt{\alpha}\,z_k,\quad z_k\sim\mathcal N(0,I)
s(x)s(x)
points uphill in log density; its length grows with the distance from the nearest high-density region
α\alpha
step size; as α→0\alpha\to0 and K→∞K\to\infty steps, xKx_K is distributed as pp (under regularity conditions)
α zk\sqrt\alpha\,z_k
injected noise; without it the iteration is gradient ascent and collapses onto the modes

Two problems make plain Langevin with a learned score fail on real data. First, inaccurate scores in low-density regions. The score is learned from samples, so it is only reliable where data lives. Images lie near a low-dimensional manifold, where the score of the clean distribution is not even defined off the manifold. A sampler starting from noise begins exactly where the estimate is worst. Second, slow mixing between modes. Song and Ermon's example is p=15N((−5,−5),I)+45N((5,5),I)p=\tfrac15\mathcal N((-5,-5),I)+\tfrac45\mathcal N((5,5),I). Far from both modes the score is almost the score of whichever component is nearer, so it carries almost no information about the mixing weights. Langevin chains started uniformly split roughly in half instead of 1:4, and the barrier between the modes is too wide to cross in practice. Both problems have the same fix: add noise to the data. A wide Gaussian blur fills the empty space with density, so the score is defined and learnable everywhere, and it connects the modes so their relative mass becomes visible to the sampler.

Denoising score matching

JESM=12 Ep∥sθ(x)−∇xlog⁡p(x)∥2  =  Ep[tr(∇xsθ(x))+12∥sθ(x)∥2]+const\mathcal J_{\mathrm{ESM}}=\tfrac12\,\mathbb E_{p}\big\|s_\theta(x)-\nabla_x\log p(x)\big\|^2\;=\;\mathbb E_{p}\Big[\mathrm{tr}\big(\nabla_xs_\theta(x)\big)+\tfrac12\|s_\theta(x)\|^2\Big]+\text{const}
JESM\mathcal J_{\mathrm{ESM}}
explicit score matching: the target ∇log⁡p\nabla\log p is unknown
tr(∇xsθ)\mathrm{tr}(\nabla_xs_\theta)
Hyvärinen's integration-by-parts form needs no target, but the trace of the Jacobian costs DD backward passes for DD-dimensional data
JDSM(σ)=12 Ex∼p, x~∼N(x,σ2I)∥sθ(x~,σ)+x~−xσ2∥2,s∗(x~,σ)=∇x~log⁡pσ(x~),pσ=p∗N(0,σ2I)\mathcal J_{\mathrm{DSM}}(\sigma)=\tfrac12\,\mathbb E_{x\sim p,\ \tilde x\sim\mathcal N(x,\sigma^2I)}\Big\|s_\theta(\tilde x,\sigma)+\frac{\tilde x-x}{\sigma^2}\Big\|^2,\qquad s^\ast(\tilde x,\sigma)=\nabla_{\tilde x}\log p_\sigma(\tilde x),\quad p_\sigma=p*\mathcal N(0,\sigma^2I)
−(x~−x)/σ2-(\tilde x-x)/\sigma^2
the score of the noising kernel qσ(x~∣x)q_\sigma(\tilde x\mid x), known in closed form; with x~=x+σz\tilde x=x+\sigma z it equals −z/σ-z/\sigma
s∗s^\ast
the minimiser (Vincent 2011): the score of the perturbed density pσp_\sigma, not of pp
residual\text{residual}
the minimum is not 0; it is 12E∥z−E[z∣x~]∥2/σ2\tfrac12\mathbb E\|z-\mathbb E[z\mid\tilde x]\|^2/\sigma^2, which only measures how ambiguous the noise is

The optimal score is also an optimal denoiser. Tweedie's formula gives E[x∣x~]=x~+σ2∇x~log⁡pσ(x~)\mathbb E[x\mid\tilde x]=\tilde x+\sigma^2\nabla_{\tilde x}\log p_\sigma(\tilde x), so a score network and a network that predicts the noise zz are the same object up to scaling: sθ(x~,σ)=−z^θ(x~,σ)/σs_\theta(\tilde x,\sigma)=-\hat z_\theta(\tilde x,\sigma)/\sigma. This is why DDPM's ϵ\epsilon-prediction loss (see Diffusion noise-prediction loss) is denoising score matching with a particular weighting.

NCSN: many noise levels, annealed Langevin

L(θ)=1L∑i=1Lλ(σi) JDSM(σi),λ(σ)=σ2,σ1>σ2>⋯>σL,  σiσi+1=const\mathcal L(\theta)=\frac1L\sum_{i=1}^{L}\lambda(\sigma_i)\,\mathcal J_{\mathrm{DSM}}(\sigma_i),\qquad \lambda(\sigma)=\sigma^2,\qquad \sigma_1>\sigma_2>\dots>\sigma_L,\ \ \frac{\sigma_{i}}{\sigma_{i+1}}=\text{const}
σ1\sigma_1
large enough to connect all modes; NCSNv2 recommends about the largest pairwise distance between training points
σL\sigma_L
small enough that pσL≈pp_{\sigma_L}\approx p
λ(σ)=σ2\lambda(\sigma)=\sigma^2
because ∥∇log⁡pσ∥∝1/σ\|\nabla\log p_\sigma\|\propto1/\sigma, this makes every level's term 12E∥σsθ+z∥2\tfrac12\mathbb E\|\sigma s_\theta+z\|^2 of the same order

Annealed Langevin dynamics (Song and Ermon 2019, Algorithm 1)

  1. 1Initialise x~0\tilde x_0 from a fixed prior (uniform noise in the paper). Choose a base step ϵ\epsilon and TT steps per level.
  2. 2For i=1,…,Li=1,\dots,L: set αi=ϵ σi2/σL2\alpha_i=\epsilon\,\sigma_i^2/\sigma_L^2, so the "signal-to-noise" of a step, ∥αi2s∥/∥αiz∥\|\tfrac{\alpha_i}{2}s\|/\|\sqrt{\alpha_i}z\|, does not depend on σi\sigma_i.
  3. 3Run TT Langevin steps x~t=x~t−1+αi2sθ(x~t−1,σi)+αi zt\tilde x_t=\tilde x_{t-1}+\tfrac{\alpha_i}{2}s_\theta(\tilde x_{t-1},\sigma_i)+\sqrt{\alpha_i}\,z_t, then start the next level from the last sample.
  4. 4Return x~T\tilde x_T of the last level. Song and Ermon used L=10L=10 geometric levels from σ1=1\sigma_1=1 to σ10=0.01\sigma_{10}=0.01 (pixels in [0,1][0,1]), T=100T=100, ϵ=2×10−5\epsilon=2\times10^{-5}.

NCSN and NCSNv2 in practice

Network
NCSN: a RefineNet-style U-Net with dilated convolutions and conditional instance normalisation (CondInstanceNorm++) that injects σi\sigma_i
CIFAR-10 (2019)
Inception score 8.87, FID 25.32, unconditional
NCSNv2 (Song & Ermon, NeurIPS 2020)
σ1\sigma_1 about the maximum pairwise data distance; more, geometrically spaced levels; noise conditioning by output scaling sθ(x,σ)=sθ(x)/σs_\theta(x,\sigma)=s_\theta(x)/\sigma; exponential moving average of weights. CIFAR-10 FID 10.87, and samples at up to 256×256

Score SDEs: infinitely many noise levels

Let the noise scale grow continuously with a time t∈[0,T]t\in[0,T]. The perturbed densities ptp_t are then the marginals of a forward SDE, and Anderson ("Reverse-time diffusion equation models", Stochastic Processes and their Applications 1982) gives the SDE that runs it backward. Its only unknown is the score of every marginal, which is what a time-conditional sθ(x,t)s_\theta(x,t) learns by denoising score matching.

dx=f(x,t) dt+g(t) dw,dx=[f(x,t)−g(t)2 ∇xlog⁡pt(x)]dt+g(t) dwˉ,dxdt=f(x,t)−12g(t)2 ∇xlog⁡pt(x)\mathrm dx=f(x,t)\,\mathrm dt+g(t)\,\mathrm dw,\qquad \mathrm dx=\big[f(x,t)-g(t)^2\,\nabla_x\log p_t(x)\big]\mathrm dt+g(t)\,\mathrm d\bar w,\qquad \frac{\mathrm dx}{\mathrm dt}=f(x,t)-\tfrac12g(t)^2\,\nabla_x\log p_t(x)
f, gf,\ g
drift and diffusion coefficients of the forward (noising) SDE; ww is a Wiener process
dwˉ\mathrm d\bar w
a Wiener process running backward in time, and dt\mathrm dt is negative: integrate from t=Tt=T to 00
third\text{third}
the probability-flow ODE: deterministic, with the same marginals ptp_t as the SDE for every tt
propertyVE SDEVP SDE
forward SDEdx=d[σ2(t)]/dt  dw\mathrm dx=\sqrt{\mathrm d[\sigma^2(t)]/\mathrm dt}\;\mathrm dwdx=−12β(t) x dt+β(t) dw\mathrm dx=-\tfrac12\beta(t)\,x\,\mathrm dt+\sqrt{\beta(t)}\,\mathrm dw
perturbation kernelN(x0, [σ2(t)−σ2(0)]I)\mathcal N\big(x_0,\ [\sigma^2(t)-\sigma^2(0)]I\big)N(x0e−12∫0tβ, (1−e−∫0tβ)I)\mathcal N\big(x_0e^{-\frac12\int_0^t\beta},\ (1-e^{-\int_0^t\beta})I\big)
variance as t growsexplodes (σ_max = 50 for CIFAR-10)stays near 1 if the data has unit variance
discrete versionNCSN, σ(t) geometric from 0.01 to 50DDPM, β(t) linear from 0.1 to 20 (β₁ = 10⁻⁴ … β₁₀₀₀ = 0.02)
prior at t = TN(0,σmax⁡2I)\mathcal N(0,\sigma_{\max}^2I)N(0,I)\mathcal N(0,I)
The two main SDEs of Song et al. (2021) and the discrete models they generalise

Any numerical solver of the reverse SDE is a sampler. The paper's reverse diffusion predictor discretises it the same way as the forward process. For VE it is xi−1=xi+(σi2−σi−12) sθ(xi,σi)+σi2−σi−12  zix_{i-1}=x_i+(\sigma_i^2-\sigma_{i-1}^2)\,s_\theta(x_i,\sigma_i)+\sqrt{\sigma_i^2-\sigma_{i-1}^2}\;z_i. A predictor–corrector (PC) sampler follows each predictor step with one or more Langevin corrector steps at the new noise level. These are score-based MCMC moves that pull the sample back to ptp_{t}, with a step size set from a target signal-to-noise ratio. Annealed Langevin is the corrector alone; ancestral DDPM sampling is the predictor alone. At equal compute, PC beat both in their CIFAR-10 experiments. The probability-flow ODE gives three further things. It is a deterministic map from noise to data, so latents can be interpolated and encoded. Adaptive black-box ODE solvers trade accuracy for steps. And the model has an exact likelihood through the instantaneous change of variables, log⁡p0(x0)=log⁡pT(xT)+∫0T∇ ⁣⋅f~(xt,t) dt\log p_0(x_0)=\log p_T(x_T)+\int_0^T\nabla\!\cdot\tilde f(x_t,t)\,\mathrm dt, with the divergence estimated by Hutchinson–Skilling. Later fast samplers such as Denoising diffusion implicit models (DDIM) are solvers of this ODE.

Score SDE results (Song et al., ICLR 2021)

CIFAR-10 samples
NCSN++ cont. (deep, VE SDE): FID 2.20, Inception score 9.89, unconditional, then a record
CIFAR-10 likelihood
DDPM++ cont. (deep, sub-VP SDE) with the probability-flow ODE: 2.99 bits/dim
High resolution
1024×1024 CelebA-HQ samples, the first from a score-based model
Controllable generation
class-conditional sampling, inpainting and colourisation with an unconditional model, by adding ∇xlog⁡pt(y∣x)\nabla_x\log p_t(y\mid x) to the score
p = 0.2·N((−5,−5), I) + 0.8·N((5,5), I) · shade = p_σ(x) · arrows = σ·∇ₓlog p_σ(x)w = 0.8 at (5, 5)w = 0.2 at (−5, −5)[−10, 10]² · dashed: x + y = 0samplerannealed Langevinstep1000 / 1000 score evalsσ now0.300startuniform boxshare at (5, 5)0.807 (target 0.800)share at (5, 5) over the run0.50.801000 stepsx₀ ~ uniform on [−8, 8]² · exact score, component variance 1 + σ²level 10/10: x ← x + (αᵢ/2)·s(x, σᵢ) + √αᵢ·z, αᵢ = ε·σᵢ²/σ_L² = 0.050σ: 20 → 0.3 geometric
sampler

Every sampler uses the exact score and the same 1000 score evaluations; only the noise schedule and the update differ. σ_min = 0.3, ε = 0.05, 300 samples.

fig.Song and Ermon's mixture p = 0.2·N((−5,−5), I) + 0.8·N((5,5), I). Every sampler uses the exact score of the perturbed mixture (component variance 1 + σ²), 300 samples and the same 1000 score evaluations, so only the sampler differs. Plain Langevin at σ = 0.3 from a uniform start on [−8, 8]² keeps the split it started with: 0.480 of the samples end at (5, 5) (0.515 averaged over 9 seeds), not 0.8. Annealed Langevin with 10 geometric levels from σ_max = 20 to 0.3 and 100 steps each ends at 0.807 (0.783 over 9 seeds). Its first step size is α₁ = 0.05·(20/0.3)² = 222. With σ_max = 2, smaller than the 14.1 gap between the modes, it reaches only 0.583, and with σ_max = 1 only 0.493. The reverse VE SDE (reverse-diffusion steps) starts from N(0, σ_max² I) and ends at 0.763 for σ_max = 20 and 0.783 for σ_max = 50. The probability-flow ODE ends at 0.760 for σ_max = 20 and changes no further after step 600, since it never injects noise. The within-mode standard deviation after annealing is 1.08 (target 1). Arrows show σ·∇ₓlog p_σ, the quantity a noise-predicting network outputs.

Worked example: one score at three noise levels

Take the same mixture and the point x~=(1,0)\tilde x=(1,0), slightly on the heavy side of the boundary x+y=0x+y=0. Each component of pσp_\sigma is N(μk,(1+σ2)I)\mathcal N(\mu_k,(1+\sigma^2)I). With responsibilities γk\gamma_k, the score is ∑kγk(μk−x~)/(1+σ2)\sum_k\gamma_k(\mu_k-\tilde x)/(1+\sigma^2).

σvariance 1 + σ²γ of the (5, 5) mode∇log p_σ(1, 0)σ·‖∇log p_σ‖Tweedie E[x | x̃]
0.31.090.999974(3.670, 4.587)1.762(1.330, 0.413)
250.967273(0.735, 0.935)2.377(3.938, 3.738)
204010.803960(0.0051, 0.0076)0.183(3.035, 3.032)
Score of p_σ at (1, 0) (computed in Node with the figure's code)

Reading the table

  1. 1At σ=0.3\sigma=0.3 the light mode has responsibility 2.6×10−52.6\times10^{-5}. The score is that of the heavy component alone: it says nothing about the 0.2 weight.
  2. 2At σ=20\sigma=20 the responsibility of the heavy mode is 0.8040.804, almost the prior weight 0.80.8. Tweedie's denoised estimate x~+σ2∇log⁡pσ=(3.035,3.032)\tilde x+\sigma^2\nabla\log p_\sigma=(3.035,3.032) is close to the overall mean 0.8⋅5+0.2⋅(−5)=30.8\cdot5+0.2\cdot(-5)=3. The weights are encoded in the field.
  3. 3Annealed Langevin at σ=2\sigma=2 (level step α=0.05⋅22/0.32=2.222\alpha=0.05\cdot2^2/0.3^2=2.222): the drift is α2∇log⁡p2=(0.816, 1.038)\tfrac\alpha2\nabla\log p_2=(0.816,\,1.038) and the noise has standard deviation 2.222=1.491\sqrt{2.222}=1.491 per coordinate.
  4. 4At σ=20\sigma=20 the step is α1=222.2\alpha_1=222.2, noise standard deviation 14.914.9: large enough to jump between the modes, which is how the sampler sets their proportions before the small levels refine the shape.
python
1import math, torch, torch.nn as nn
2
3torch.manual_seed(0)
4def sample_data(n): # 0.2·N((−5,−5), I) + 0.8·N((5,5), I)
5 heavy = (torch.rand(n) < 0.8).float().unsqueeze(1)
6 return torch.randn(n, 2) + (10 * heavy - 5) * torch.ones(n, 2)
7
8sigmas = torch.logspace(math.log10(20), math.log10(0.3), 10) # σ_1 = 20 > … > σ_L = 0.3
9
10class NCSN(nn.Module): # s_θ(x, σ) = f_θ(x, log σ) / σ
11 def __init__(self, h=128):
12 super().__init__()
13 self.f = nn.Sequential(nn.Linear(3, h), nn.SiLU(), nn.Linear(h, h), nn.SiLU(), nn.Linear(h, 2))
14 def forward(self, x, sigma): # x (N, 2), sigma (N, 1)
15 return self.f(torch.cat([x / (1 + sigma**2).sqrt(), sigma.log()], 1)) / sigma
16
17def dsm_loss(model, x):
18 idx = torch.randint(0, len(sigmas), (x.size(0),))
19 sigma = sigmas[idx].unsqueeze(1) # (N, 1)
20 z = torch.randn_like(x)
21 x_tilde = x + sigma * z
22 target = -z / sigma # ∇ log q_σ(x̃ | x) = −(x̃ − x)/σ²
23 return 0.5 * ((sigma * (model(x_tilde, sigma) - target)) ** 2).sum(1).mean() # λ(σ) = σ²
24
25model = NCSN()
26opt = torch.optim.Adam(model.parameters(), lr=1e-3)
27for step in range(3000):
28 loss = dsm_loss(model, sample_data(512))
29 opt.zero_grad(); loss.backward(); opt.step()
30print(f"final DSM loss {loss.item():.3f}") # ≈ 0.39, mostly the irreducible residual, not 0
31
32@torch.no_grad()
33def annealed_langevin(model, n=2000, T=100, eps=0.05):
34 x = torch.rand(n, 2) * 16 - 8 # uniform on [−8, 8]²
35 for sigma in sigmas:
36 alpha = eps * (sigma / sigmas[-1]) ** 2 # α_i = ε σ_i² / σ_L²
37 s = torch.full((n, 1), sigma.item())
38 for _ in range(T):
39 x = x + 0.5 * alpha * model(x, s) + alpha.sqrt() * torch.randn_like(x)
40 return x
41
42x = annealed_langevin(model)
43print(x.shape) # torch.Size([2000, 2])
44print(f"share in the (5, 5) mode: {(x.sum(1) > 0).float().mean():.3f}") # 0.801 in our run (target 0.8)
NCSN on the 2-D mixture: denoising score matching over 10 noise levels, then annealed Langevin sampling (runs on CPU in about 30 s)

Use when

  • +No normalising constant, no adversarial game, no architectural invertibility constraint: any network that outputs a vector of the input's shape
  • +Stable regression-style training (denoising score matching) that scales to high-resolution images
  • +One trained score model supports many samplers: annealed Langevin, reverse SDE, predictor–corrector, probability-flow ODE
  • +Exact likelihoods and a deterministic encoder through the probability-flow ODE; conditional generation by adding a likelihood score

Avoid when

  • −Sampling needs hundreds to thousands of network evaluations unless distilled or solved with a fast solver
  • −Quality depends on the noise schedule, σ\sigma range, step sizes and corrector settings
  • −Plain (unannealed) Langevin with a learned score fails on multimodal or manifold data
  • −Likelihood via the ODE needs a full ODE solve plus a trace estimator per data point

Check yourself

Why does annealed Langevin start at a large σ instead of running plain Langevin at the smallest σ?

Check yourself

What does the probability-flow ODE share with the reverse-time SDE?

Remember

  • ✓Score s(x)=∇xlog⁡p(x)s(x)=\nabla_x\log p(x) needs no normalising constant; Langevin dynamics x←x+α2s(x)+α zx\leftarrow x+\tfrac{\alpha}{2}s(x)+\sqrt{\alpha}\,z samples pp as α→0\alpha\to0.
  • ✓Denoising score matching: minimise E λ(σ)∥sθ(x+σz,σ)+z/σ∥2\mathbb E\,\lambda(\sigma)\|s_\theta(x+\sigma z,\sigma)+z/\sigma\|^2 with λ=σ2\lambda=\sigma^2; the optimum is ∇log⁡pσ\nabla\log p_\sigma, and sθ=−ϵθ/σs_\theta=-\epsilon_\theta/\sigma links it to noise prediction.
  • ✓NCSN (Song & Ermon 2019): LL geometric noise levels and annealed Langevin with αi=ϵσi2/σL2\alpha_i=\epsilon\sigma_i^2/\sigma_L^2; σmax⁡\sigma_{\max} must span the distance between modes.
  • ✓Score SDE (Song et al. 2021): reverse SDE dx=[f−g2∇log⁡pt]dt+g dwˉ\mathrm dx=[f-g^2\nabla\log p_t]\mathrm dt+g\,\mathrm d\bar w (Anderson 1982) and probability-flow ODE dx/dt=f−12g2∇log⁡pt\mathrm dx/\mathrm dt=f-\tfrac12g^2\nabla\log p_t; NCSN = VE SDE, DDPM = VP SDE.
  • ✓Predictor–corrector sampling, exact ODE likelihoods; CIFAR-10 FID 2.20 and 2.99 bits/dim, and 1024×1024 samples.
← back to Deep Learning roadmap