Definition
Score-based generative model A generative model that represents a distribution by its score, the vector field , instead of by itself. A network is trained to match the score of the data distribution perturbed by Gaussian noise of scale , for many scales at once. Samples are produced by Langevin dynamics run from large to small (Song and Ermon, NeurIPS 2019), or, in the continuous-time view, by solving a reverse-time stochastic differential equation or the equivalent deterministic probability-flow ODE (Song, Sohl-Dickstein, Kingma, Kumar, Ermon and Poole, ICLR 2021).
A density model needs the normalising constant , which is intractable for images. The score does not: , and drops out. Hyvärinen (JMLR 2005) showed how to fit a score without knowing the true one (score matching). Vincent ("A Connection Between Score Matching and Denoising Autoencoders", Neural Computation 2011) showed that matching the score of noise-perturbed data is the same as training a denoiser. Yang Song and Stefano Ermon ("Generative Modeling by Estimating Gradients of the Data Distribution", NeurIPS 2019) turned this into a generator, the noise conditional score network (NCSN), sampled with annealed Langevin dynamics. In parallel, Ho, Jain and Abbeel's DDPM (2020) reached state-of-the-art image quality with a noise-prediction loss (see Denoising diffusion probabilistic models (DDPM)). Song et al. (2021) showed that NCSN and DDPM are two discretisations of one stochastic differential equation, and that view now underlies most of Diffusion models.
The score and Langevin dynamics
- points uphill in log density; its length grows with the distance from the nearest high-density region
- step size; as and steps, is distributed as (under regularity conditions)
- injected noise; without it the iteration is gradient ascent and collapses onto the modes
Two problems make plain Langevin with a learned score fail on real data. First, inaccurate scores in low-density regions. The score is learned from samples, so it is only reliable where data lives. Images lie near a low-dimensional manifold, where the score of the clean distribution is not even defined off the manifold. A sampler starting from noise begins exactly where the estimate is worst. Second, slow mixing between modes. Song and Ermon's example is . Far from both modes the score is almost the score of whichever component is nearer, so it carries almost no information about the mixing weights. Langevin chains started uniformly split roughly in half instead of 1:4, and the barrier between the modes is too wide to cross in practice. Both problems have the same fix: add noise to the data. A wide Gaussian blur fills the empty space with density, so the score is defined and learnable everywhere, and it connects the modes so their relative mass becomes visible to the sampler.
Denoising score matching
- explicit score matching: the target is unknown
- Hyvärinen's integration-by-parts form needs no target, but the trace of the Jacobian costs backward passes for -dimensional data
- the score of the noising kernel , known in closed form; with it equals
- the minimiser (Vincent 2011): the score of the perturbed density , not of
- the minimum is not 0; it is , which only measures how ambiguous the noise is
The optimal score is also an optimal denoiser. Tweedie's formula gives , so a score network and a network that predicts the noise are the same object up to scaling: . This is why DDPM's -prediction loss (see Diffusion noise-prediction loss) is denoising score matching with a particular weighting.
NCSN: many noise levels, annealed Langevin
- large enough to connect all modes; NCSNv2 recommends about the largest pairwise distance between training points
- small enough that
- because , this makes every level's term of the same order
Annealed Langevin dynamics (Song and Ermon 2019, Algorithm 1)
- 1Initialise from a fixed prior (uniform noise in the paper). Choose a base step and steps per level.
- 2For : set , so the "signal-to-noise" of a step, , does not depend on .
- 3Run Langevin steps , then start the next level from the last sample.
- 4Return of the last level. Song and Ermon used geometric levels from to (pixels in ), , .
NCSN and NCSNv2 in practice
- Network
- NCSN: a RefineNet-style U-Net with dilated convolutions and conditional instance normalisation (CondInstanceNorm++) that injects
- CIFAR-10 (2019)
- Inception score 8.87, FID 25.32, unconditional
- NCSNv2 (Song & Ermon, NeurIPS 2020)
- about the maximum pairwise data distance; more, geometrically spaced levels; noise conditioning by output scaling ; exponential moving average of weights. CIFAR-10 FID 10.87, and samples at up to 256×256
Score SDEs: infinitely many noise levels
Let the noise scale grow continuously with a time . The perturbed densities are then the marginals of a forward SDE, and Anderson ("Reverse-time diffusion equation models", Stochastic Processes and their Applications 1982) gives the SDE that runs it backward. Its only unknown is the score of every marginal, which is what a time-conditional learns by denoising score matching.
- drift and diffusion coefficients of the forward (noising) SDE; is a Wiener process
- a Wiener process running backward in time, and is negative: integrate from to
- the probability-flow ODE: deterministic, with the same marginals as the SDE for every
| property | VE SDE | VP SDE |
|---|---|---|
| forward SDE | ||
| perturbation kernel | ||
| variance as t grows | explodes (σ_max = 50 for CIFAR-10) | stays near 1 if the data has unit variance |
| discrete version | NCSN, σ(t) geometric from 0.01 to 50 | DDPM, β(t) linear from 0.1 to 20 (β₁ = 10⁻⁴ … β₁₀₀₀ = 0.02) |
| prior at t = T |
Any numerical solver of the reverse SDE is a sampler. The paper's reverse diffusion predictor discretises it the same way as the forward process. For VE it is . A predictor–corrector (PC) sampler follows each predictor step with one or more Langevin corrector steps at the new noise level. These are score-based MCMC moves that pull the sample back to , with a step size set from a target signal-to-noise ratio. Annealed Langevin is the corrector alone; ancestral DDPM sampling is the predictor alone. At equal compute, PC beat both in their CIFAR-10 experiments. The probability-flow ODE gives three further things. It is a deterministic map from noise to data, so latents can be interpolated and encoded. Adaptive black-box ODE solvers trade accuracy for steps. And the model has an exact likelihood through the instantaneous change of variables, , with the divergence estimated by Hutchinson–Skilling. Later fast samplers such as Denoising diffusion implicit models (DDIM) are solvers of this ODE.
Score SDE results (Song et al., ICLR 2021)
- CIFAR-10 samples
- NCSN++ cont. (deep, VE SDE): FID 2.20, Inception score 9.89, unconditional, then a record
- CIFAR-10 likelihood
- DDPM++ cont. (deep, sub-VP SDE) with the probability-flow ODE: 2.99 bits/dim
- High resolution
- 1024×1024 CelebA-HQ samples, the first from a score-based model
- Controllable generation
- class-conditional sampling, inpainting and colourisation with an unconditional model, by adding to the score
Every sampler uses the exact score and the same 1000 score evaluations; only the noise schedule and the update differ. σ_min = 0.3, ε = 0.05, 300 samples.
Worked example: one score at three noise levels
Take the same mixture and the point , slightly on the heavy side of the boundary . Each component of is . With responsibilities , the score is .
| σ | variance 1 + σ² | γ of the (5, 5) mode | ∇log p_σ(1, 0) | σ·‖∇log p_σ‖ | Tweedie E[x | x̃] |
|---|---|---|---|---|---|
| 0.3 | 1.09 | 0.999974 | (3.670, 4.587) | 1.762 | (1.330, 0.413) |
| 2 | 5 | 0.967273 | (0.735, 0.935) | 2.377 | (3.938, 3.738) |
| 20 | 401 | 0.803960 | (0.0051, 0.0076) | 0.183 | (3.035, 3.032) |
Reading the table
- 1At the light mode has responsibility . The score is that of the heavy component alone: it says nothing about the 0.2 weight.
- 2At the responsibility of the heavy mode is , almost the prior weight . Tweedie's denoised estimate is close to the overall mean . The weights are encoded in the field.
- 3Annealed Langevin at (level step ): the drift is and the noise has standard deviation per coordinate.
- 4At the step is , noise standard deviation : large enough to jump between the modes, which is how the sampler sets their proportions before the small levels refine the shape.
1import math, torch, torch.nn as nn23torch.manual_seed(0)4def sample_data(n): # 0.2·N((−5,−5), I) + 0.8·N((5,5), I)5 heavy = (torch.rand(n) < 0.8).float().unsqueeze(1)6 return torch.randn(n, 2) + (10 * heavy - 5) * torch.ones(n, 2)78sigmas = torch.logspace(math.log10(20), math.log10(0.3), 10) # σ_1 = 20 > … > σ_L = 0.3910class NCSN(nn.Module): # s_θ(x, σ) = f_θ(x, log σ) / σ11 def __init__(self, h=128):12 super().__init__()13 self.f = nn.Sequential(nn.Linear(3, h), nn.SiLU(), nn.Linear(h, h), nn.SiLU(), nn.Linear(h, 2))14 def forward(self, x, sigma): # x (N, 2), sigma (N, 1)15 return self.f(torch.cat([x / (1 + sigma**2).sqrt(), sigma.log()], 1)) / sigma1617def dsm_loss(model, x):18 idx = torch.randint(0, len(sigmas), (x.size(0),))19 sigma = sigmas[idx].unsqueeze(1) # (N, 1)20 z = torch.randn_like(x)21 x_tilde = x + sigma * z22 target = -z / sigma # ∇ log q_σ(x̃ | x) = −(x̃ − x)/σ²23 return 0.5 * ((sigma * (model(x_tilde, sigma) - target)) ** 2).sum(1).mean() # λ(σ) = σ²2425model = NCSN()26opt = torch.optim.Adam(model.parameters(), lr=1e-3)27for step in range(3000):28 loss = dsm_loss(model, sample_data(512))29 opt.zero_grad(); loss.backward(); opt.step()30print(f"final DSM loss {loss.item():.3f}") # ≈ 0.39, mostly the irreducible residual, not 03132@torch.no_grad()33def annealed_langevin(model, n=2000, T=100, eps=0.05):34 x = torch.rand(n, 2) * 16 - 8 # uniform on [−8, 8]²35 for sigma in sigmas:36 alpha = eps * (sigma / sigmas[-1]) ** 2 # α_i = ε σ_i² / σ_L²37 s = torch.full((n, 1), sigma.item())38 for _ in range(T):39 x = x + 0.5 * alpha * model(x, s) + alpha.sqrt() * torch.randn_like(x)40 return x4142x = annealed_langevin(model)43print(x.shape) # torch.Size([2000, 2])44print(f"share in the (5, 5) mode: {(x.sum(1) > 0).float().mean():.3f}") # 0.801 in our run (target 0.8)Use when
- +No normalising constant, no adversarial game, no architectural invertibility constraint: any network that outputs a vector of the input's shape
- +Stable regression-style training (denoising score matching) that scales to high-resolution images
- +One trained score model supports many samplers: annealed Langevin, reverse SDE, predictor–corrector, probability-flow ODE
- +Exact likelihoods and a deterministic encoder through the probability-flow ODE; conditional generation by adding a likelihood score
Avoid when
- −Sampling needs hundreds to thousands of network evaluations unless distilled or solved with a fast solver
- −Quality depends on the noise schedule, range, step sizes and corrector settings
- −Plain (unannealed) Langevin with a learned score fails on multimodal or manifold data
- −Likelihood via the ODE needs a full ODE solve plus a trace estimator per data point
Check yourself
Why does annealed Langevin start at a large σ instead of running plain Langevin at the smallest σ?
Check yourself
What does the probability-flow ODE share with the reverse-time SDE?
Remember
- ✓Score needs no normalising constant; Langevin dynamics samples as .
- ✓Denoising score matching: minimise with ; the optimum is , and links it to noise prediction.
- ✓NCSN (Song & Ermon 2019): geometric noise levels and annealed Langevin with ; must span the distance between modes.
- ✓Score SDE (Song et al. 2021): reverse SDE (Anderson 1982) and probability-flow ODE ; NCSN = VE SDE, DDPM = VP SDE.
- ✓Predictor–corrector sampling, exact ODE likelihoods; CIFAR-10 FID 2.20 and 2.99 bits/dim, and 1024×1024 samples.