Consistency models

A consistency model learns f(xₜ, t) = x_ε, the end point of the probability-flow ODE trajectory through xₜ, by forcing outputs at adjacent noise levels on one trajectory to agree. Then a single evaluation f(x_T, T) turns noise into a sample, and a few re-noise-and-map steps refine it.

15 min read1069 wordsadvanced0 viewsupdated 2026-10-12

Definition

Consistency model A consistency model, from Yang Song, Prafulla Dhariwal, Mark Chen and Ilya Sutskever, "Consistency Models" (ICML 2023), is a network fθ(x,t)f_\theta(x,t) trained to approximate the consistency function of a diffusion model's probability-flow (PF) ODE. That function maps every point xtx_t of a trajectory {xt}t∈[ϵ,T]\{x_t\}_{t\in[\epsilon,T]} to the trajectory's origin xϵx_\epsilon. It satisfies self-consistency: outputs are equal for any two points on the same trajectory. It also satisfies the boundary condition f(xϵ,ϵ)=xϵf(x_\epsilon,\epsilon)=x_\epsilon. A consistency model generates in one step, xϵ=fθ(xT,T)x_\epsilon=f_\theta(x_T,T) with xT∼N(0,T2I)x_T\sim\mathcal N(0,T^2I), and can trade compute for quality by alternating re-noising and mapping. It is trained either by consistency distillation (CD) from a pretrained diffusion model or by consistency training (CT) from data alone.

Diffusion models and Score-based generative models sample by solving an SDE or ODE in tens to thousands of network evaluations. Faster ODE solvers such as Denoising diffusion implicit models (DDIM) and the Heun sampler of EDM (Karras et al. 2022) still need around 10–80 steps for good quality. Distillation methods (Luhman and Luhman 2021, progressive distillation by Salimans and Ho 2022) cut this further, but each step of the student is trained to copy several steps of the teacher. Song et al. instead asked the network to predict the end of the trajectory from any point on it, which gives one-step generation directly. The output at time tt then serves as the training target for the output at the next, larger time. Their one-step CD model set a new state of the art among one-step methods on CIFAR-10 (FID 3.55) and ImageNet 64×6464\times64 (FID 6.20). The same model supports zero-shot editing (inpainting, colorization, super-resolution) by changing the multistep sampler.

The consistency function

dxtdt=−t ∇xlog⁡pt(xt)=xt−D(xt,t)t,t∈[ϵ,T],f(xt,t)=xϵ  ∀t,f(xϵ,ϵ)=xϵ\frac{dx_t}{dt}=-t\,\nabla_x\log p_t(x_t)=\frac{x_t-D(x_t,t)}{t},\quad t\in[\epsilon,T],\qquad f(x_t,t)=x_\epsilon\ \ \forall t,\qquad f(x_\epsilon,\epsilon)=x_\epsilon
ptp_t
the data distribution convolved with N(0,t2I)\mathcal N(0,t^2I) (the EDM convention σ(t)=t\sigma(t)=t)
D(x,t)D(x,t)
the denoiser E[x0∣xt=x]\mathbb E[x_0\mid x_t=x]; Tweedie gives ∇log⁡pt=(D−x)/t2\nabla\log p_t=(D-x)/t^2
ϵ, T\epsilon,\ T
ϵ=0.002\epsilon=0.002 and T=80T=80 in the paper; the ODE is stopped at ϵ\epsilon for numerical stability, and xϵx_\epsilon is taken as the sample
ff
the consistency function: constant along each trajectory, the identity at t=ϵt=\epsilon
fθ(x,t)=cskip(t) x+cout(t) Fθ(x,t),cskip(t)=σdata2(t−ϵ)2+σdata2,cout(t)=σdata(t−ϵ)σdata2+t2f_\theta(x,t)=c_{\rm skip}(t)\,x+c_{\rm out}(t)\,F_\theta(x,t),\qquad c_{\rm skip}(t)=\frac{\sigma_{\rm data}^2}{(t-\epsilon)^2+\sigma_{\rm data}^2},\qquad c_{\rm out}(t)=\frac{\sigma_{\rm data}(t-\epsilon)}{\sqrt{\sigma_{\rm data}^2+t^2}}
FθF_\theta
a free-form network (the EDM / ADM U-Net), with inputs scaled as in EDM
cskip(ϵ)=1, cout(ϵ)=0c_{\rm skip}(\epsilon)=1,\ c_{\rm out}(\epsilon)=0
the boundary condition holds exactly for every θ\theta; it is built in, not learned
σdata=0.5\sigma_{\rm data}=0.5
data standard deviation assumed by the EDM preconditioning (images scaled to [−1,1][-1,1])

Without the boundary condition, f≡0f\equiv0 would be perfectly self-consistent. The skip parameterisation rules out that collapse. It also means fθf_\theta behaves like an EDM denoiser: at small tt the output is mostly the input, at large tt mostly the network's prediction. This is why a pretrained diffusion network is a good initialisation.

Consistency distillation and consistency training

ti=(ϵ1/ρ+i−1N−1(T1/ρ−ϵ1/ρ))ρ,  ρ=7LCDN=E[λ(tn) d(fθ(xtn+1,tn+1), fθ−(x^tnϕ,tn))]t_i=\Big(\epsilon^{1/\rho}+\tfrac{i-1}{N-1}\big(T^{1/\rho}-\epsilon^{1/\rho}\big)\Big)^{\rho},\ \ \rho=7\qquad \mathcal L^N_{\rm CD}=\mathbb E\Big[\lambda(t_n)\,d\big(f_\theta(x_{t_{n+1}},t_{n+1}),\,f_{\theta^-}(\hat x^{\phi}_{t_n},t_n)\big)\Big]
t1=ϵ<⋯<tN=Tt_1=\epsilon<\dots<t_N=T
the Karras et al. discretisation into N−1N-1 sub-intervals, dense at low noise
xtn+1=x+tn+1zx_{t_{n+1}}=x+t_{n+1}z
a noised data point, n∼U{1,…,N−1}n\sim\mathcal U\{1,\dots,N-1\}
x^tnϕ\hat x^{\phi}_{t_n}
one step of an ODE solver from tn+1t_{n+1} down to tnt_n with the pretrained score model sϕs_\phi (Heun in the experiments)
d, λd,\ \lambda
a distance (squared ℓ2\ell_2, ℓ1\ell_1 or LPIPS; LPIPS worked best on images) and a weight (λ≡1\lambda\equiv1 in the paper)
θ−\theta^-
the target network, an EMA of θ\theta: θ−←stopgrad(μθ−+(1−μ)θ)\theta^-\leftarrow\mathrm{stopgrad}(\mu\theta^-+(1-\mu)\theta); gradients flow only through the online fθf_\theta

Read the loss as one-step bootstrapping, like temporal-difference learning. The teacher moves the point one grid step closer to the data. The output of the target network there is treated as correct, and the online network at the noisier point is pulled toward it. Consistency propagates outward from t=ϵt=\epsilon, where the boundary condition makes ff exact. Theorem 1: if LCDN=0\mathcal L^N_{\rm CD}=0 and the solver has local error O((Δt)p+1)O((\Delta t)^{p+1}), then sup⁡∣fθ−f∣=O((Δt)p)\sup|f_\theta-f|=O((\Delta t)^p). The distilled model is as accurate as running the teacher's solver along the grid, which it never actually runs at sampling time.

LCTN=E[λ(tn) d(fθ(x+tn+1z, tn+1), fθ−(x+tnz, tn))],z∼N(0,I)\mathcal L^N_{\rm CT}=\mathbb E\Big[\lambda(t_n)\,d\big(f_\theta(x+t_{n+1}z,\,t_{n+1}),\ f_{\theta^-}(x+t_nz,\,t_n)\big)\Big],\qquad z\sim\mathcal N(0,I)
x+tnzx+t_nz
the same noise zz at both levels: this replaces the teacher step, using the unbiased score estimate ∇log⁡pt(xt)≈−(xt−x)/t2\nabla\log p_t(x_t)\approx-(x_t-x)/t^2
LCT=LCD+o(Δt)\mathcal L_{\rm CT}=\mathcal L_{\rm CD}+o(\Delta t)
Theorem 2 (Euler solver, smooth metric dd): consistency training is a valid stand-alone objective in the limit of a fine grid
N(k), μ(k)N(k),\ \mu(k)
CT schedules: the number of grid points grows during training, from s0=2s_0=2 toward s1s_1, and the EMA rate μ(k)=exp⁡(s0log⁡μ0/N(k))\mu(k)=\exp(s_0\log\mu_0/N(k)) grows with it

CT targets are noisy. Two points sharing the same zz lie on the same straight noising line, but not on the same ODE trajectory. They agree on average only. A coarse grid early in training gives low variance and high bias. The grid is refined later to cut the bias. Sampling (Algorithm 1): x←fθ(xT,T)x\leftarrow f_\theta(x_T,T). Then, for a decreasing set of times τ1>τ2>…\tau_1>\tau_2>\dots, add noise xτ=x+τ2−ϵ2 zx_{\tau}=x+\sqrt{\tau^2-\epsilon^2}\,z and map back, x←fθ(xτ,τ)x\leftarrow f_\theta(x_\tau,\tau). Each extra step costs one evaluation and corrects errors of the previous map.

Setup in the paper

Noise levels
EDM: σ(t)=t\sigma(t)=t, ϵ=0.002\epsilon=0.002, T=80T=80, ρ=7\rho=7, σdata=0.5\sigma_{\rm data}=0.5
Teachers
EDM-trained diffusion models (NCSN++ architecture on CIFAR-10, ADM on ImageNet and LSUN); Heun solver for x^tnϕ\hat x^\phi_{t_n}; N=18N=18 for CIFAR-10 CD
Metric
LPIPS for the main results, with squared ℓ2\ell_2 and ℓ1\ell_1 in ablations
Optimiser
Rectified Adam, no learning-rate decay, warm-up or weight decay; an extra EMA of the online weights is used for evaluation
Datasets
CIFAR-10, ImageNet 64×6464\times64, LSUN Bedroom and Cat at 256×256256\times256
Editing
zero-shot inpainting, colorization, super-resolution, denoising, interpolation and stroke-guided generation, by modifying Algorithm 1
datasetCD, 1 stepCD, 2 stepsCT, 1 stepCT, 2 steps
CIFAR-103.552.938.705.83
ImageNet 64×646.204.7013.011.1
LSUN Bedroom 256×2567.805.22
FID ↓ with 1 and 2 network evaluations (Song et al. 2023, Tables 1–2)

CD beat progressive distillation at one and two steps on all datasets. CT, trained without any pretrained model, was the best one-step generator outside adversarial methods at the time, but clearly behind CD. The follow-up "Improved Techniques for Training Consistency Models" (Song and Dhariwal, ICLR 2024) closed the gap: it removed the EMA from the target (μ=0\mu=0), replaced LPIPS by a pseudo-Huber loss and used a log-normal noise schedule, reaching one-step FID 2.51 on CIFAR-10 and 3.25 on ImageNet 64264^2 with CT alone. Latent consistency models (Luo, Tan, Huang, Li and Zhao, 2023) distil Stable Diffusion in its latent space (see Latent diffusion models). They fold classifier-free guidance into an augmented PF-ODE and skip several grid steps per target, and they produce 768×768768\times768 images in 2–4 steps after about 32 A100 GPU hours of distillation.

PF-ODE dx/dt = (x − D(x,t))/t · f(x_t, t) = x_ε on every patht = εt = T = 80 (log scale) · y = x/√(t² + σ_d²)- - teacher Heun chain on the N = 10 grid (vertical lines)■ 600 samples, 1 step · — data density · x ∈ [−1.2, 1.4]selected pointt 1.009 · x 1.031c_skip 0.198 · c_out 0.447f*(x, t) 0.6939f_θ, CD N = 10 0.7481|f_θ − f*| 0.0543target error, interval 4/9t_n → t_n+1 0.470 → 1.502CD (Heun step) RMS 0.1105CT (same z) RMS 0.3891sampling, 1 step, N = 10on a mode (±3σ) 96.3%W1 to data sample 0.104exact f*, 1 step: W1 0.039W1 of two 600-point samplesof the same data is ≈ 0.02–0.04.Coarse N: the chain's solvererror is baked into f_θ.
CD grid Nsampling steps
fig.1-D data: three Gaussians at −0.6, 0.1 and 0.7 (std 0.08, weights 0.35/0.25/0.4). The denoiser is exact, so the PF-ODE trajectories (25 paths, y-axis scaled by 1/√(t² + σ_d²)) and the true consistency function f* are exact. A 'perfectly distilled' model with grid N is one with zero CD loss, i.e. the chain of teacher Heun steps along the Karras grid (dashed). For the selected point (t = 1.009, x = 1.031) f* = 0.6939. With N = 10 the model returns 0.7481 (error 0.054). With N = 40 at t = 20.8 the error is 0.0022. That is Theorem 1: the error is the teacher solver's discretisation error. One-step samples (600 points) land within 3σ of a mode for 23.2% with N = 3, 53.3% with N = 5, 96.3% with N = 10 and 99.5% with N = 18. Multistep sampling repairs a coarse model: with N = 5, 2 steps give 74.7% and 4 steps 92.2%. The bars compare the two training targets on the grid interval containing t. For N = 10 on [0.470, 1.502], the RMS distance of the CD target from the exact online value is 0.11 and that of the CT target (same z) 0.39. For N = 40 at t ≈ 20 they are 1.2e−6 and 0.020: CD targets are accurate, CT targets are unbiased but noisy. W1 between two 600-point samples has a floor of about 0.02–0.04 (exact f*: 0.039).

Worked example: one CT update on the N = 18 grid

tc_skip(t)c_out(t)
0.002 (= ε)1.00000.0000
0.10.96300.0961
0.50.50200.3521
10.20060.4463
80 (= T)0.00000.5000
Skip parameterisation with σ_data = 0.5, ε = 0.002 (computed in Node)

From a data point to the CT loss

  1. 1Grid for N=18N=18, ρ=7\rho=7: t1,…,t4=0.0020, 0.0075, 0.0229, 0.0599t_1,\dots,t_4=0.0020,\ 0.0075,\ 0.0229,\ 0.0599, and t15,…,t18=28.37, 40.79, 57.59, 80t_{15},\dots,t_{18}=28.37,\ 40.79,\ 57.59,\ 80. Pick the interval [t7,t8]=[0.5853, 1.0882][t_7,t_8]=[0.5853,\ 1.0882].
  2. 2Data x=0.7x=0.7, noise z=1.2z=1.2 (the same zz for both points): xt8=0.7+1.0882⋅1.2=2.0058x_{t_8}=0.7+1.0882\cdot1.2=2.0058 and xt7=0.7+0.5853⋅1.2=1.4024x_{t_7}=0.7+0.5853\cdot1.2=1.4024.
  3. 3Coefficients: cskip(1.0882)=0.1749c_{\rm skip}(1.0882)=0.1749, cout(1.0882)=0.4535c_{\rm out}(1.0882)=0.4535; cskip(0.5853)=0.4235c_{\rm skip}(0.5853)=0.4235, cout(0.5853)=0.3789c_{\rm out}(0.5853)=0.3789.
  4. 4Suppose the online network outputs Fθ=0.10F_\theta=0.10 and the target network Fθ−=0.30F_{\theta^-}=0.30. Then fθ=0.1749⋅2.0058+0.4535⋅0.10=0.3961f_\theta=0.1749\cdot2.0058+0.4535\cdot0.10=0.3961 and fθ−=0.4235⋅1.4024+0.3789⋅0.30=0.7076f_{\theta^-}=0.4235\cdot1.4024+0.3789\cdot0.30=0.7076.
  5. 5With squared ℓ2\ell_2, the loss is (0.3961−0.7076)2=0.0971(0.3961-0.7076)^2=0.0971. The gradient moves only fθf_\theta toward 0.70760.7076. Then θ−←μθ−+(1−μ)θ\theta^-\leftarrow\mu\theta^-+(1-\mu)\theta.
  6. 6For CD, replace xt7x_{t_7} by one Heun step of the teacher from (2.0058, 1.0882)(2.0058,\ 1.0882) to t7t_7. That point lies on the same ODE trajectory as xt8x_{t_8}, up to solver error.
python
1import copy, math, torch, torch.nn as nn
2
3EPS, T, RHO, SIGMA_D = 0.002, 80.0, 7.0, 0.5
4
5def karras_grid(N): # t_1 = eps < ... < t_N = T
6 i = torch.arange(N) / (N - 1)
7 return (EPS ** (1 / RHO) + i * (T ** (1 / RHO) - EPS ** (1 / RHO))) ** RHO
8
9class ConsistencyModel(nn.Module):
10 def __init__(self, d=2, h=256):
11 super().__init__()
12 self.F = nn.Sequential(nn.Linear(d + 1, h), nn.SiLU(), nn.Linear(h, h), nn.SiLU(),
13 nn.Linear(h, h), nn.SiLU(), nn.Linear(h, d))
14
15 def forward(self, x, t): # x (B, d), t (B,)
16 c_skip = SIGMA_D ** 2 / ((t - EPS) ** 2 + SIGMA_D ** 2)
17 c_out = SIGMA_D * (t - EPS) / (SIGMA_D ** 2 + t ** 2).sqrt()
18 c_in = 1 / (SIGMA_D ** 2 + t ** 2).sqrt()
19 F = self.F(torch.cat([c_in[:, None] * x, t.log()[:, None] / 4], -1))
20 return c_skip[:, None] * x + c_out[:, None] * F # f(x, eps) = x exactly
21
22# Toy data: 4 Gaussians. Its denoiser D(x, t) = E[x_0 | x_t = x] is exact, so it stands in for a
23# pretrained diffusion model (the "teacher" of consistency distillation).
24modes, S = torch.tensor([[1., 1.], [-1., 1.], [-1., -1.], [1., -1.]]) * 0.6, 0.05
25data = lambda n: modes[torch.randint(0, 4, (n,))] + S * torch.randn(n, 2)
26
27def denoiser(x, t):
28 v = S ** 2 + t[:, None, None] ** 2 # (B, 1, 1)
29 logw = -((x[:, None] - modes) ** 2).sum(-1, keepdim=True) / (2 * v)
30 w = logw.softmax(1) # posterior over modes
31 return (w * (modes + S ** 2 / v * (x[:, None] - modes))).sum(1)
32
33def heun(x, t, s): # one PF-ODE step dx/dt = (x - D(x, t)) / t
34 d = (x - denoiser(x, t)) / t[:, None]
35 xe = x + (s - t)[:, None] * d
36 d2 = (xe - denoiser(xe, s)) / s[:, None]
37 return x + (s - t)[:, None] * (d + d2) / 2
38
39def cd_loss(model, target, x0, ts):
40 n = torch.randint(0, len(ts) - 1, (x0.size(0),)) # adjacent pair (t_n, t_{n+1})
41 t_hi, t_lo = ts[n + 1], ts[n]
42 x_hi = x0 + t_hi[:, None] * torch.randn_like(x0)
43 with torch.no_grad():
44 tgt = target(heun(x_hi, t_hi, t_lo), t_lo) # teacher step, then EMA target network
45 return ((model(x_hi, t_hi) - tgt) ** 2).sum(-1).mean() # d = squared L2 (LPIPS for images)
46
47def ct_loss(model, target, x0, ts): # consistency training: no teacher
48 n = torch.randint(0, len(ts) - 1, (x0.size(0),))
49 t_hi, t_lo, z = ts[n + 1], ts[n], torch.randn_like(x0) # the SAME z at both times
50 with torch.no_grad():
51 tgt = target(x0 + t_lo[:, None] * z, t_lo)
52 return ((model(x0 + t_hi[:, None] * z, t_hi) - tgt) ** 2).sum(-1).mean()
53
54@torch.no_grad()
55def sample(model, n, taus=()): # Algorithm 1: one step, then optional refinements
56 x = model(T * torch.randn(n, 2), torch.full((n,), T))
57 for tau in taus:
58 x = x + math.sqrt(tau ** 2 - EPS ** 2) * torch.randn_like(x)
59 x = model(x, torch.full((n,), tau))
60 return x
61
62torch.manual_seed(0)
63model = ConsistencyModel()
64target = copy.deepcopy(model).requires_grad_(False)
65opt = torch.optim.Adam(model.parameters(), lr=1e-3)
66ts, mu = karras_grid(18), 0.95
67for k in range(3000):
68 loss = cd_loss(model, target, data(512), ts)
69 opt.zero_grad(); loss.backward(); opt.step()
70 with torch.no_grad():
71 for p, q in zip(target.parameters(), model.parameters()):
72 p.mul_(mu).add_(q, alpha=1 - mu) # theta^- <- stopgrad(mu theta^- + (1 - mu) theta)
73
74x = torch.randn(4, 2)
75print(model(x, torch.full((4,), EPS)).sub(x).abs().max().item()) # 0.0: the boundary condition is built in
76one, two = sample(model, 2000), sample(model, 2000, taus=(0.8,))
77print(one.shape, two.shape) # torch.Size([2000, 2]) torch.Size([2000, 2])
78near = lambda s: (s[:, None] - modes).norm(dim=-1).min(-1).values.mean().item()
79print(round(near(one), 3), round(near(two), 3)) # mean distance to the nearest mode, 1 vs 2 steps
Consistency distillation on a 2-D toy (exact teacher denoiser), with the CT loss for comparison and multistep sampling

Use when

  • +One network evaluation per sample; quality improves with a few more steps (2-step FID 2.93 on CIFAR-10)
  • +Distils any diffusion or flow ODE (CD), or trains from scratch (CT)
  • +Zero-shot editing (inpainting, colorization, super-resolution) by modifying the sampler
  • +Deterministic map from noise to data at every noise level

Avoid when

  • −CD inherits the teacher's errors and the grid's discretisation error
  • −Training is delicate: grid schedules, EMA rates, metric (LPIPS) and weighting matter a lot
  • −Quality saturates after a few steps; it does not improve like a full ODE solve
  • −No exact likelihood, unlike the underlying ODE

Check yourself

What enforces f(x, ε) = x in a consistency model?

Check yourself

In consistency training, the two inputs x + t_{n+1}z and x + t_n z use the same z. Why?

Remember

  • ✓Consistency model: fθ(xt,t)≈xϵf_\theta(x_t,t)\approx x_\epsilon for every point of a PF-ODE trajectory, with fθ=cskip(t)x+cout(t)Fθf_\theta=c_{\rm skip}(t)x+c_{\rm out}(t)F_\theta and cskip(ϵ)=1, cout(ϵ)=0c_{\rm skip}(\epsilon)=1,\ c_{\rm out}(\epsilon)=0; one-step sample fθ(xT,T)f_\theta(x_T,T), xT∼N(0,T2I)x_T\sim\mathcal N(0,T^2I).
  • ✓Training pulls fθ(xtn+1,tn+1)f_\theta(x_{t_{n+1}},t_{n+1}) toward the EMA target fθ−(xtn,tn)f_{\theta^-}(x_{t_n},t_n): CD gets xtnx_{t_n} from one teacher ODE step, CT from the same noise zz at the lower level.
  • ✓Theorem 1: zero CD loss with a pp-th order solver gives sup⁡∣fθ−f∣=O((Δt)p)\sup|f_\theta-f|=O((\Delta t)^p); grid tit_i from Karras et al. with ϵ=0.002, T=80, ρ=7\epsilon=0.002,\ T=80,\ \rho=7.
  • ✓Multistep sampling alternates re-noising to τ and mapping back with f_θ, one evaluation per step.
  • ✓FID (1 / 2 steps): CIFAR-10 CD 3.55 / 2.93, CT 8.70 / 5.83; ImageNet 64 CD 6.20 / 4.70. Latent consistency models apply CD to Stable Diffusion for 2–4-step text-to-image.
← back to Deep Learning roadmap