Definition
Consistency model A consistency model, from Yang Song, Prafulla Dhariwal, Mark Chen and Ilya Sutskever, "Consistency Models" (ICML 2023), is a network trained to approximate the consistency function of a diffusion model's probability-flow (PF) ODE. That function maps every point of a trajectory to the trajectory's origin . It satisfies self-consistency: outputs are equal for any two points on the same trajectory. It also satisfies the boundary condition . A consistency model generates in one step, with , and can trade compute for quality by alternating re-noising and mapping. It is trained either by consistency distillation (CD) from a pretrained diffusion model or by consistency training (CT) from data alone.
Diffusion models and Score-based generative models sample by solving an SDE or ODE in tens to thousands of network evaluations. Faster ODE solvers such as Denoising diffusion implicit models (DDIM) and the Heun sampler of EDM (Karras et al. 2022) still need around 10–80 steps for good quality. Distillation methods (Luhman and Luhman 2021, progressive distillation by Salimans and Ho 2022) cut this further, but each step of the student is trained to copy several steps of the teacher. Song et al. instead asked the network to predict the end of the trajectory from any point on it, which gives one-step generation directly. The output at time then serves as the training target for the output at the next, larger time. Their one-step CD model set a new state of the art among one-step methods on CIFAR-10 (FID 3.55) and ImageNet (FID 6.20). The same model supports zero-shot editing (inpainting, colorization, super-resolution) by changing the multistep sampler.
The consistency function
- the data distribution convolved with (the EDM convention )
- the denoiser ; Tweedie gives
- and in the paper; the ODE is stopped at for numerical stability, and is taken as the sample
- the consistency function: constant along each trajectory, the identity at
- a free-form network (the EDM / ADM U-Net), with inputs scaled as in EDM
- the boundary condition holds exactly for every ; it is built in, not learned
- data standard deviation assumed by the EDM preconditioning (images scaled to )
Without the boundary condition, would be perfectly self-consistent. The skip parameterisation rules out that collapse. It also means behaves like an EDM denoiser: at small the output is mostly the input, at large mostly the network's prediction. This is why a pretrained diffusion network is a good initialisation.
Consistency distillation and consistency training
- the Karras et al. discretisation into sub-intervals, dense at low noise
- a noised data point,
- one step of an ODE solver from down to with the pretrained score model (Heun in the experiments)
- a distance (squared , or LPIPS; LPIPS worked best on images) and a weight ( in the paper)
- the target network, an EMA of : ; gradients flow only through the online
Read the loss as one-step bootstrapping, like temporal-difference learning. The teacher moves the point one grid step closer to the data. The output of the target network there is treated as correct, and the online network at the noisier point is pulled toward it. Consistency propagates outward from , where the boundary condition makes exact. Theorem 1: if and the solver has local error , then . The distilled model is as accurate as running the teacher's solver along the grid, which it never actually runs at sampling time.
- the same noise at both levels: this replaces the teacher step, using the unbiased score estimate
- Theorem 2 (Euler solver, smooth metric ): consistency training is a valid stand-alone objective in the limit of a fine grid
- CT schedules: the number of grid points grows during training, from toward , and the EMA rate grows with it
CT targets are noisy. Two points sharing the same lie on the same straight noising line, but not on the same ODE trajectory. They agree on average only. A coarse grid early in training gives low variance and high bias. The grid is refined later to cut the bias. Sampling (Algorithm 1): . Then, for a decreasing set of times , add noise and map back, . Each extra step costs one evaluation and corrects errors of the previous map.
Setup in the paper
- Noise levels
- EDM: , , , ,
- Teachers
- EDM-trained diffusion models (NCSN++ architecture on CIFAR-10, ADM on ImageNet and LSUN); Heun solver for ; for CIFAR-10 CD
- Metric
- LPIPS for the main results, with squared and in ablations
- Optimiser
- Rectified Adam, no learning-rate decay, warm-up or weight decay; an extra EMA of the online weights is used for evaluation
- Datasets
- CIFAR-10, ImageNet , LSUN Bedroom and Cat at
- Editing
- zero-shot inpainting, colorization, super-resolution, denoising, interpolation and stroke-guided generation, by modifying Algorithm 1
| dataset | CD, 1 step | CD, 2 steps | CT, 1 step | CT, 2 steps |
|---|---|---|---|---|
| CIFAR-10 | 3.55 | 2.93 | 8.70 | 5.83 |
| ImageNet 64×64 | 6.20 | 4.70 | 13.0 | 11.1 |
| LSUN Bedroom 256×256 | 7.80 | 5.22 |
CD beat progressive distillation at one and two steps on all datasets. CT, trained without any pretrained model, was the best one-step generator outside adversarial methods at the time, but clearly behind CD. The follow-up "Improved Techniques for Training Consistency Models" (Song and Dhariwal, ICLR 2024) closed the gap: it removed the EMA from the target (), replaced LPIPS by a pseudo-Huber loss and used a log-normal noise schedule, reaching one-step FID 2.51 on CIFAR-10 and 3.25 on ImageNet with CT alone. Latent consistency models (Luo, Tan, Huang, Li and Zhao, 2023) distil Stable Diffusion in its latent space (see Latent diffusion models). They fold classifier-free guidance into an augmented PF-ODE and skip several grid steps per target, and they produce images in 2–4 steps after about 32 A100 GPU hours of distillation.
Worked example: one CT update on the N = 18 grid
| t | c_skip(t) | c_out(t) |
|---|---|---|
| 0.002 (= ε) | 1.0000 | 0.0000 |
| 0.1 | 0.9630 | 0.0961 |
| 0.5 | 0.5020 | 0.3521 |
| 1 | 0.2006 | 0.4463 |
| 80 (= T) | 0.0000 | 0.5000 |
From a data point to the CT loss
- 1Grid for , : , and . Pick the interval .
- 2Data , noise (the same for both points): and .
- 3Coefficients: , ; , .
- 4Suppose the online network outputs and the target network . Then and .
- 5With squared , the loss is . The gradient moves only toward . Then .
- 6For CD, replace by one Heun step of the teacher from to . That point lies on the same ODE trajectory as , up to solver error.
1import copy, math, torch, torch.nn as nn23EPS, T, RHO, SIGMA_D = 0.002, 80.0, 7.0, 0.545def karras_grid(N): # t_1 = eps < ... < t_N = T6 i = torch.arange(N) / (N - 1)7 return (EPS ** (1 / RHO) + i * (T ** (1 / RHO) - EPS ** (1 / RHO))) ** RHO89class ConsistencyModel(nn.Module):10 def __init__(self, d=2, h=256):11 super().__init__()12 self.F = nn.Sequential(nn.Linear(d + 1, h), nn.SiLU(), nn.Linear(h, h), nn.SiLU(),13 nn.Linear(h, h), nn.SiLU(), nn.Linear(h, d))1415 def forward(self, x, t): # x (B, d), t (B,)16 c_skip = SIGMA_D ** 2 / ((t - EPS) ** 2 + SIGMA_D ** 2)17 c_out = SIGMA_D * (t - EPS) / (SIGMA_D ** 2 + t ** 2).sqrt()18 c_in = 1 / (SIGMA_D ** 2 + t ** 2).sqrt()19 F = self.F(torch.cat([c_in[:, None] * x, t.log()[:, None] / 4], -1))20 return c_skip[:, None] * x + c_out[:, None] * F # f(x, eps) = x exactly2122# Toy data: 4 Gaussians. Its denoiser D(x, t) = E[x_0 | x_t = x] is exact, so it stands in for a23# pretrained diffusion model (the "teacher" of consistency distillation).24modes, S = torch.tensor([[1., 1.], [-1., 1.], [-1., -1.], [1., -1.]]) * 0.6, 0.0525data = lambda n: modes[torch.randint(0, 4, (n,))] + S * torch.randn(n, 2)2627def denoiser(x, t):28 v = S ** 2 + t[:, None, None] ** 2 # (B, 1, 1)29 logw = -((x[:, None] - modes) ** 2).sum(-1, keepdim=True) / (2 * v)30 w = logw.softmax(1) # posterior over modes31 return (w * (modes + S ** 2 / v * (x[:, None] - modes))).sum(1)3233def heun(x, t, s): # one PF-ODE step dx/dt = (x - D(x, t)) / t34 d = (x - denoiser(x, t)) / t[:, None]35 xe = x + (s - t)[:, None] * d36 d2 = (xe - denoiser(xe, s)) / s[:, None]37 return x + (s - t)[:, None] * (d + d2) / 23839def cd_loss(model, target, x0, ts):40 n = torch.randint(0, len(ts) - 1, (x0.size(0),)) # adjacent pair (t_n, t_{n+1})41 t_hi, t_lo = ts[n + 1], ts[n]42 x_hi = x0 + t_hi[:, None] * torch.randn_like(x0)43 with torch.no_grad():44 tgt = target(heun(x_hi, t_hi, t_lo), t_lo) # teacher step, then EMA target network45 return ((model(x_hi, t_hi) - tgt) ** 2).sum(-1).mean() # d = squared L2 (LPIPS for images)4647def ct_loss(model, target, x0, ts): # consistency training: no teacher48 n = torch.randint(0, len(ts) - 1, (x0.size(0),))49 t_hi, t_lo, z = ts[n + 1], ts[n], torch.randn_like(x0) # the SAME z at both times50 with torch.no_grad():51 tgt = target(x0 + t_lo[:, None] * z, t_lo)52 return ((model(x0 + t_hi[:, None] * z, t_hi) - tgt) ** 2).sum(-1).mean()5354@torch.no_grad()55def sample(model, n, taus=()): # Algorithm 1: one step, then optional refinements56 x = model(T * torch.randn(n, 2), torch.full((n,), T))57 for tau in taus:58 x = x + math.sqrt(tau ** 2 - EPS ** 2) * torch.randn_like(x)59 x = model(x, torch.full((n,), tau))60 return x6162torch.manual_seed(0)63model = ConsistencyModel()64target = copy.deepcopy(model).requires_grad_(False)65opt = torch.optim.Adam(model.parameters(), lr=1e-3)66ts, mu = karras_grid(18), 0.9567for k in range(3000):68 loss = cd_loss(model, target, data(512), ts)69 opt.zero_grad(); loss.backward(); opt.step()70 with torch.no_grad():71 for p, q in zip(target.parameters(), model.parameters()):72 p.mul_(mu).add_(q, alpha=1 - mu) # theta^- <- stopgrad(mu theta^- + (1 - mu) theta)7374x = torch.randn(4, 2)75print(model(x, torch.full((4,), EPS)).sub(x).abs().max().item()) # 0.0: the boundary condition is built in76one, two = sample(model, 2000), sample(model, 2000, taus=(0.8,))77print(one.shape, two.shape) # torch.Size([2000, 2]) torch.Size([2000, 2])78near = lambda s: (s[:, None] - modes).norm(dim=-1).min(-1).values.mean().item()79print(round(near(one), 3), round(near(two), 3)) # mean distance to the nearest mode, 1 vs 2 stepsUse when
- +One network evaluation per sample; quality improves with a few more steps (2-step FID 2.93 on CIFAR-10)
- +Distils any diffusion or flow ODE (CD), or trains from scratch (CT)
- +Zero-shot editing (inpainting, colorization, super-resolution) by modifying the sampler
- +Deterministic map from noise to data at every noise level
Avoid when
- −CD inherits the teacher's errors and the grid's discretisation error
- −Training is delicate: grid schedules, EMA rates, metric (LPIPS) and weighting matter a lot
- −Quality saturates after a few steps; it does not improve like a full ODE solve
- −No exact likelihood, unlike the underlying ODE
Check yourself
What enforces f(x, ε) = x in a consistency model?
Check yourself
In consistency training, the two inputs x + t_{n+1}z and x + t_n z use the same z. Why?
Remember
- ✓Consistency model: for every point of a PF-ODE trajectory, with and ; one-step sample , .
- ✓Training pulls toward the EMA target : CD gets from one teacher ODE step, CT from the same noise at the lower level.
- ✓Theorem 1: zero CD loss with a -th order solver gives ; grid from Karras et al. with .
- ✓Multistep sampling alternates re-noising to τ and mapping back with f_θ, one evaluation per step.
- ✓FID (1 / 2 steps): CIFAR-10 CD 3.55 / 2.93, CT 8.70 / 5.83; ImageNet 64 CD 6.20 / 4.70. Latent consistency models apply CD to Stable Diffusion for 2–4-step text-to-image.