# Consistency models

> A consistency model learns f(xₜ, t) = x_ε, the end point of the probability-flow ODE trajectory through xₜ, by forcing outputs at adjacent noise levels on one trajectory to agree. Then a single evaluation f(x_T, T) turns noise into a sample, and a few re-noise-and-map steps refine it.

Source: https://www.mlfoundry.in/learn/consistency-models  
Section: Deep Learning › Generative Deep Learning Algorithms  
Level: advanced · 15 min read · updated 2026-10-12

**Consistency model:** A consistency model, from Yang Song, Prafulla Dhariwal, Mark Chen and Ilya Sutskever, "Consistency Models" (ICML 2023), is a network $f_\theta(x,t)$ trained to approximate the **consistency function** of a diffusion model's probability-flow (PF) ODE. That function maps every point $x_t$ of a trajectory $\{x_t\}_{t\in[\epsilon,T]}$ to the trajectory's origin $x_\epsilon$. It satisfies **self-consistency**: outputs are equal for any two points on the same trajectory. It also satisfies the **boundary condition** $f(x_\epsilon,\epsilon)=x_\epsilon$. A consistency model generates in one step, $x_\epsilon=f_\theta(x_T,T)$ with $x_T\sim\mathcal N(0,T^2I)$, and can trade compute for quality by alternating re-noising and mapping. It is trained either by **consistency distillation** (CD) from a pretrained diffusion model or by **consistency training** (CT) from data alone.

**Diffusion models** and **Score-based generative models** sample by solving an SDE or ODE in tens to thousands of network evaluations. Faster ODE solvers such as **Denoising diffusion implicit models (DDIM)** and the Heun sampler of EDM (Karras et al. 2022) still need around 10–80 steps for good quality. Distillation methods (Luhman and Luhman 2021, progressive distillation by Salimans and Ho 2022) cut this further, but each step of the student is trained to copy several steps of the teacher. Song et al. instead asked the network to predict the end of the trajectory from **any** point on it, which gives one-step generation directly. The output at time $t$ then serves as the training target for the output at the next, larger time. Their one-step CD model set a new state of the art among one-step methods on CIFAR-10 (FID 3.55) and ImageNet $64\times64$ (FID 6.20). The same model supports zero-shot editing (inpainting, colorization, super-resolution) by changing the multistep sampler.

## The consistency function

$$
\frac{dx_t}{dt}=-t\,\nabla_x\log p_t(x_t)=\frac{x_t-D(x_t,t)}{t},\quad t\in[\epsilon,T],\qquad f(x_t,t)=x_\epsilon\ \ \forall t,\qquad f(x_\epsilon,\epsilon)=x_\epsilon
$$

- $p_t$: the data distribution convolved with $\mathcal N(0,t^2I)$ (the EDM convention $\sigma(t)=t$)
- $D(x,t)$: the denoiser $\mathbb E[x_0\mid x_t=x]$; Tweedie gives $\nabla\log p_t=(D-x)/t^2$
- $\epsilon,\ T$: $\epsilon=0.002$ and $T=80$ in the paper; the ODE is stopped at $\epsilon$ for numerical stability, and $x_\epsilon$ is taken as the sample
- $f$: the consistency function: constant along each trajectory, the identity at $t=\epsilon$

$$
f_\theta(x,t)=c_{\rm skip}(t)\,x+c_{\rm out}(t)\,F_\theta(x,t),\qquad c_{\rm skip}(t)=\frac{\sigma_{\rm data}^2}{(t-\epsilon)^2+\sigma_{\rm data}^2},\qquad c_{\rm out}(t)=\frac{\sigma_{\rm data}(t-\epsilon)}{\sqrt{\sigma_{\rm data}^2+t^2}}
$$

- $F_\theta$: a free-form network (the EDM / ADM U-Net), with inputs scaled as in EDM
- $c_{\rm skip}(\epsilon)=1,\ c_{\rm out}(\epsilon)=0$: the boundary condition holds exactly for every $\theta$; it is built in, not learned
- $\sigma_{\rm data}=0.5$: data standard deviation assumed by the EDM preconditioning (images scaled to $[-1,1]$)

Without the boundary condition, $f\equiv0$ would be perfectly self-consistent. The skip parameterisation rules out that collapse. It also means $f_\theta$ behaves like an EDM denoiser: at small $t$ the output is mostly the input, at large $t$ mostly the network's prediction. This is why a pretrained diffusion network is a good initialisation.

## Consistency distillation and consistency training

$$
t_i=\Big(\epsilon^{1/\rho}+\tfrac{i-1}{N-1}\big(T^{1/\rho}-\epsilon^{1/\rho}\big)\Big)^{\rho},\ \ \rho=7\qquad \mathcal L^N_{\rm CD}=\mathbb E\Big[\lambda(t_n)\,d\big(f_\theta(x_{t_{n+1}},t_{n+1}),\,f_{\theta^-}(\hat x^{\phi}_{t_n},t_n)\big)\Big]
$$

- $t_1=\epsilon<\dots<t_N=T$: the Karras et al. discretisation into $N-1$ sub-intervals, dense at low noise
- $x_{t_{n+1}}=x+t_{n+1}z$: a noised data point, $n\sim\mathcal U\{1,\dots,N-1\}$
- $\hat x^{\phi}_{t_n}$: one step of an ODE solver from $t_{n+1}$ down to $t_n$ with the pretrained score model $s_\phi$ (Heun in the experiments)
- $d,\ \lambda$: a distance (squared $\ell_2$, $\ell_1$ or LPIPS; LPIPS worked best on images) and a weight ($\lambda\equiv1$ in the paper)
- $\theta^-$: the **target network**, an EMA of $\theta$: $\theta^-\leftarrow\mathrm{stopgrad}(\mu\theta^-+(1-\mu)\theta)$; gradients flow only through the online $f_\theta$

Read the loss as one-step bootstrapping, like temporal-difference learning. The teacher moves the point one grid step closer to the data. The output of the target network there is treated as correct, and the online network at the noisier point is pulled toward it. Consistency propagates outward from $t=\epsilon$, where the boundary condition makes $f$ exact. Theorem 1: if $\mathcal L^N_{\rm CD}=0$ and the solver has local error $O((\Delta t)^{p+1})$, then $\sup|f_\theta-f|=O((\Delta t)^p)$. The distilled model is as accurate as running the teacher's solver along the grid, which it never actually runs at sampling time.

$$
\mathcal L^N_{\rm CT}=\mathbb E\Big[\lambda(t_n)\,d\big(f_\theta(x+t_{n+1}z,\,t_{n+1}),\ f_{\theta^-}(x+t_nz,\,t_n)\big)\Big],\qquad z\sim\mathcal N(0,I)
$$

- $x+t_nz$: the **same** noise $z$ at both levels: this replaces the teacher step, using the unbiased score estimate $\nabla\log p_t(x_t)\approx-(x_t-x)/t^2$
- $\mathcal L_{\rm CT}=\mathcal L_{\rm CD}+o(\Delta t)$: Theorem 2 (Euler solver, smooth metric $d$): consistency training is a valid stand-alone objective in the limit of a fine grid
- $N(k),\ \mu(k)$: CT schedules: the number of grid points grows during training, from $s_0=2$ toward $s_1$, and the EMA rate $\mu(k)=\exp(s_0\log\mu_0/N(k))$ grows with it

CT targets are noisy. Two points sharing the same $z$ lie on the same **straight** noising line, but not on the same ODE trajectory. They agree on average only. A coarse grid early in training gives low variance and high bias. The grid is refined later to cut the bias. **Sampling** (Algorithm 1): $x\leftarrow f_\theta(x_T,T)$. Then, for a decreasing set of times $\tau_1>\tau_2>\dots$, add noise $x_{\tau}=x+\sqrt{\tau^2-\epsilon^2}\,z$ and map back, $x\leftarrow f_\theta(x_\tau,\tau)$. Each extra step costs one evaluation and corrects errors of the previous map.

**Setup in the paper**

- **Noise levels:** EDM: $\sigma(t)=t$, $\epsilon=0.002$, $T=80$, $\rho=7$, $\sigma_{\rm data}=0.5$
- **Teachers:** EDM-trained diffusion models (NCSN++ architecture on CIFAR-10, ADM on ImageNet and LSUN); Heun solver for $\hat x^\phi_{t_n}$; $N=18$ for CIFAR-10 CD
- **Metric:** LPIPS for the main results, with squared $\ell_2$ and $\ell_1$ in ablations
- **Optimiser:** Rectified Adam, no learning-rate decay, warm-up or weight decay; an extra EMA of the online weights is used for evaluation
- **Datasets:** CIFAR-10, ImageNet $64\times64$, LSUN Bedroom and Cat at $256\times256$
- **Editing:** zero-shot inpainting, colorization, super-resolution, denoising, interpolation and stroke-guided generation, by modifying Algorithm 1

FID ↓ with 1 and 2 network evaluations (Song et al. 2023, Tables 1–2)

| dataset | CD, 1 step | CD, 2 steps | CT, 1 step | CT, 2 steps |
| --- | --- | --- | --- | --- |
| CIFAR-10 | 3.55 | 2.93 | 8.70 | 5.83 |
| ImageNet 64×64 | 6.20 | 4.70 | 13.0 | 11.1 |
| LSUN Bedroom 256×256 | 7.80 | 5.22 |  |  |

CD beat progressive distillation at one and two steps on all datasets. CT, trained without any pretrained model, was the best one-step generator outside adversarial methods at the time, but clearly behind CD. The follow-up "Improved Techniques for Training Consistency Models" (Song and Dhariwal, ICLR 2024) closed the gap: it removed the EMA from the target ($\mu=0$), replaced LPIPS by a pseudo-Huber loss and used a log-normal noise schedule, reaching one-step FID 2.51 on CIFAR-10 and 3.25 on ImageNet $64^2$ with CT alone. Latent consistency models (Luo, Tan, Huang, Li and Zhao, 2023) distil **Stable Diffusion** in its latent space (see **Latent diffusion models**). They fold classifier-free guidance into an augmented PF-ODE and skip several grid steps per target, and they produce $768\times768$ images in 2–4 steps after about 32 A100 GPU hours of distillation.

> Figure: 1-D data: three Gaussians at −0.6, 0.1 and 0.7 (std 0.08, weights 0.35/0.25/0.4). The denoiser is exact, so the PF-ODE trajectories (25 paths, y-axis scaled by 1/√(t² + σ_d²)) and the true consistency function f* are exact. A 'perfectly distilled' model with grid N is one with zero CD loss, i.e. the chain of teacher Heun steps along the Karras grid (dashed). For the selected point (t = 1.009, x = 1.031) f* = 0.6939. With N = 10 the model returns 0.7481 (error 0.054). With N = 40 at t = 20.8 the error is 0.0022. That is Theorem 1: the error is the teacher solver's discretisation error. One-step samples (600 points) land within 3σ of a mode for 23.2% with N = 3, 53.3% with N = 5, 96.3% with N = 10 and 99.5% with N = 18. Multistep sampling repairs a coarse model: with N = 5, 2 steps give 74.7% and 4 steps 92.2%. The bars compare the two training targets on the grid interval containing t. For N = 10 on [0.470, 1.502], the RMS distance of the CD target from the exact online value is 0.11 and that of the CT target (same z) 0.39. For N = 40 at t ≈ 20 they are 1.2e−6 and 0.020: CD targets are accurate, CT targets are unbiased but noisy. W1 between two 600-point samples has a floor of about 0.02–0.04 (exact f*: 0.039).

## Worked example: one CT update on the N = 18 grid

Skip parameterisation with σ_data = 0.5, ε = 0.002 (computed in Node)

| t | c_skip(t) | c_out(t) |
| --- | --- | --- |
| 0.002 (= ε) | 1.0000 | 0.0000 |
| 0.1 | 0.9630 | 0.0961 |
| 0.5 | 0.5020 | 0.3521 |
| 1 | 0.2006 | 0.4463 |
| 80 (= T) | 0.0000 | 0.5000 |

**From a data point to the CT loss**

1. Grid for $N=18$, $\rho=7$: $t_1,\dots,t_4=0.0020,\ 0.0075,\ 0.0229,\ 0.0599$, and $t_{15},\dots,t_{18}=28.37,\ 40.79,\ 57.59,\ 80$. Pick the interval $[t_7,t_8]=[0.5853,\ 1.0882]$.
2. Data $x=0.7$, noise $z=1.2$ (the same $z$ for both points): $x_{t_8}=0.7+1.0882\cdot1.2=2.0058$ and $x_{t_7}=0.7+0.5853\cdot1.2=1.4024$.
3. Coefficients: $c_{\rm skip}(1.0882)=0.1749$, $c_{\rm out}(1.0882)=0.4535$; $c_{\rm skip}(0.5853)=0.4235$, $c_{\rm out}(0.5853)=0.3789$.
4. Suppose the online network outputs $F_\theta=0.10$ and the target network $F_{\theta^-}=0.30$. Then $f_\theta=0.1749\cdot2.0058+0.4535\cdot0.10=0.3961$ and $f_{\theta^-}=0.4235\cdot1.4024+0.3789\cdot0.30=0.7076$.
5. With squared $\ell_2$, the loss is $(0.3961-0.7076)^2=0.0971$. The gradient moves only $f_\theta$ toward $0.7076$. Then $\theta^-\leftarrow\mu\theta^-+(1-\mu)\theta$.
6. For CD, replace $x_{t_7}$ by one Heun step of the teacher from $(2.0058,\ 1.0882)$ to $t_7$. That point lies on the same ODE trajectory as $x_{t_8}$, up to solver error.

```python
import copy, math, torch, torch.nn as nn

EPS, T, RHO, SIGMA_D = 0.002, 80.0, 7.0, 0.5

def karras_grid(N):                                    # t_1 = eps < ... < t_N = T
    i = torch.arange(N) / (N - 1)
    return (EPS ** (1 / RHO) + i * (T ** (1 / RHO) - EPS ** (1 / RHO))) ** RHO

class ConsistencyModel(nn.Module):
    def __init__(self, d=2, h=256):
        super().__init__()
        self.F = nn.Sequential(nn.Linear(d + 1, h), nn.SiLU(), nn.Linear(h, h), nn.SiLU(),
                               nn.Linear(h, h), nn.SiLU(), nn.Linear(h, d))

    def forward(self, x, t):                           # x (B, d), t (B,)
        c_skip = SIGMA_D ** 2 / ((t - EPS) ** 2 + SIGMA_D ** 2)
        c_out = SIGMA_D * (t - EPS) / (SIGMA_D ** 2 + t ** 2).sqrt()
        c_in = 1 / (SIGMA_D ** 2 + t ** 2).sqrt()
        F = self.F(torch.cat([c_in[:, None] * x, t.log()[:, None] / 4], -1))
        return c_skip[:, None] * x + c_out[:, None] * F   # f(x, eps) = x exactly

# Toy data: 4 Gaussians. Its denoiser D(x, t) = E[x_0 | x_t = x] is exact, so it stands in for a
# pretrained diffusion model (the "teacher" of consistency distillation).
modes, S = torch.tensor([[1., 1.], [-1., 1.], [-1., -1.], [1., -1.]]) * 0.6, 0.05
data = lambda n: modes[torch.randint(0, 4, (n,))] + S * torch.randn(n, 2)

def denoiser(x, t):
    v = S ** 2 + t[:, None, None] ** 2                               # (B, 1, 1)
    logw = -((x[:, None] - modes) ** 2).sum(-1, keepdim=True) / (2 * v)
    w = logw.softmax(1)                                              # posterior over modes
    return (w * (modes + S ** 2 / v * (x[:, None] - modes))).sum(1)

def heun(x, t, s):                                     # one PF-ODE step dx/dt = (x - D(x, t)) / t
    d = (x - denoiser(x, t)) / t[:, None]
    xe = x + (s - t)[:, None] * d
    d2 = (xe - denoiser(xe, s)) / s[:, None]
    return x + (s - t)[:, None] * (d + d2) / 2

def cd_loss(model, target, x0, ts):
    n = torch.randint(0, len(ts) - 1, (x0.size(0),))   # adjacent pair (t_n, t_{n+1})
    t_hi, t_lo = ts[n + 1], ts[n]
    x_hi = x0 + t_hi[:, None] * torch.randn_like(x0)
    with torch.no_grad():
        tgt = target(heun(x_hi, t_hi, t_lo), t_lo)     # teacher step, then EMA target network
    return ((model(x_hi, t_hi) - tgt) ** 2).sum(-1).mean()   # d = squared L2 (LPIPS for images)

def ct_loss(model, target, x0, ts):                    # consistency training: no teacher
    n = torch.randint(0, len(ts) - 1, (x0.size(0),))
    t_hi, t_lo, z = ts[n + 1], ts[n], torch.randn_like(x0)        # the SAME z at both times
    with torch.no_grad():
        tgt = target(x0 + t_lo[:, None] * z, t_lo)
    return ((model(x0 + t_hi[:, None] * z, t_hi) - tgt) ** 2).sum(-1).mean()

@torch.no_grad()
def sample(model, n, taus=()):                         # Algorithm 1: one step, then optional refinements
    x = model(T * torch.randn(n, 2), torch.full((n,), T))
    for tau in taus:
        x = x + math.sqrt(tau ** 2 - EPS ** 2) * torch.randn_like(x)
        x = model(x, torch.full((n,), tau))
    return x

torch.manual_seed(0)
model = ConsistencyModel()
target = copy.deepcopy(model).requires_grad_(False)
opt = torch.optim.Adam(model.parameters(), lr=1e-3)
ts, mu = karras_grid(18), 0.95
for k in range(3000):
    loss = cd_loss(model, target, data(512), ts)
    opt.zero_grad(); loss.backward(); opt.step()
    with torch.no_grad():
        for p, q in zip(target.parameters(), model.parameters()):
            p.mul_(mu).add_(q, alpha=1 - mu)           # theta^- <- stopgrad(mu theta^- + (1 - mu) theta)

x = torch.randn(4, 2)
print(model(x, torch.full((4,), EPS)).sub(x).abs().max().item())   # 0.0: the boundary condition is built in
one, two = sample(model, 2000), sample(model, 2000, taus=(0.8,))
print(one.shape, two.shape)                            # torch.Size([2000, 2]) torch.Size([2000, 2])
near = lambda s: (s[:, None] - modes).norm(dim=-1).min(-1).values.mean().item()
print(round(near(one), 3), round(near(two), 3))        # mean distance to the nearest mode, 1 vs 2 steps
```

Consistency distillation on a 2-D toy (exact teacher denoiser), with the CT loss for comparison and multistep sampling

> **Self-consistency alone is trivial:** $f\equiv\text{const}$ makes every consistency loss zero. The constraint that pins the solution is the boundary $f(x,\epsilon)=x$, and it must hold **exactly**. Hence the $c_{\rm skip}/c_{\rm out}$ form with $c_{\rm out}(\epsilon)=0$, not a penalty term. Likewise, never backpropagate through the target $f_{\theta^-}$. With both sides trainable, the network can lower the loss by making $f$ flat in $t$ instead of correct.

> **Grid size is a bias–variance knob:** A coarse grid (small $N$) puts a large solver error into every CD target, and the model inherits it: in the figure the $N=3$ model puts only 23% of samples on a mode. A fine grid makes adjacent targets almost identical, so the signal per step is tiny and errors propagate slowly from $\epsilon$ to $T$. CD on CIFAR-10 used $N=18$. CT increases $N$ during training, starting at $s_0=2$.

> **Why one step works at all:** Self-consistency turns an ODE-solving problem into a regression problem on pairs of nearby points. The network never integrates anything at test time. It has learned the integral, much as a value function in reinforcement learning learns a sum of future rewards by bootstrapping, with the EMA target network playing the same role as in DQN. Compare **Rectified flow**, which instead straightens the ODE so that integration becomes trivial. **Flow matching** models can be distilled into consistency models in the same way.

**Strengths**

- One network evaluation per sample; quality improves with a few more steps (2-step FID 2.93 on CIFAR-10)
- Distils any diffusion or flow ODE (CD), or trains from scratch (CT)
- Zero-shot editing (inpainting, colorization, super-resolution) by modifying the sampler
- Deterministic map from noise to data at every noise level

**Limitations**

- CD inherits the teacher's errors and the grid's discretisation error
- Training is delicate: grid schedules, EMA rates, metric (LPIPS) and weighting matter a lot
- Quality saturates after a few steps; it does not improve like a full ODE solve
- No exact likelihood, unlike the underlying ODE

**Question:** What enforces f(x, ε) = x in a consistency model?

- A. An extra loss term that penalises f(x, ε) − x
- B. The parameterisation f = c_skip(t)x + c_out(t)F with c_skip(ε) = 1 and c_out(ε) = 0
- C. The EMA target network
- D. Training only on t = ε

**Answer:** B. The parameterisation f = c_skip(t)x + c_out(t)F with c_skip(ε) = 1 and c_out(ε) = 0 — The boundary condition is built into the architecture, so it holds for every $\theta$. Without it, a constant function would satisfy self-consistency perfectly. The EMA target stabilises the bootstrapped regression but does not anchor the solution.

**Question:** In consistency training, the two inputs x + t_{n+1}z and x + t_n z use the same z. Why?

- A. To make the loss exactly zero at the optimum
- B. Because they then lie on the same PF-ODE trajectory
- C. Because this is an unbiased (single-sample) substitute for the teacher's ODE step, which is exact as Δt → 0
- D. To reduce memory use

**Answer:** C. Because this is an unbiased (single-sample) substitute for the teacher's ODE step, which is exact as Δt → 0 — With the same $z$, the pair lies on one straight noising line. Its direction gives the one-sample score estimate $-(x_t-x)/t^2$, so the target approximates the teacher step on average. The points are not on the same ODE trajectory, which is why CT targets are noisy (0.39 against 0.11 RMS for CD on one interval in the figure). $\mathcal L_{\rm CT}=\mathcal L_{\rm CD}+o(\Delta t)$ (Theorem 2).

**Key points**

- Consistency model: $f_\theta(x_t,t)\approx x_\epsilon$ for every point of a PF-ODE trajectory, with $f_\theta=c_{\rm skip}(t)x+c_{\rm out}(t)F_\theta$ and $c_{\rm skip}(\epsilon)=1,\ c_{\rm out}(\epsilon)=0$; one-step sample $f_\theta(x_T,T)$, $x_T\sim\mathcal N(0,T^2I)$.
- Training pulls $f_\theta(x_{t_{n+1}},t_{n+1})$ toward the EMA target $f_{\theta^-}(x_{t_n},t_n)$: CD gets $x_{t_n}$ from one teacher ODE step, CT from the same noise $z$ at the lower level.
- Theorem 1: zero CD loss with a $p$-th order solver gives $\sup|f_\theta-f|=O((\Delta t)^p)$; grid $t_i$ from Karras et al. with $\epsilon=0.002,\ T=80,\ \rho=7$.
- Multistep sampling alternates re-noising to τ and mapping back with f_θ, one evaluation per step.
- FID (1 / 2 steps): CIFAR-10 CD 3.55 / 2.93, CT 8.70 / 5.83; ImageNet 64 CD 6.20 / 4.70. Latent consistency models apply CD to Stable Diffusion for 2–4-step text-to-image.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
