Rectified flow

Rectified flow learns an ODE dZₜ = v(Zₜ, t)dt by regressing v on X₁ − X₀ along straight lines Xₜ = tX₁ + (1 − t)X₀. Reflow, retraining on the model's own (noise, sample) pairs, makes the paths straighter, until one Euler step, or a distilled one-step model, generates good samples.

14 min read985 wordsadvanced0 viewsupdated 2026-10-12

Definition

Rectified flow is a method for learning a transport map between two distributions π0\pi_0 and π1\pi_1 from samples, introduced by Xingchao Liu, Chengyue Gong and Qiang Liu, "Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow" (ICLR 2023). Given any coupling (X0,X1)(X_0,X_1) (for generation, independent noise X0∼N(0,I)X_0\sim\mathcal N(0,I) and data X1X_1), it fits a velocity field v(x,t)v(x,t) to the direction X1−X0X_1-X_0 of the straight line Xt=tX1+(1−t)X0X_t=tX_1+(1-t)X_0. The ODE dZt=v(Zt,t) dtdZ_t=v(Z_t,t)\,dt then transports π0\pi_0 to π1\pi_1 along non-crossing paths. Applying the procedure again to the coupling the ODE produces, reflow, makes the paths straighter, so the ODE can be solved accurately in very few steps.

ODE samplers for Diffusion models follow curved trajectories, so they need dozens to hundreds of network evaluations. One Euler step is exact only along a straight path travelled at constant speed. Liu, Gong and Liu asked how to learn such paths directly. Straight lines between independently drawn noise and data cross each other, and an ODE cannot cross itself, so no flow can follow them exactly. The best ODE in the least-squares sense averages the crossing directions. Its trajectories untangle the lines, and the coupling it induces has lower transport cost than the independent one. Repeat the step, and the lines of the new coupling cross less, so the next flow is straighter. The same objective, with straight paths, appeared concurrently as the OT path of Flow matching (Lipman et al., ICLR 2023). Rectified flow adds the reflow procedure and the straightness analysis. It also covers transfer between two arbitrary distributions, such as cats to wild animals, not only noise to data.

The rectified flow of a coupling

min⁡v∫01E[∥(X1−X0)−v(Xt,t)∥2]dt,Xt=tX1+(1−t)X0,vX(x,t)=E[X1−X0∣Xt=x]\min_v\int_0^1\mathbb E\Big[\big\|(X_1-X_0)-v(X_t,t)\big\|^2\Big]dt,\qquad X_t=tX_1+(1-t)X_0,\qquad v^X(x,t)=\mathbb E\big[X_1-X_0\mid X_t=x\big]
(X0,X1)(X_0,X_1)
any coupling of π0\pi_0 and π1\pi_1; for generation X0∼N(0,I)X_0\sim\mathcal N(0,I) independent of data X1X_1
XtX_t
linear interpolation; X˙t=X1−X0\dot X_t=X_1-X_0 is constant along each line
vXv^X
the minimiser: the average direction of all lines passing through xx at time tt
dZt=vX(Zt,t) dt,  Z0∼π0⟹Law(Zt)=Law(Xt)  ∀t,E c(Z1−Z0)≤E c(X1−X0)  for convex cdZ_t=v^X(Z_t,t)\,dt,\ \ Z_0\sim\pi_0\qquad\Longrightarrow\qquad \mathrm{Law}(Z_t)=\mathrm{Law}(X_t)\ \ \forall t,\qquad \mathbb E\,c(Z_1-Z_0)\le\mathbb E\,c(X_1-X_0)\ \text{ for convex }c
Z=Rectflow(X0,X1)Z=\mathrm{Rectflow}(X_0,X_1)
the rectified flow; (Z0,Z1)(Z_0,Z_1) is a new, deterministic coupling of the same two distributions
Law(Zt)=Law(Xt)\mathrm{Law}(Z_t)=\mathrm{Law}(X_t)
marginal preservation: the flow reproduces every intermediate marginal of the interpolation, so Z1∼π1Z_1\sim\pi_1
cc
any convex cost, e.g. ∥⋅∥2\|\cdot\|^2; rectification never increases it, and it is not tied to one particular cost

Paths of ZZ do not cross: an ODE with a well-behaved field has unique solutions, so two trajectories cannot occupy the same point at the same time. Where lines of XX cross, vXv^X averages their directions, and the trajectories of ZZ rewire around the crossing. That is why the induced coupling is cheaper. Paired points that were far apart are swapped for nearer ones.

Reflow and straightness

Zk+1=Rectflow(Z0k,Z1k),Z0=(X0,X1),S(Z)=∫01E∥(Z1−Z0)−Z˙t∥2dt,min⁡k≤KS(Zk)≤E∥X1−X0∥2KZ^{k+1}=\mathrm{Rectflow}\big(Z^k_0,Z^k_1\big),\quad Z^0=(X_0,X_1),\qquad S(Z)=\int_0^1\mathbb E\big\|(Z_1-Z_0)-\dot Z_t\big\|^2dt,\qquad \min_{k\le K}S(Z^k)\le\frac{\mathbb E\|X_1-X_0\|^2}{K}
ZkZ^k
the kk-rectified flow; Z1Z^1 is trained on the independent coupling, Z2Z^2 on pairs (Z0,Z1)(Z_0,Z_1) obtained by simulating Z1Z^1
S(Z)S(Z)
straightness: zero exactly when every path is a straight line travelled at constant speed
1/K1/K
the paper's bound: straightness decays at rate O(1/K)O(1/K) in the number of reflows

If S(Z)=0S(Z)=0, then Zt=tZ1+(1−t)Z0Z_t=tZ_1+(1-t)Z_0, and a single Euler step Z1=Z0+v(Z0,0)Z_1=Z_0+v(Z_0,0) is exact. In practice one reflow already gets close. The remaining gap is closed by distillation: fit a one-step model to the deterministic pairs with E∥(Z1k−Z0k)−v(Z0k,0)∥2\mathbb E\|(Z^k_1-Z^k_0)-v(Z^k_0,0)\|^2. Distilling directly from the 1-rectified flow is much harder, because its pairs come from curved paths. Each reflow costs a full round of simulation, Z1=ODE(Z0)Z_1=\mathrm{ODE}(Z_0) on many noise draws, plus retraining. Errors also accumulate: every round fits an imperfect vv, so the samples drift slightly from π1\pi_1.

Xt=αtX1+βtX0,target α˙tX1+β˙tX0πln(t;m,s)=1s2π 1t(1−t)exp⁡ ⁣(−(logit t−m)22s2)X_t=\alpha_tX_1+\beta_tX_0,\quad \text{target }\dot\alpha_tX_1+\dot\beta_tX_0\qquad\qquad \pi_{\rm ln}(t;m,s)=\frac{1}{s\sqrt{2\pi}}\,\frac{1}{t(1-t)}\exp\!\Big(-\frac{(\mathrm{logit}\,t-m)^2}{2s^2}\Big)
αt,βt\alpha_t,\beta_t
the paper's nonlinear extension; the VP probability-flow ODE of diffusion is one such choice, and straight lines (αt=t, βt=1−t\alpha_t=t,\ \beta_t=1-t) are the canonical one
πln\pi_{\rm ln}
logit-normal density over tt used by Stable Diffusion 3 to sample training times; it puts more weight on intermediate tt, where the velocity target is hardest
m,sm,s
location and scale; m=0, s=1m=0,\ s=1 was the best setting overall in Esser et al. (2024)

Setup and follow-ups

Images
CIFAR-10 with the DDPM++ U-Net of Song et al. (2021); also LSUN, CelebA-HQ and AFHQ at 2562256^2, and image-to-image transfer
Solvers
RK45 adaptive (for the full-quality numbers) or fixed-step Euler with NN steps
Reflow data
pairs (Z0,Z1)(Z_0,Z_1) generated by simulating the previous flow; distillation regresses Z1−Z0Z_1-Z_0 at t=0t=0
InstaFlow
Liu, Zhang, Ma, Peng and Liu (ICLR 2024): text-conditioned reflow of Stable Diffusion 1.5 and distillation to one step; FID 23.3 on MS COCO 2017-5k (previous best one-step 37.2) and 13.1 on MS COCO 2014-30k at 0.09 s per image, for 199 A100 GPU days of training
Stable Diffusion 3
Esser et al., "Scaling Rectified Flow Transformers for High-Resolution Image Synthesis" (ICML 2024): rectified-flow objective with logit-normal tt sampling, an MM-DiT backbone, and data at t=0t=0, noise at t=1t=1 (zt=(1−t)x0+tϵz_t=(1-t)x_0+t\epsilon)
modelNFEIS ↑FID ↓recall ↑
1-rectified flow (+distill)11.13 (9.08)378 (6.18)0.0 (0.45)
2-rectified flow (+distill)18.08 (9.01)12.21 (4.85)0.34 (0.50)
3-rectified flow (+distill)18.47 (8.79)8.15 (5.21)0.41 (0.51)
VP ODE (+distill)11.20 (8.73)451 (16.23)0.0 (0.29)
1-rectified flow, RK451279.602.580.57
2-rectified flow, RK451109.243.360.54
3-rectified flow, RK451049.013.960.53
VP SDE (2000 steps)20009.582.550.58
CIFAR-10, DDPM++ architecture (Liu et al. 2023, Table 1a); one-step numbers without distillation, distilled in parentheses

The table shows both effects. One Euler step of the 1-rectified flow is useless (FID 378): at t=0t=0 its field points every sample at the data mean. After one reflow the same single step gives 12.21, and after distillation 4.85. Reflow costs some quality at full NFE (2.58 → 3.36 → 3.96 with RK45), because each round refits an approximate field. Two or three rectifications are the useful range. The adaptive solver also needs fewer steps on the straighter flows (127 → 110 → 104 NFE).

N(0, I) → 4 Gaussians · X_t = t X₁ + (1 − t) X₀ · v ≈ E[X₁ − X₀ | X_t]160 samples, 1 Euler step; lines = first 50 paths○ data modes (radius 3σ) · t = 1.00coupling (Z₀, Z₁)cost E‖Z₁−Z₀‖² · straightness S independent 10.230 · —▸ 1-rectified 3.158 · 2.628 2-rectified 3.217 · 0.012crossing chords (of 1225) 5sampling with 1 stepNFE 1on a mode at t = 1 0.0%mean |x₁ − x₁(16 steps)| 2.7801-step on a mode 0.0%S = ∫ E‖(Z₁ − Z₀) − v(Z_t, t)‖² dtis 0 only for straight paths, whereone Euler step is exact.1-rectified at t = 0: v = E[X₁] − x,so one step sends every x₀ to thedata mean. Reflow regresses on theflow's own (Z₀, Z₁) pairs instead.
coupling
Euler steps
fig.N(0, I) to four Gaussians at (±2, ±2) with std 0.2. The 1-rectified field E[X₁ − X₀ | Xₜ = x] is exact in closed form. The 2-rectified field is a kernel regression (width 0.12) on 800 reflow pairs (Z₀, Z₁), with Z₁ obtained by solving the 1-flow with RK4. It stands in for a second trained network. For 160 test points, the independent pairs have transport cost E‖X₁ − X₀‖² = 10.23, and 251 of the 1225 pairs of drawn chords cross. The 1-rectified coupling costs 3.158, only 5 chord pairs cross, but its paths are curved (straightness S = 2.628). One Euler step sends every sample to the data mean (0% on a mode, mean error 2.780 against the 16-step solution). Two steps give 37.5%, four 96.3%. After one reflow S drops to 0.012, and a single Euler step already puts 72.5% of samples on a mode, with mean error 0.208. The 2-rectified flow ends slightly worse at 16 steps (95.0% against 98.8%) and its cost is 3.217, not lower. That is the fitted field's error, the same effect as the paper's FID rising from 2.58 to 3.36 after reflow. The theorem guarantees that cost does not increase only for an exact fit.

Worked example: two points on a line

Let X0∼N(0,1)X_0\sim\mathcal N(0,1) and X1=±2X_1=\pm2 with probability 12\tfrac12 each, independent. In one dimension any ODE flow preserves order, so the 1-rectified coupling is already the monotone (optimal) one, Z1=2 sign(Z0)Z_1=2\,\mathrm{sign}(Z_0). Its paths are still curved. Reflow on that coupling gives straight lines that never cross.

couplingE(X₁ − X₀)²velocity at t = 0one Euler step from x₀ = 0.8
independent (lines only)5——
1-rectified1.8085E[X₁] − x = −0.80 (the mean)
2-rectified1.80852 sign(x) − x = 1.22 (exact)
Couplings of N(0, 1) and ±2 (computed in Node)

Computing the fields

  1. 1Independent cost: E(X1−X0)2=EX12+EX02=4+1=5\mathbb E(X_1-X_0)^2=\mathbb EX_1^2+\mathbb EX_0^2=4+1=5.
  2. 21-rectified field at t=0.5, x=0.5t=0.5,\ x=0.5: given X1=±2X_1=\pm2, Xt∼N(±1, 0.25)X_t\sim\mathcal N(\pm1,\,0.25), so the posterior weights are ∝e−0.5\propto e^{-0.5} and e−4.5e^{-4.5}, i.e. 0.982010.98201 and 0.017990.01799.
  3. 3Along the line through xx with endpoint X1X_1, the direction is X1−X0=(X1−x)/(1−t)X_1-X_0=(X_1-x)/(1-t): 33 for +2+2 and −5-5 for −2-2. So v(0.5,0.5)=0.98201⋅3+0.01799⋅(−5)=2.8561v(0.5,0.5)=0.98201\cdot3+0.01799\cdot(-5)=2.8561.
  4. 4At t=0t=0, Xt=X0X_t=X_0 carries no information about X1X_1: v(x,0)=E[X1]−x=−xv(x,0)=\mathbb E[X_1]-x=-x. One Euler step maps every x0x_0 to 00, the mean of π1\pi_1, which is not in its support.
  5. 5Monotone coupling cost: E(2 signZ0−Z0)2=4−4 E∣Z0∣+1=5−42/π=1.8085\mathbb E(2\,\mathrm{sign}Z_0-Z_0)^2=4-4\,\mathbb E|Z_0|+1=5-4\sqrt{2/\pi}=1.8085. Rectification lowered the cost from 5 to 1.8085.
  6. 6Reflow: the lines from z0z_0 to 2 sign(z0)2\,\mathrm{sign}(z_0) keep their order for t<1t<1, so they never cross and v2(x,t)v^{2}(x,t) equals the line's slope. S=0S=0, and one step 0.8+1.2=20.8+1.2=2 is exact.
python
1import torch, torch.nn as nn
2
3class Velocity(nn.Module):
4 def __init__(self, d=2, h=256):
5 super().__init__()
6 self.net = nn.Sequential(nn.Linear(d + 1, h), nn.SiLU(), nn.Linear(h, h), nn.SiLU(),
7 nn.Linear(h, h), nn.SiLU(), nn.Linear(h, d))
8
9 def forward(self, x, t): # x (B, d), t (B,)
10 return self.net(torch.cat([x, t[:, None]], -1))
11
12def rf_loss(v, x0, x1):
13 t = torch.rand(x0.size(0))
14 xt = t[:, None] * x1 + (1 - t[:, None]) * x0 # X_t = t X1 + (1 - t) X0
15 return ((v(xt, t) - (x1 - x0)) ** 2).sum(-1).mean()
16
17@torch.no_grad()
18def euler(v, x, steps): # dZ_t = v(Z_t, t) dt from t = 0 to 1
19 for i in range(steps):
20 x = x + v(x, torch.full((x.size(0),), i / steps)) / steps
21 return x
22
23def train(v, pair_fn, iters=2000, bs=512):
24 opt = torch.optim.Adam(v.parameters(), lr=1e-3)
25 for _ in range(iters):
26 x0, x1 = pair_fn(bs)
27 loss = rf_loss(v, x0, x1)
28 opt.zero_grad(); loss.backward(); opt.step()
29 return v
30
31modes = torch.tensor([[2., 2.], [-2., 2.], [-2., -2.], [2., -2.]])
32data = lambda n: modes[torch.randint(0, 4, (n,))] + 0.2 * torch.randn(n, 2)
33
34torch.manual_seed(0)
35# 1-rectified flow: independent coupling (X0, X1) ~ pi_0 x pi_1
36v1 = train(Velocity(), lambda n: (torch.randn(n, 2), data(n)))
37
38# Reflow: simulate the 1-flow once to get a deterministic coupling (Z0, Z1), then refit
39z0 = torch.randn(20000, 2)
40z1 = euler(v1, z0, 100)
41def reflow_pairs(n):
42 i = torch.randint(0, z0.size(0), (n,))
43 return z0[i], z1[i]
44v2 = train(Velocity(), reflow_pairs)
45
46x0 = torch.randn(2000, 2)
47for name, v in [("1-rectified", v1), ("2-rectified", v2)]:
48 one, ref = euler(v, x0, 1), euler(v, x0, 100)
49 print(name, one.shape, round((one - ref).norm(dim=-1).mean().item(), 2))
50# 1-rectified torch.Size([2000, 2]) 2.81 one Euler step collapses toward the data mean
51# 2-rectified torch.Size([2000, 2]) 0.17 near-straight paths: one step is close to 100 steps
Rectified flow, one reflow, and the one-step error on a 2-D toy

Use when

  • +A simple least-squares objective; any pair of distributions (noise→data or domain→domain)
  • +Marginals are preserved and every convex transport cost decreases under rectification
  • +Reflow straightens paths, so 1–2 Euler steps work, and distillation to one step becomes easy
  • +Scales: InstaFlow (one-step text-to-image) and Stable Diffusion 3 use it

Avoid when

  • −Each reflow needs a large simulated dataset and a full retraining
  • −Approximation error accumulates over rounds; full-NFE quality drops slightly (FID 2.58 → 3.36 → 3.96)
  • −The 1-rectified flow itself is curved; one-step sampling without reflow fails (FID 378)
  • −Straightness is achieved for the fitted coupling, not for the optimal-transport coupling of a given cost

Check yourself

A 1-rectified flow from N(0, I) to data is sampled with one Euler step. What does it produce?

Check yourself

What does reflow change?

Remember

  • ✓Rectified flow: min⁡v∫01E∥(X1−X0)−v(Xt,t)∥2dt\min_v\int_0^1\mathbb E\|(X_1-X_0)-v(X_t,t)\|^2dt with Xt=tX1+(1−t)X0X_t=tX_1+(1-t)X_0; sample with dZt=v(Zt,t)dtdZ_t=v(Z_t,t)dt from Z0∼N(0,I)Z_0\sim\mathcal N(0,I).
  • ✓Reflow: retrain on simulated pairs (Z0,Z1)(Z_0,Z_1); paths become straighter (min⁡k≤KS(Zk)≤E∥X1−X0∥2/K\min_{k\le K}S(Z^k)\le\mathbb E\|X_1-X_0\|^2/K), so one Euler step or a distilled one-step model works.
  • ✓Rectification preserves all marginals and does not increase any convex transport cost; its trajectories never cross.
  • ✓CIFAR-10 (Table 1): one step FID 378 for 1-RF, 12.21 after one reflow, 4.85 after reflow plus distillation; 2.58 with RK45 at 127 NFE.
  • ✓Stable Diffusion 3 trains a rectified-flow transformer with logit-normal timestep sampling; InstaFlow reflows and distils Stable Diffusion into one step.
← back to Deep Learning roadmap