Definition
Rectified flow is a method for learning a transport map between two distributions and from samples, introduced by Xingchao Liu, Chengyue Gong and Qiang Liu, "Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow" (ICLR 2023). Given any coupling (for generation, independent noise and data ), it fits a velocity field to the direction of the straight line . The ODE then transports to along non-crossing paths. Applying the procedure again to the coupling the ODE produces, reflow, makes the paths straighter, so the ODE can be solved accurately in very few steps.
ODE samplers for Diffusion models follow curved trajectories, so they need dozens to hundreds of network evaluations. One Euler step is exact only along a straight path travelled at constant speed. Liu, Gong and Liu asked how to learn such paths directly. Straight lines between independently drawn noise and data cross each other, and an ODE cannot cross itself, so no flow can follow them exactly. The best ODE in the least-squares sense averages the crossing directions. Its trajectories untangle the lines, and the coupling it induces has lower transport cost than the independent one. Repeat the step, and the lines of the new coupling cross less, so the next flow is straighter. The same objective, with straight paths, appeared concurrently as the OT path of Flow matching (Lipman et al., ICLR 2023). Rectified flow adds the reflow procedure and the straightness analysis. It also covers transfer between two arbitrary distributions, such as cats to wild animals, not only noise to data.
The rectified flow of a coupling
- any coupling of and ; for generation independent of data
- linear interpolation; is constant along each line
- the minimiser: the average direction of all lines passing through at time
- the rectified flow; is a new, deterministic coupling of the same two distributions
- marginal preservation: the flow reproduces every intermediate marginal of the interpolation, so
- any convex cost, e.g. ; rectification never increases it, and it is not tied to one particular cost
Paths of do not cross: an ODE with a well-behaved field has unique solutions, so two trajectories cannot occupy the same point at the same time. Where lines of cross, averages their directions, and the trajectories of rewire around the crossing. That is why the induced coupling is cheaper. Paired points that were far apart are swapped for nearer ones.
Reflow and straightness
- the -rectified flow; is trained on the independent coupling, on pairs obtained by simulating
- straightness: zero exactly when every path is a straight line travelled at constant speed
- the paper's bound: straightness decays at rate in the number of reflows
If , then , and a single Euler step is exact. In practice one reflow already gets close. The remaining gap is closed by distillation: fit a one-step model to the deterministic pairs with . Distilling directly from the 1-rectified flow is much harder, because its pairs come from curved paths. Each reflow costs a full round of simulation, on many noise draws, plus retraining. Errors also accumulate: every round fits an imperfect , so the samples drift slightly from .
- the paper's nonlinear extension; the VP probability-flow ODE of diffusion is one such choice, and straight lines () are the canonical one
- logit-normal density over used by Stable Diffusion 3 to sample training times; it puts more weight on intermediate , where the velocity target is hardest
- location and scale; was the best setting overall in Esser et al. (2024)
Setup and follow-ups
- Images
- CIFAR-10 with the DDPM++ U-Net of Song et al. (2021); also LSUN, CelebA-HQ and AFHQ at , and image-to-image transfer
- Solvers
- RK45 adaptive (for the full-quality numbers) or fixed-step Euler with steps
- Reflow data
- pairs generated by simulating the previous flow; distillation regresses at
- InstaFlow
- Liu, Zhang, Ma, Peng and Liu (ICLR 2024): text-conditioned reflow of Stable Diffusion 1.5 and distillation to one step; FID 23.3 on MS COCO 2017-5k (previous best one-step 37.2) and 13.1 on MS COCO 2014-30k at 0.09 s per image, for 199 A100 GPU days of training
- Stable Diffusion 3
- Esser et al., "Scaling Rectified Flow Transformers for High-Resolution Image Synthesis" (ICML 2024): rectified-flow objective with logit-normal sampling, an MM-DiT backbone, and data at , noise at ()
| model | NFE | IS ↑ | FID ↓ | recall ↑ |
|---|---|---|---|---|
| 1-rectified flow (+distill) | 1 | 1.13 (9.08) | 378 (6.18) | 0.0 (0.45) |
| 2-rectified flow (+distill) | 1 | 8.08 (9.01) | 12.21 (4.85) | 0.34 (0.50) |
| 3-rectified flow (+distill) | 1 | 8.47 (8.79) | 8.15 (5.21) | 0.41 (0.51) |
| VP ODE (+distill) | 1 | 1.20 (8.73) | 451 (16.23) | 0.0 (0.29) |
| 1-rectified flow, RK45 | 127 | 9.60 | 2.58 | 0.57 |
| 2-rectified flow, RK45 | 110 | 9.24 | 3.36 | 0.54 |
| 3-rectified flow, RK45 | 104 | 9.01 | 3.96 | 0.53 |
| VP SDE (2000 steps) | 2000 | 9.58 | 2.55 | 0.58 |
The table shows both effects. One Euler step of the 1-rectified flow is useless (FID 378): at its field points every sample at the data mean. After one reflow the same single step gives 12.21, and after distillation 4.85. Reflow costs some quality at full NFE (2.58 → 3.36 → 3.96 with RK45), because each round refits an approximate field. Two or three rectifications are the useful range. The adaptive solver also needs fewer steps on the straighter flows (127 → 110 → 104 NFE).
Worked example: two points on a line
Let and with probability each, independent. In one dimension any ODE flow preserves order, so the 1-rectified coupling is already the monotone (optimal) one, . Its paths are still curved. Reflow on that coupling gives straight lines that never cross.
| coupling | E(X₁ − X₀)² | velocity at t = 0 | one Euler step from x₀ = 0.8 |
|---|---|---|---|
| independent (lines only) | 5 | — | — |
| 1-rectified | 1.8085 | E[X₁] − x = −0.8 | 0 (the mean) |
| 2-rectified | 1.8085 | 2 sign(x) − x = 1.2 | 2 (exact) |
Computing the fields
- 1Independent cost: .
- 21-rectified field at : given , , so the posterior weights are and , i.e. and .
- 3Along the line through with endpoint , the direction is : for and for . So .
- 4At , carries no information about : . One Euler step maps every to , the mean of , which is not in its support.
- 5Monotone coupling cost: . Rectification lowered the cost from 5 to 1.8085.
- 6Reflow: the lines from to keep their order for , so they never cross and equals the line's slope. , and one step is exact.
1import torch, torch.nn as nn23class Velocity(nn.Module):4 def __init__(self, d=2, h=256):5 super().__init__()6 self.net = nn.Sequential(nn.Linear(d + 1, h), nn.SiLU(), nn.Linear(h, h), nn.SiLU(),7 nn.Linear(h, h), nn.SiLU(), nn.Linear(h, d))89 def forward(self, x, t): # x (B, d), t (B,)10 return self.net(torch.cat([x, t[:, None]], -1))1112def rf_loss(v, x0, x1):13 t = torch.rand(x0.size(0))14 xt = t[:, None] * x1 + (1 - t[:, None]) * x0 # X_t = t X1 + (1 - t) X015 return ((v(xt, t) - (x1 - x0)) ** 2).sum(-1).mean()1617@torch.no_grad()18def euler(v, x, steps): # dZ_t = v(Z_t, t) dt from t = 0 to 119 for i in range(steps):20 x = x + v(x, torch.full((x.size(0),), i / steps)) / steps21 return x2223def train(v, pair_fn, iters=2000, bs=512):24 opt = torch.optim.Adam(v.parameters(), lr=1e-3)25 for _ in range(iters):26 x0, x1 = pair_fn(bs)27 loss = rf_loss(v, x0, x1)28 opt.zero_grad(); loss.backward(); opt.step()29 return v3031modes = torch.tensor([[2., 2.], [-2., 2.], [-2., -2.], [2., -2.]])32data = lambda n: modes[torch.randint(0, 4, (n,))] + 0.2 * torch.randn(n, 2)3334torch.manual_seed(0)35# 1-rectified flow: independent coupling (X0, X1) ~ pi_0 x pi_136v1 = train(Velocity(), lambda n: (torch.randn(n, 2), data(n)))3738# Reflow: simulate the 1-flow once to get a deterministic coupling (Z0, Z1), then refit39z0 = torch.randn(20000, 2)40z1 = euler(v1, z0, 100)41def reflow_pairs(n):42 i = torch.randint(0, z0.size(0), (n,))43 return z0[i], z1[i]44v2 = train(Velocity(), reflow_pairs)4546x0 = torch.randn(2000, 2)47for name, v in [("1-rectified", v1), ("2-rectified", v2)]:48 one, ref = euler(v, x0, 1), euler(v, x0, 100)49 print(name, one.shape, round((one - ref).norm(dim=-1).mean().item(), 2))50# 1-rectified torch.Size([2000, 2]) 2.81 one Euler step collapses toward the data mean51# 2-rectified torch.Size([2000, 2]) 0.17 near-straight paths: one step is close to 100 stepsUse when
- +A simple least-squares objective; any pair of distributions (noise→data or domain→domain)
- +Marginals are preserved and every convex transport cost decreases under rectification
- +Reflow straightens paths, so 1–2 Euler steps work, and distillation to one step becomes easy
- +Scales: InstaFlow (one-step text-to-image) and Stable Diffusion 3 use it
Avoid when
- −Each reflow needs a large simulated dataset and a full retraining
- −Approximation error accumulates over rounds; full-NFE quality drops slightly (FID 2.58 → 3.36 → 3.96)
- −The 1-rectified flow itself is curved; one-step sampling without reflow fails (FID 378)
- −Straightness is achieved for the fitted coupling, not for the optimal-transport coupling of a given cost
Check yourself
A 1-rectified flow from N(0, I) to data is sampled with one Euler step. What does it produce?
Check yourself
What does reflow change?
Remember
- ✓Rectified flow: with ; sample with from .
- ✓Reflow: retrain on simulated pairs ; paths become straighter (), so one Euler step or a distilled one-step model works.
- ✓Rectification preserves all marginals and does not increase any convex transport cost; its trajectories never cross.
- ✓CIFAR-10 (Table 1): one step FID 378 for 1-RF, 12.21 after one reflow, 4.85 after reflow plus distillation; 2.58 with RK45 at 127 NFE.
- ✓Stable Diffusion 3 trains a rectified-flow transformer with logit-normal timestep sampling; InstaFlow reflows and distils Stable Diffusion into one step.