Normalizing flows

A normalizing flow writes data as an invertible, differentiable transform x = g(z) of simple noise z, so the exact log-likelihood is log p(z) plus the log-determinant of the Jacobian of g⁻¹; the whole design problem is building bijections whose determinants are cheap.

15 min read1420 wordscore0 viewsupdated 2026-10-12

Definition

Normalizing flow A generative model that maps a sample zz from a simple base density pZp_Z (usually N(0,I)\mathcal N(0,I)) through an invertible, differentiable function x=g(z)x=g(z) with inverse f=g−1f=g^{-1}. By the change-of-variables formula the model density is exact: log⁡pX(x)=log⁡pZ(f(x))+log⁡∣det⁡∂f(x)/∂x⊤∣\log p_X(x)=\log p_Z(f(x))+\log\big|\det \partial f(x)/\partial x^\top\big|. Sampling is z∼pZz\sim p_Z followed by one pass of gg. A flow is built by composing many simple bijections, and each layer is designed so that its Jacobian determinant costs O(D)O(D) rather than O(D3)O(D^3).

A Variational autoencoder (VAE) only bounds the likelihood, and a Generative adversarial network (GAN) has none at all. Autoregressive models give exact likelihoods but sample one dimension at a time (see Autoregressive generative models). Flows give an exact likelihood, a latent code for every data point, and, for coupling-based designs, one-pass sampling. The idea of transforming a density by a chain of simple maps goes back to Tabak and Vanden-Eijnden (2010) and Tabak and Turner (2013). The name and its use in deep learning come from Danilo Rezende and Shakir Mohamed, "Variational Inference with Normalizing Flows" (ICML 2015). They used planar and radial flows to make VAE posteriors more flexible. Laurent Dinh, David Krueger and Yoshua Bengio's NICE (arXiv 2014, ICLR 2015 workshop) introduced coupling layers for density estimation of data. That line led to RealNVP and Glow.

Change of variables

pX(x)=pZ(f(x)) ∣det⁡∂f(x)∂x⊤∣⟺log⁡pX(x)=log⁡pZ(z)−log⁡∣det⁡∂g(z)∂z⊤∣,  z=f(x)p_X(x)=p_Z\big(f(x)\big)\,\Big|\det\frac{\partial f(x)}{\partial x^\top}\Big|\qquad\Longleftrightarrow\qquad \log p_X(x)=\log p_Z(z)-\log\Big|\det\frac{\partial g(z)}{\partial z^\top}\Big|,\ \ z=f(x)
f=g−1f=g^{-1}
the normalizing direction, data →\to noise; gg is the generative direction
∂f/∂x⊤\partial f/\partial x^\top
the D×DD\times D Jacobian; ∣det⁡∣|\det| is the local factor by which ff scales volume
pZp_Z
base density, usually N(0,I)\mathcal N(0,I): log⁡pZ(z)=−12∥z∥2−D2log⁡2π\log p_Z(z)=-\tfrac12\|z\|^2-\tfrac D2\log 2\pi

The determinant accounts for volume. Probability mass in a small region is preserved by the map, so if gg stretches a neighbourhood of zz by a factor of 3 in area, the density at x=g(z)x=g(z) is 3 times lower. In one dimension this is pX(x)=pZ(z) ∣dz/dx∣p_X(x)=p_Z(z)\,|dz/dx|. Both ff and gg must be bijections between RD\mathbb R^D and RD\mathbb R^D, so xx and zz have the same dimension. A flow never compresses.

Composing bijections

f=fK∘⋯∘f1,h0=x,  hk=fk(hk−1),log⁡pX(x)=log⁡pZ(hK)+∑k=1Klog⁡∣det⁡∂fk∂hk−1⊤∣f=f_K\circ\cdots\circ f_1,\qquad h_0=x,\ \ h_k=f_k(h_{k-1}),\qquad \log p_X(x)=\log p_Z(h_K)+\sum_{k=1}^{K}\log\Big|\det\frac{\partial f_k}{\partial h_{k-1}^\top}\Big|
fkf_k
one layer; each must be invertible with a tractable determinant
∑k\sum_k
Jacobians of a composition multiply (chain rule), so their log-determinants add

Training maximises 1N∑nlog⁡pX(xn)\frac1N\sum_n\log p_X(x_n) directly, which is the same as minimising the forward KL divergence from the data distribution to the model. One forward pass x→zx\to z produces zz and the running sum of log-determinants. Sampling runs the layers backwards, z→xz\to x. Each layer can be weak, because depth provides the expressivity. What any layer must provide is (1) invertibility, (2) a cheap log⁡∣det⁡∣\log|\det|, and (3) a cheap inverse if you want to sample. A generic dense layer WxWx with a nonlinearity fails (2): det⁡\det of a D×DD\times D matrix costs O(D3)O(D^3), about 2⋅10102\cdot10^{10} flops for one 32×32×332\times32\times3 image, at every layer and every step.

Planar flows

g(z)=z+u tanh⁡(w⊤z+b),det⁡∂g∂z⊤=1+u⊤ψ(z),ψ(z)=(1−tanh⁡2(w⊤z+b)) wg(z)=z+u\,\tanh(w^\top z+b),\qquad \det\frac{\partial g}{\partial z^\top}=1+u^\top\psi(z),\quad \psi(z)=\big(1-\tanh^2(w^\top z+b)\big)\,w
u,w∈RD, b∈Ru,w\in\mathbb R^D,\ b\in\mathbb R
parameters; the layer pushes zz along uu by an amount that depends on its side of the hyperplane w⊤z+b=0w^\top z+b=0
1+u⊤ψ1+u^\top\psi
matrix determinant lemma: det⁡(I+uψ⊤)=1+ψ⊤u\det(I+u\psi^\top)=1+\psi^\top u, an O(D)O(D) dot product

Rezende and Mohamed's planar layer is invertible when w⊤u≥−1w^\top u\ge-1. They enforce this by replacing uu with u^=u+[m(w⊤u)−w⊤u]w∥w∥2\hat u=u+\big[m(w^\top u)-w^\top u\big]\frac{w}{\|w\|^2}, where m(a)=−1+log⁡(1+ea)m(a)=-1+\log(1+e^a). Its inverse has no closed form: you solve a scalar equation for w⊤zw^\top z. That is acceptable in their setting, which is variational inference: samples go from zz to xx, and the density of each sample is needed only at the point that was just generated. The bottleneck is capacity. Each layer modifies the density along one direction only, so planar flows need many layers in high dimensions. The paper also has radial flows, g(z)=z+βh(α,r)(z−z0)g(z)=z+\beta h(\alpha,r)(z-z_0), which contract or expand around a point.

x = g_K ∘ … ∘ g_1(z), z ~ N(0, I), g(z) = z + u·tanh(wᵀz + b) · K = 2q(x) = N(g⁻¹(x)) · Π |det ∂g/∂z|⁻¹ (exact)histogram of 40,000 pushed samples−55−55=wᵀu per layer3.00, 2.40min 1 + uᵀψ over samples1.000E[Σ log|det ∂g/∂z|]0.952H(q) = H(N) + E[Σ log|det|]3.790 nats∫ q(x) dx in box (formula)1.000samples inside box1.000TV(formula, histogram)0.071statusbijective: formula exactdet ∂g/∂z = 1 + uᵀψ, ψ = (1 − tanh²(wᵀz + b))·w: one dot product, O(D), not O(D³)raw u: if wᵀu < −1 the layer folds, det changes sign, the formula breaks
invertibility
fig.Up to four planar layers g(z) = z + u·tanh(wᵀz + b) push a 2-D standard Gaussian. The weights are fixed (not trained): |w| = 2.5 with preset directions, and λ sets wᵀu = λ, 0.8λ, 0.6λ, 0.5λ for layers 1–4. The left panel is the density given by the change-of-variables formula: invert the stack (bisection on wᵀz), then subtract Σ log|1 + uᵀψ|. The right panel is a histogram of 40,000 pushed samples. With K = 0 their total-variation distance is 0.049, which is pure Monte-Carlo noise. With K = 2 and λ = 3 (wᵀu = 3.0, 2.4) the Gaussian splits into four lobes. The panels still agree (TV 0.071, ∫q = 1.000), E[Σ log|det|] = 0.952, and the entropy rises from H(N) = 2.838 to 3.790 nats. With K = 4 the lobes merge into a ring: TV 0.082, and the formula's mass inside the box, 0.987, equals the fraction of samples inside it. Back at K = 2, press fold (λ = −1.2, raw u): layer 1 has wᵀu = −1.2 < −1, the Jacobian factor reaches −0.200, and the map is no longer one-to-one. The formula's mass drops to 0.662 and TV jumps to 0.194, while every sample is still inside the box. Switch to constrained û and the same λ gives wᵀû = −0.737 and −0.676. The minimum factor is then 0.263 > 0, and the panels agree again (TV 0.044).

Coupling layers: NICE

y1:d=x1:d,yd+1:D=xd+1:D+m(x1:d),∂y∂x⊤=(I0∂m/∂x1:d⊤I)y_{1:d}=x_{1:d},\qquad y_{d+1:D}=x_{d+1:D}+m(x_{1:d}),\qquad \frac{\partial y}{\partial x^\top}=\begin{pmatrix}I&0\\ \partial m/\partial x_{1:d}^\top&I\end{pmatrix}
mm
any neural network; never inverted and never differentiated for the determinant
det⁡=1\det=1
the Jacobian is unit lower triangular: additive coupling preserves volume

NICE (non-linear independent components estimation) splits the input, copies one half, and shifts the other half by a function of the copy. The inverse is xd+1:D=yd+1:D−m(y1:d)x_{d+1:D}=y_{d+1:D}-m(y_{1:d}), which is exact and as cheap as the forward pass. Layers alternate which half is copied. Because every coupling has determinant 1, NICE ends with a learned diagonal scaling layer so that the model can change volume at all. RealNVP made the coupling affine, xu⊙es(xc)+t(xc)x_u\odot e^{s(x_c)}+t(x_c), with log⁡∣det⁡∣=∑s\log|\det|=\sum s, and added a multi-scale image architecture. Glow added actnorm and a learned invertible 1×11\times1 convolution in place of fixed permutations.

Autoregressive flows: MAF and IAF

MAF:  zi=(xi−μi(x<i)) e−αi(x<i)IAF:  xi=zi eσi(z<i)+μi(z<i)\text{MAF:}\ \ z_i=\big(x_i-\mu_i(x_{<i})\big)\,e^{-\alpha_i(x_{<i})}\qquad\qquad \text{IAF:}\ \ x_i=z_i\,e^{\sigma_i(z_{<i})}+\mu_i(z_{<i})
μi,αi\mu_i,\alpha_i
outputs of a masked autoregressive network (MADE): output ii depends only on inputs <i<i
log⁡∣det⁡∣\log|\det|
triangular Jacobian: −∑iαi-\sum_i\alpha_i for MAF (x→zx\to z), ∑iσi\sum_i\sigma_i for IAF (z→xz\to x)

An autoregressive flow transforms each dimension conditioned on all previous ones. It is strictly more general than a coupling layer, which is the special case where the first dd dimensions condition everything else in one block. The cost is asymmetric. In MAF (masked autoregressive flow; Papamakarios, Pavlakou and Murray, NeurIPS 2017) the conditioners read xx, so the density of a data point takes one parallel network pass, but sampling needs DD sequential passes, since xix_i requires x<ix_{<i}. IAF (inverse autoregressive flow; Kingma, Salimans, Jozefowicz, Chen, Sutskever and Welling, NeurIPS 2016) is the same map run in the other direction. Its conditioners read zz, so sampling is one pass and the density of the model's own samples is free, but the density of an external xx needs DD sequential steps. That made IAF a natural fit for VAE posteriors and for fast synthesis, for example Parallel WaveNet. Coupling layers sit in between: both directions take one pass, and each layer is less expressive.

Continuous-time flows

dz(t)dt=vθ(z(t),t),dlog⁡p(z(t))dt=−tr⁡(∂vθ∂z⊤)\frac{dz(t)}{dt}=v_\theta\big(z(t),t\big),\qquad \frac{d\log p\big(z(t)\big)}{dt}=-\operatorname{tr}\Big(\frac{\partial v_\theta}{\partial z^\top}\Big)
vθv_\theta
any network; the ODE solution map is a bijection for well-behaved vθv_\theta
tr⁡\operatorname{tr}
instantaneous change of variables: a trace instead of a determinant

Neural ODEs (Chen, Rubanova, Bettencourt and Duvenaud, NeurIPS 2018) take the limit of infinitely many infinitesimal layers. The log-density then evolves by the trace of the Jacobian. FFJORD (Grathwohl et al., ICLR 2019) estimates that trace without bias using Hutchinson's estimator Eϵ[ϵ⊤(∂v/∂z)ϵ]\mathbb E_\epsilon[\epsilon^\top(\partial v/\partial z)\epsilon], one vector-Jacobian product per sample, so vθv_\theta needs no structure at all. The price is an ODE solve, with many function evaluations, for every likelihood and every sample, and simulation during training. Flow matching and Rectified flow keep the continuous flow and train vθv_\theta by regression, with no simulation and no likelihood during training.

Dequantisation and bits per dimension

Images are discrete: each subpixel takes one of 256 values. A continuous density can put arbitrarily high density on those points and drive the likelihood to infinity. The standard fix is uniform dequantisation: train on y=(x+u)/256y=(x+u)/256 with u∼U[0,1)Du\sim\mathcal U[0,1)^D. Theis, van den Oord and Bethge (ICLR 2016) showed that the continuous log-likelihood of dequantised data is a lower bound on the discrete log-likelihood of xx. Flow++ (Ho et al., ICML 2019) learned the noise distribution instead (variational dequantisation), which tightens the bound. Results are reported in bits per dimension:

bpd=−log⁡pX(x)Dln⁡2=−log⁡pY(y)+Dln⁡256Dln⁡2\text{bpd}=\frac{-\log p_X(x)}{D\ln 2}=\frac{-\log p_Y(y)+D\ln 256}{D\ln 2}
pYp_Y
density of the model on data scaled to [0,1)D[0,1)^D; rescaling by 256 adds Dln⁡256D\ln256 nats
DD
number of subpixels, e.g. 3⋅32⋅32=30723\cdot32\cdot32=3072 for CIFAR-10
88
bits/dim of a model that puts uniform mass on all 256D256^D images; lower is better

Worked example: one planar layer

Take D=2D=2, z=(0.5,−0.3)z=(0.5,-0.3), w=(1,0.5)w=(1,0.5), u=(0.8,−0.4)u=(0.8,-0.4), b=0.1b=0.1. Here w⊤u=0.6>−1w^\top u=0.6>-1, so the layer is invertible.

quantityvalue
a=w⊤z+ba=w^\top z+b0.5 − 0.15 + 0.1 = 0.45
tanh⁡a\tanh a0.4219
x=z+utanh⁡ax=z+u\tanh a(0.8375, −0.4688)
1−tanh⁡2a1-\tanh^2 a0.8220
u⊤ψ=(1−tanh⁡2a) w⊤uu^\top\psi=(1-\tanh^2a)\,w^\top u0.8220 × 0.6 = 0.4932
det⁡∂g/∂z=1+u⊤ψ\det\partial g/\partial z=1+u^\top\psi1.4932
log⁡N(z;0,I)\log\mathcal N(z;0,I)−2.0079
log⁡pX(x)\log p_X(x)−2.0079 − 0.4009 = −2.4088
Computed in Python (float64)

Reading the example, and a bits/dim conversion

  1. 1The layer stretches area near zz by a factor of 1.4932, so the density at xx is lower than at zz by log⁡1.4932=0.4009\log1.4932=0.4009 nats.
  2. 2The full 2×22\times2 Jacobian is I+uψ⊤I+u\psi^\top. Its determinant equals 1+ψ⊤u1+\psi^\top u without forming the matrix. In DD dimensions this costs O(D)O(D), against O(D3)O(D^3) for a dense determinant.
  3. 3Bits/dim: suppose a flow assigns log⁡pY(y)=9600\log p_Y(y)=9600 nats to a dequantised CIFAR-10 image scaled to [0,1)3072[0,1)^{3072}. A positive log-density is fine, because densities on a unit cube can exceed 1.
  4. 4Rescaling to the integer grid subtracts 3072ln⁡256=17034.83072\ln256=17034.8 nats, so log⁡pX(x)=−7434.8\log p_X(x)=-7434.8. Then bpd=7434.8/(3072ln⁡2)=3.4916\text{bpd}=7434.8/(3072\ln2)=3.4916, which is RealNVP's level on CIFAR-10.
python
1import torch, torch.nn as nn, torch.nn.functional as F, math
2
3class Planar(nn.Module):
4 """f(z) = z + u_hat * tanh(w.z + b), with w.u_hat >= -1 so f is invertible."""
5 def __init__(self, d):
6 super().__init__()
7 self.w = nn.Parameter(torch.randn(d) * 0.1)
8 self.u = nn.Parameter(torch.randn(d) * 0.1)
9 self.b = nn.Parameter(torch.zeros(1))
10
11 def u_hat(self): # Rezende & Mohamed, appendix A.1
12 wu = self.w @ self.u
13 return self.u + (-1 + F.softplus(wu) - wu) * self.w / (self.w @ self.w)
14
15 def forward(self, z): # (N, d) -> (N, d), (N,)
16 u = self.u_hat()
17 a = z @ self.w + self.b # (N,)
18 h = torch.tanh(a)
19 psi = (1 - h ** 2)[:, None] * self.w # (N, d)
20 logdet = torch.log((1 + psi @ u).abs() + 1e-8) # det(I + u psi^T) = 1 + u^T psi
21 return z + u * h[:, None], logdet
22
23def U1(z): # Rezende & Mohamed's test energy 1
24 r = z.norm(dim=1)
25 lobes = torch.stack([-0.5 * ((z[:, 0] - 2) / 0.6) ** 2, -0.5 * ((z[:, 0] + 2) / 0.6) ** 2])
26 return 0.5 * ((r - 2) / 0.4) ** 2 - torch.logsumexp(lobes, 0)
27
28torch.manual_seed(0)
29flow = nn.ModuleList(Planar(2) for _ in range(8))
30opt = torch.optim.Adam(flow.parameters(), lr=1e-2)
31for step in range(300): # reverse KL: E_q[log q(x) + U(x)]
32 z = torch.randn(256, 2)
33 logq = -0.5 * (z ** 2).sum(1) - math.log(2 * math.pi)
34 for f in flow:
35 z, ld = f(z)
36 logq = logq - ld # log q(z_k) = log q(z_0) - sum log|det|
37 loss = (logq + U1(z)).mean()
38 opt.zero_grad(); loss.backward(); opt.step()
39print(z.shape, logq.shape) # torch.Size([256, 2]) torch.Size([256])
40
41# check the O(D) determinant against the full Jacobian on one point
42f = flow[0]; z0 = torch.randn(1, 2)
43J = torch.autograd.functional.jacobian(lambda v: f(v[None])[0][0], z0[0])
44print(J.shape) # torch.Size([2, 2])
45print(torch.allclose(torch.linalg.det(J).abs().log(), f(z0)[1][0], atol=1e-5)) # True
46
47def bits_per_dim(log_px_unit, D):
48 """log p of dequantised data scaled to [0,1)^D (nats) -> bits per 8-bit dimension."""
49 return -(log_px_unit - D * math.log(256)) / (D * math.log(2))
50
51print(round(bits_per_dim(torch.tensor(9600.0), 3 * 32 * 32).item(), 4)) # 3.4916
A planar flow trained by reverse KL on Rezende & Mohamed's energy U₁, with the O(D) determinant checked against the full Jacobian

Use when

  • +Exact log-likelihood: direct maximum-likelihood training, model comparison in bits/dim, density evaluation for anomaly scores
  • +Exact inverse: every data point has a latent code, and encoding is lossless
  • +Coupling flows sample in one parallel pass; training is stable (no adversary, no posterior)
  • +Modular: any invertible layer with a tractable Jacobian can be stacked

Avoid when

  • −Architectural restrictions (invertibility, cheap determinant) cost expressivity; many layers are needed
  • −Latent dimension equals data dimension: large memory, no compression
  • −Sample quality and likelihood trail diffusion and autoregressive models on images
  • −A bijection keeps topology: mapping a Gaussian onto disconnected modes needs very steep, near-singular maps

Check yourself

A flow layer maps a neighbourhood of x to a region of z with 4 times the volume (|det ∂f/∂x| = 4). How does the density p_X(x) relate to p_Z(z)?

Check yourself

You need to evaluate log p(x) on millions of external data points quickly, and sampling speed does not matter. Which design fits best?

Remember

  • ✓Change of variables: log⁡pX(x)=log⁡pZ(f(x))+log⁡∣det⁡∂f/∂x⊤∣\log p_X(x)=\log p_Z(f(x))+\log|\det\partial f/\partial x^\top|, with ff invertible and dim⁡z=dim⁡x\dim z=\dim x.
  • ✓Composition: f=fK∘⋯∘f1f=f_K\circ\dots\circ f_1 gives log⁡pX(x)=log⁡pZ(hK)+∑klog⁡∣det⁡Jk∣\log p_X(x)=\log p_Z(h_K)+\sum_k\log|\det J_k|; each layer is built for an O(D)O(D) determinant.
  • ✓Planar flow z+utanh⁡(w⊤z+b)z+u\tanh(w^\top z+b): det⁡=1+u⊤ψ\det=1+u^\top\psi by the matrix determinant lemma; invertible iff w⊤u≥−1w^\top u\ge-1.
  • ✓Coupling layers (NICE, RealNVP, Glow) have triangular Jacobians and are cheap in both directions; MAF is fast for density, IAF for sampling.
  • ✓Continuous flows replace log⁡∣det⁡∣\log|\det| by −∫tr⁡(∂v/∂z) dt-\int\operatorname{tr}(\partial v/\partial z)\,dt; flow matching trains such flows without simulation.
  • ✓Report likelihoods on dequantised data in bits/dim =(−log⁡pY(y)+Dln⁡256)/(Dln⁡2)=(-\log p_Y(y)+D\ln256)/(D\ln2); 8 is the uniform baseline.
← back to Deep Learning roadmap