Definition
Normalizing flow A generative model that maps a sample from a simple base density (usually ) through an invertible, differentiable function with inverse . By the change-of-variables formula the model density is exact: . Sampling is followed by one pass of . A flow is built by composing many simple bijections, and each layer is designed so that its Jacobian determinant costs rather than .
A Variational autoencoder (VAE) only bounds the likelihood, and a Generative adversarial network (GAN) has none at all. Autoregressive models give exact likelihoods but sample one dimension at a time (see Autoregressive generative models). Flows give an exact likelihood, a latent code for every data point, and, for coupling-based designs, one-pass sampling. The idea of transforming a density by a chain of simple maps goes back to Tabak and Vanden-Eijnden (2010) and Tabak and Turner (2013). The name and its use in deep learning come from Danilo Rezende and Shakir Mohamed, "Variational Inference with Normalizing Flows" (ICML 2015). They used planar and radial flows to make VAE posteriors more flexible. Laurent Dinh, David Krueger and Yoshua Bengio's NICE (arXiv 2014, ICLR 2015 workshop) introduced coupling layers for density estimation of data. That line led to RealNVP and Glow.
Change of variables
- the normalizing direction, data noise; is the generative direction
- the Jacobian; is the local factor by which scales volume
- base density, usually :
The determinant accounts for volume. Probability mass in a small region is preserved by the map, so if stretches a neighbourhood of by a factor of 3 in area, the density at is 3 times lower. In one dimension this is . Both and must be bijections between and , so and have the same dimension. A flow never compresses.
Composing bijections
- one layer; each must be invertible with a tractable determinant
- Jacobians of a composition multiply (chain rule), so their log-determinants add
Training maximises directly, which is the same as minimising the forward KL divergence from the data distribution to the model. One forward pass produces and the running sum of log-determinants. Sampling runs the layers backwards, . Each layer can be weak, because depth provides the expressivity. What any layer must provide is (1) invertibility, (2) a cheap , and (3) a cheap inverse if you want to sample. A generic dense layer with a nonlinearity fails (2): of a matrix costs , about flops for one image, at every layer and every step.
Planar flows
- parameters; the layer pushes along by an amount that depends on its side of the hyperplane
- matrix determinant lemma: , an dot product
Rezende and Mohamed's planar layer is invertible when . They enforce this by replacing with , where . Its inverse has no closed form: you solve a scalar equation for . That is acceptable in their setting, which is variational inference: samples go from to , and the density of each sample is needed only at the point that was just generated. The bottleneck is capacity. Each layer modifies the density along one direction only, so planar flows need many layers in high dimensions. The paper also has radial flows, , which contract or expand around a point.
Coupling layers: NICE
- any neural network; never inverted and never differentiated for the determinant
- the Jacobian is unit lower triangular: additive coupling preserves volume
NICE (non-linear independent components estimation) splits the input, copies one half, and shifts the other half by a function of the copy. The inverse is , which is exact and as cheap as the forward pass. Layers alternate which half is copied. Because every coupling has determinant 1, NICE ends with a learned diagonal scaling layer so that the model can change volume at all. RealNVP made the coupling affine, , with , and added a multi-scale image architecture. Glow added actnorm and a learned invertible convolution in place of fixed permutations.
Autoregressive flows: MAF and IAF
- outputs of a masked autoregressive network (MADE): output depends only on inputs
- triangular Jacobian: for MAF (), for IAF ()
An autoregressive flow transforms each dimension conditioned on all previous ones. It is strictly more general than a coupling layer, which is the special case where the first dimensions condition everything else in one block. The cost is asymmetric. In MAF (masked autoregressive flow; Papamakarios, Pavlakou and Murray, NeurIPS 2017) the conditioners read , so the density of a data point takes one parallel network pass, but sampling needs sequential passes, since requires . IAF (inverse autoregressive flow; Kingma, Salimans, Jozefowicz, Chen, Sutskever and Welling, NeurIPS 2016) is the same map run in the other direction. Its conditioners read , so sampling is one pass and the density of the model's own samples is free, but the density of an external needs sequential steps. That made IAF a natural fit for VAE posteriors and for fast synthesis, for example Parallel WaveNet. Coupling layers sit in between: both directions take one pass, and each layer is less expressive.
Continuous-time flows
- any network; the ODE solution map is a bijection for well-behaved
- instantaneous change of variables: a trace instead of a determinant
Neural ODEs (Chen, Rubanova, Bettencourt and Duvenaud, NeurIPS 2018) take the limit of infinitely many infinitesimal layers. The log-density then evolves by the trace of the Jacobian. FFJORD (Grathwohl et al., ICLR 2019) estimates that trace without bias using Hutchinson's estimator , one vector-Jacobian product per sample, so needs no structure at all. The price is an ODE solve, with many function evaluations, for every likelihood and every sample, and simulation during training. Flow matching and Rectified flow keep the continuous flow and train by regression, with no simulation and no likelihood during training.
Dequantisation and bits per dimension
Images are discrete: each subpixel takes one of 256 values. A continuous density can put arbitrarily high density on those points and drive the likelihood to infinity. The standard fix is uniform dequantisation: train on with . Theis, van den Oord and Bethge (ICLR 2016) showed that the continuous log-likelihood of dequantised data is a lower bound on the discrete log-likelihood of . Flow++ (Ho et al., ICML 2019) learned the noise distribution instead (variational dequantisation), which tightens the bound. Results are reported in bits per dimension:
- density of the model on data scaled to ; rescaling by 256 adds nats
- number of subpixels, e.g. for CIFAR-10
- bits/dim of a model that puts uniform mass on all images; lower is better
Worked example: one planar layer
Take , , , , . Here , so the layer is invertible.
| quantity | value |
|---|---|
| 0.5 − 0.15 + 0.1 = 0.45 | |
| 0.4219 | |
| (0.8375, −0.4688) | |
| 0.8220 | |
| 0.8220 × 0.6 = 0.4932 | |
| 1.4932 | |
| −2.0079 | |
| −2.0079 − 0.4009 = −2.4088 |
Reading the example, and a bits/dim conversion
- 1The layer stretches area near by a factor of 1.4932, so the density at is lower than at by nats.
- 2The full Jacobian is . Its determinant equals without forming the matrix. In dimensions this costs , against for a dense determinant.
- 3Bits/dim: suppose a flow assigns nats to a dequantised CIFAR-10 image scaled to . A positive log-density is fine, because densities on a unit cube can exceed 1.
- 4Rescaling to the integer grid subtracts nats, so . Then , which is RealNVP's level on CIFAR-10.
1import torch, torch.nn as nn, torch.nn.functional as F, math23class Planar(nn.Module):4 """f(z) = z + u_hat * tanh(w.z + b), with w.u_hat >= -1 so f is invertible."""5 def __init__(self, d):6 super().__init__()7 self.w = nn.Parameter(torch.randn(d) * 0.1)8 self.u = nn.Parameter(torch.randn(d) * 0.1)9 self.b = nn.Parameter(torch.zeros(1))1011 def u_hat(self): # Rezende & Mohamed, appendix A.112 wu = self.w @ self.u13 return self.u + (-1 + F.softplus(wu) - wu) * self.w / (self.w @ self.w)1415 def forward(self, z): # (N, d) -> (N, d), (N,)16 u = self.u_hat()17 a = z @ self.w + self.b # (N,)18 h = torch.tanh(a)19 psi = (1 - h ** 2)[:, None] * self.w # (N, d)20 logdet = torch.log((1 + psi @ u).abs() + 1e-8) # det(I + u psi^T) = 1 + u^T psi21 return z + u * h[:, None], logdet2223def U1(z): # Rezende & Mohamed's test energy 124 r = z.norm(dim=1)25 lobes = torch.stack([-0.5 * ((z[:, 0] - 2) / 0.6) ** 2, -0.5 * ((z[:, 0] + 2) / 0.6) ** 2])26 return 0.5 * ((r - 2) / 0.4) ** 2 - torch.logsumexp(lobes, 0)2728torch.manual_seed(0)29flow = nn.ModuleList(Planar(2) for _ in range(8))30opt = torch.optim.Adam(flow.parameters(), lr=1e-2)31for step in range(300): # reverse KL: E_q[log q(x) + U(x)]32 z = torch.randn(256, 2)33 logq = -0.5 * (z ** 2).sum(1) - math.log(2 * math.pi)34 for f in flow:35 z, ld = f(z)36 logq = logq - ld # log q(z_k) = log q(z_0) - sum log|det|37 loss = (logq + U1(z)).mean()38 opt.zero_grad(); loss.backward(); opt.step()39print(z.shape, logq.shape) # torch.Size([256, 2]) torch.Size([256])4041# check the O(D) determinant against the full Jacobian on one point42f = flow[0]; z0 = torch.randn(1, 2)43J = torch.autograd.functional.jacobian(lambda v: f(v[None])[0][0], z0[0])44print(J.shape) # torch.Size([2, 2])45print(torch.allclose(torch.linalg.det(J).abs().log(), f(z0)[1][0], atol=1e-5)) # True4647def bits_per_dim(log_px_unit, D):48 """log p of dequantised data scaled to [0,1)^D (nats) -> bits per 8-bit dimension."""49 return -(log_px_unit - D * math.log(256)) / (D * math.log(2))5051print(round(bits_per_dim(torch.tensor(9600.0), 3 * 32 * 32).item(), 4)) # 3.4916Use when
- +Exact log-likelihood: direct maximum-likelihood training, model comparison in bits/dim, density evaluation for anomaly scores
- +Exact inverse: every data point has a latent code, and encoding is lossless
- +Coupling flows sample in one parallel pass; training is stable (no adversary, no posterior)
- +Modular: any invertible layer with a tractable Jacobian can be stacked
Avoid when
- −Architectural restrictions (invertibility, cheap determinant) cost expressivity; many layers are needed
- −Latent dimension equals data dimension: large memory, no compression
- −Sample quality and likelihood trail diffusion and autoregressive models on images
- −A bijection keeps topology: mapping a Gaussian onto disconnected modes needs very steep, near-singular maps
Check yourself
A flow layer maps a neighbourhood of x to a region of z with 4 times the volume (|det ∂f/∂x| = 4). How does the density p_X(x) relate to p_Z(z)?
Check yourself
You need to evaluate log p(x) on millions of external data points quickly, and sampling speed does not matter. Which design fits best?
Remember
- ✓Change of variables: , with invertible and .
- ✓Composition: gives ; each layer is built for an determinant.
- ✓Planar flow : by the matrix determinant lemma; invertible iff .
- ✓Coupling layers (NICE, RealNVP, Glow) have triangular Jacobians and are cheap in both directions; MAF is fast for density, IAF for sampling.
- ✓Continuous flows replace by ; flow matching trains such flows without simulation.
- ✓Report likelihoods on dequantised data in bits/dim ; 8 is the uniform baseline.