# Normalizing flows

> A normalizing flow writes data as an invertible, differentiable transform x = g(z) of simple noise z, so the exact log-likelihood is log p(z) plus the log-determinant of the Jacobian of g⁻¹; the whole design problem is building bijections whose determinants are cheap.

Source: https://www.mlfoundry.in/learn/normalizing-flows  
Section: Deep Learning › Generative Deep Learning Algorithms  
Level: core · 15 min read · updated 2026-10-12

**Normalizing flow:** A generative model that maps a sample $z$ from a simple base density $p_Z$ (usually $\mathcal N(0,I)$) through an **invertible, differentiable** function $x=g(z)$ with inverse $f=g^{-1}$. By the change-of-variables formula the model density is exact: $\log p_X(x)=\log p_Z(f(x))+\log\big|\det \partial f(x)/\partial x^\top\big|$. Sampling is $z\sim p_Z$ followed by one pass of $g$. A flow is built by composing many simple bijections, and each layer is designed so that its Jacobian determinant costs $O(D)$ rather than $O(D^3)$.

A **Variational autoencoder (VAE)** only bounds the likelihood, and a **Generative adversarial network (GAN)** has none at all. Autoregressive models give exact likelihoods but sample one dimension at a time (see **Autoregressive generative models**). Flows give an exact likelihood, a latent code for every data point, and, for coupling-based designs, one-pass sampling. The idea of transforming a density by a chain of simple maps goes back to Tabak and Vanden-Eijnden (2010) and Tabak and Turner (2013). The name and its use in deep learning come from Danilo Rezende and Shakir Mohamed, "Variational Inference with Normalizing Flows" (ICML 2015). They used planar and radial flows to make VAE posteriors more flexible. Laurent Dinh, David Krueger and Yoshua Bengio's NICE (arXiv 2014, ICLR 2015 workshop) introduced coupling layers for density estimation of data. That line led to **RealNVP** and **Glow**.

## Change of variables

$$
p_X(x)=p_Z\big(f(x)\big)\,\Big|\det\frac{\partial f(x)}{\partial x^\top}\Big|\qquad\Longleftrightarrow\qquad \log p_X(x)=\log p_Z(z)-\log\Big|\det\frac{\partial g(z)}{\partial z^\top}\Big|,\ \ z=f(x)
$$

- $f=g^{-1}$: the normalizing direction, data $\to$ noise; $g$ is the generative direction
- $\partial f/\partial x^\top$: the $D\times D$ Jacobian; $|\det|$ is the local factor by which $f$ scales volume
- $p_Z$: base density, usually $\mathcal N(0,I)$: $\log p_Z(z)=-\tfrac12\|z\|^2-\tfrac D2\log 2\pi$

The determinant accounts for volume. Probability mass in a small region is preserved by the map, so if $g$ stretches a neighbourhood of $z$ by a factor of 3 in area, the density at $x=g(z)$ is 3 times lower. In one dimension this is $p_X(x)=p_Z(z)\,|dz/dx|$. Both $f$ and $g$ must be bijections between $\mathbb R^D$ and $\mathbb R^D$, so $x$ and $z$ have the same dimension. A flow never compresses.

## Composing bijections

$$
f=f_K\circ\cdots\circ f_1,\qquad h_0=x,\ \ h_k=f_k(h_{k-1}),\qquad \log p_X(x)=\log p_Z(h_K)+\sum_{k=1}^{K}\log\Big|\det\frac{\partial f_k}{\partial h_{k-1}^\top}\Big|
$$

- $f_k$: one layer; each must be invertible with a tractable determinant
- $\sum_k$: Jacobians of a composition multiply (chain rule), so their log-determinants add

Training maximises $\frac1N\sum_n\log p_X(x_n)$ directly, which is the same as minimising the forward KL divergence from the data distribution to the model. One forward pass $x\to z$ produces $z$ and the running sum of log-determinants. Sampling runs the layers backwards, $z\to x$. Each layer can be weak, because depth provides the expressivity. What any layer must provide is (1) invertibility, (2) a cheap $\log|\det|$, and (3) a cheap inverse if you want to sample. A generic dense layer $Wx$ with a nonlinearity fails (2): $\det$ of a $D\times D$ matrix costs $O(D^3)$, about $2\cdot10^{10}$ flops for one $32\times32\times3$ image, at every layer and every step.

## Planar flows

$$
g(z)=z+u\,\tanh(w^\top z+b),\qquad \det\frac{\partial g}{\partial z^\top}=1+u^\top\psi(z),\quad \psi(z)=\big(1-\tanh^2(w^\top z+b)\big)\,w
$$

- $u,w\in\mathbb R^D,\ b\in\mathbb R$: parameters; the layer pushes $z$ along $u$ by an amount that depends on its side of the hyperplane $w^\top z+b=0$
- $1+u^\top\psi$: matrix determinant lemma: $\det(I+u\psi^\top)=1+\psi^\top u$, an $O(D)$ dot product

Rezende and Mohamed's planar layer is invertible when $w^\top u\ge-1$. They enforce this by replacing $u$ with $\hat u=u+\big[m(w^\top u)-w^\top u\big]\frac{w}{\|w\|^2}$, where $m(a)=-1+\log(1+e^a)$. Its inverse has no closed form: you solve a scalar equation for $w^\top z$. That is acceptable in their setting, which is variational inference: samples go from $z$ to $x$, and the density of each sample is needed only at the point that was just generated. The bottleneck is capacity. Each layer modifies the density along one direction only, so planar flows need many layers in high dimensions. The paper also has **radial flows**, $g(z)=z+\beta h(\alpha,r)(z-z_0)$, which contract or expand around a point.

> Figure: Up to four planar layers g(z) = z + u·tanh(wᵀz + b) push a 2-D standard Gaussian. The weights are fixed (not trained): |w| = 2.5 with preset directions, and λ sets wᵀu = λ, 0.8λ, 0.6λ, 0.5λ for layers 1–4. The left panel is the density given by the change-of-variables formula: invert the stack (bisection on wᵀz), then subtract Σ log|1 + uᵀψ|. The right panel is a histogram of 40,000 pushed samples. With K = 0 their total-variation distance is 0.049, which is pure Monte-Carlo noise. With K = 2 and λ = 3 (wᵀu = 3.0, 2.4) the Gaussian splits into four lobes. The panels still agree (TV 0.071, ∫q = 1.000), E[Σ log|det|] = 0.952, and the entropy rises from H(N) = 2.838 to 3.790 nats. With K = 4 the lobes merge into a ring: TV 0.082, and the formula's mass inside the box, 0.987, equals the fraction of samples inside it. Back at K = 2, press fold (λ = −1.2, raw u): layer 1 has wᵀu = −1.2 < −1, the Jacobian factor reaches −0.200, and the map is no longer one-to-one. The formula's mass drops to 0.662 and TV jumps to 0.194, while every sample is still inside the box. Switch to constrained û and the same λ gives wᵀû = −0.737 and −0.676. The minimum factor is then 0.263 > 0, and the panels agree again (TV 0.044).

## Coupling layers: NICE

$$
y_{1:d}=x_{1:d},\qquad y_{d+1:D}=x_{d+1:D}+m(x_{1:d}),\qquad \frac{\partial y}{\partial x^\top}=\begin{pmatrix}I&0\\ \partial m/\partial x_{1:d}^\top&I\end{pmatrix}
$$

- $m$: any neural network; never inverted and never differentiated for the determinant
- $\det=1$: the Jacobian is unit lower triangular: additive coupling preserves volume

NICE (non-linear independent components estimation) splits the input, copies one half, and shifts the other half by a function of the copy. The inverse is $x_{d+1:D}=y_{d+1:D}-m(y_{1:d})$, which is exact and as cheap as the forward pass. Layers alternate which half is copied. Because every coupling has determinant 1, NICE ends with a learned diagonal scaling layer so that the model can change volume at all. **RealNVP** made the coupling affine, $x_u\odot e^{s(x_c)}+t(x_c)$, with $\log|\det|=\sum s$, and added a multi-scale image architecture. **Glow** added actnorm and a learned invertible $1\times1$ convolution in place of fixed permutations.

## Autoregressive flows: MAF and IAF

$$
\text{MAF:}\ \ z_i=\big(x_i-\mu_i(x_{<i})\big)\,e^{-\alpha_i(x_{<i})}\qquad\qquad \text{IAF:}\ \ x_i=z_i\,e^{\sigma_i(z_{<i})}+\mu_i(z_{<i})
$$

- $\mu_i,\alpha_i$: outputs of a masked autoregressive network (MADE): output $i$ depends only on inputs $<i$
- $\log|\det|$: triangular Jacobian: $-\sum_i\alpha_i$ for MAF ($x\to z$), $\sum_i\sigma_i$ for IAF ($z\to x$)

An autoregressive flow transforms each dimension conditioned on all previous ones. It is strictly more general than a coupling layer, which is the special case where the first $d$ dimensions condition everything else in one block. The cost is asymmetric. In **MAF** (masked autoregressive flow; Papamakarios, Pavlakou and Murray, NeurIPS 2017) the conditioners read $x$, so the density of a data point takes one parallel network pass, but sampling needs $D$ sequential passes, since $x_i$ requires $x_{<i}$. **IAF** (inverse autoregressive flow; Kingma, Salimans, Jozefowicz, Chen, Sutskever and Welling, NeurIPS 2016) is the same map run in the other direction. Its conditioners read $z$, so sampling is one pass and the density of the model's own samples is free, but the density of an external $x$ needs $D$ sequential steps. That made IAF a natural fit for VAE posteriors and for fast synthesis, for example Parallel WaveNet. Coupling layers sit in between: both directions take one pass, and each layer is less expressive.

## Continuous-time flows

$$
\frac{dz(t)}{dt}=v_\theta\big(z(t),t\big),\qquad \frac{d\log p\big(z(t)\big)}{dt}=-\operatorname{tr}\Big(\frac{\partial v_\theta}{\partial z^\top}\Big)
$$

- $v_\theta$: any network; the ODE solution map is a bijection for well-behaved $v_\theta$
- $\operatorname{tr}$: instantaneous change of variables: a trace instead of a determinant

Neural ODEs (Chen, Rubanova, Bettencourt and Duvenaud, NeurIPS 2018) take the limit of infinitely many infinitesimal layers. The log-density then evolves by the trace of the Jacobian. FFJORD (Grathwohl et al., ICLR 2019) estimates that trace without bias using Hutchinson's estimator $\mathbb E_\epsilon[\epsilon^\top(\partial v/\partial z)\epsilon]$, one vector-Jacobian product per sample, so $v_\theta$ needs no structure at all. The price is an ODE solve, with many function evaluations, for every likelihood and every sample, and simulation during training. **Flow matching** and **Rectified flow** keep the continuous flow and train $v_\theta$ by regression, with no simulation and no likelihood during training.

## Dequantisation and bits per dimension

Images are discrete: each subpixel takes one of 256 values. A continuous density can put arbitrarily high density on those points and drive the likelihood to infinity. The standard fix is **uniform dequantisation**: train on $y=(x+u)/256$ with $u\sim\mathcal U[0,1)^D$. Theis, van den Oord and Bethge (ICLR 2016) showed that the continuous log-likelihood of dequantised data is a lower bound on the discrete log-likelihood of $x$. Flow++ (Ho et al., ICML 2019) learned the noise distribution instead (**variational dequantisation**), which tightens the bound. Results are reported in **bits per dimension**:

$$
\text{bpd}=\frac{-\log p_X(x)}{D\ln 2}=\frac{-\log p_Y(y)+D\ln 256}{D\ln 2}
$$

- $p_Y$: density of the model on data scaled to $[0,1)^D$; rescaling by 256 adds $D\ln256$ nats
- $D$: number of subpixels, e.g. $3\cdot32\cdot32=3072$ for CIFAR-10
- $8$: bits/dim of a model that puts uniform mass on all $256^D$ images; lower is better

## Worked example: one planar layer

Take $D=2$, $z=(0.5,-0.3)$, $w=(1,0.5)$, $u=(0.8,-0.4)$, $b=0.1$. Here $w^\top u=0.6>-1$, so the layer is invertible.

Computed in Python (float64)

| quantity | value |
| --- | --- |
| $a=w^\top z+b$ | 0.5 − 0.15 + 0.1 = 0.45 |
| $\tanh a$ | 0.4219 |
| $x=z+u\tanh a$ | (0.8375, −0.4688) |
| $1-\tanh^2 a$ | 0.8220 |
| $u^\top\psi=(1-\tanh^2a)\,w^\top u$ | 0.8220 × 0.6 = 0.4932 |
| $\det\partial g/\partial z=1+u^\top\psi$ | 1.4932 |
| $\log\mathcal N(z;0,I)$ | −2.0079 |
| $\log p_X(x)$ | −2.0079 − 0.4009 = −2.4088 |

**Reading the example, and a bits/dim conversion**

1. The layer stretches area near $z$ by a factor of 1.4932, so the density at $x$ is lower than at $z$ by $\log1.4932=0.4009$ nats.
2. The full $2\times2$ Jacobian is $I+u\psi^\top$. Its determinant equals $1+\psi^\top u$ without forming the matrix. In $D$ dimensions this costs $O(D)$, against $O(D^3)$ for a dense determinant.
3. Bits/dim: suppose a flow assigns $\log p_Y(y)=9600$ nats to a dequantised CIFAR-10 image scaled to $[0,1)^{3072}$. A positive log-density is fine, because densities on a unit cube can exceed 1.
4. Rescaling to the integer grid subtracts $3072\ln256=17034.8$ nats, so $\log p_X(x)=-7434.8$. Then $\text{bpd}=7434.8/(3072\ln2)=3.4916$, which is RealNVP's level on CIFAR-10.

```python
import torch, torch.nn as nn, torch.nn.functional as F, math

class Planar(nn.Module):
    """f(z) = z + u_hat * tanh(w.z + b), with w.u_hat >= -1 so f is invertible."""
    def __init__(self, d):
        super().__init__()
        self.w = nn.Parameter(torch.randn(d) * 0.1)
        self.u = nn.Parameter(torch.randn(d) * 0.1)
        self.b = nn.Parameter(torch.zeros(1))

    def u_hat(self):                                   # Rezende & Mohamed, appendix A.1
        wu = self.w @ self.u
        return self.u + (-1 + F.softplus(wu) - wu) * self.w / (self.w @ self.w)

    def forward(self, z):                              # (N, d) -> (N, d), (N,)
        u = self.u_hat()
        a = z @ self.w + self.b                        # (N,)
        h = torch.tanh(a)
        psi = (1 - h ** 2)[:, None] * self.w           # (N, d)
        logdet = torch.log((1 + psi @ u).abs() + 1e-8)  # det(I + u psi^T) = 1 + u^T psi
        return z + u * h[:, None], logdet

def U1(z):                                             # Rezende & Mohamed's test energy 1
    r = z.norm(dim=1)
    lobes = torch.stack([-0.5 * ((z[:, 0] - 2) / 0.6) ** 2, -0.5 * ((z[:, 0] + 2) / 0.6) ** 2])
    return 0.5 * ((r - 2) / 0.4) ** 2 - torch.logsumexp(lobes, 0)

torch.manual_seed(0)
flow = nn.ModuleList(Planar(2) for _ in range(8))
opt = torch.optim.Adam(flow.parameters(), lr=1e-2)
for step in range(300):                                # reverse KL: E_q[log q(x) + U(x)]
    z = torch.randn(256, 2)
    logq = -0.5 * (z ** 2).sum(1) - math.log(2 * math.pi)
    for f in flow:
        z, ld = f(z)
        logq = logq - ld                               # log q(z_k) = log q(z_0) - sum log|det|
    loss = (logq + U1(z)).mean()
    opt.zero_grad(); loss.backward(); opt.step()
print(z.shape, logq.shape)                             # torch.Size([256, 2]) torch.Size([256])

# check the O(D) determinant against the full Jacobian on one point
f = flow[0]; z0 = torch.randn(1, 2)
J = torch.autograd.functional.jacobian(lambda v: f(v[None])[0][0], z0[0])
print(J.shape)                                         # torch.Size([2, 2])
print(torch.allclose(torch.linalg.det(J).abs().log(), f(z0)[1][0], atol=1e-5))  # True

def bits_per_dim(log_px_unit, D):
    """log p of dequantised data scaled to [0,1)^D (nats) -> bits per 8-bit dimension."""
    return -(log_px_unit - D * math.log(256)) / (D * math.log(2))

print(round(bits_per_dim(torch.tensor(9600.0), 3 * 32 * 32).item(), 4))  # 3.4916
```

A planar flow trained by reverse KL on Rezende & Mohamed's energy U₁, with the O(D) determinant checked against the full Jacobian

> **Two directions, two costs:** Every flow has a density direction $x\to z$ and a sampling direction $z\to x$, and the architecture decides which one is cheap. Planar flows and IAF are fast from $z$ to $x$. MAF is fast from $x$ to $z$. Coupling layers (**RealNVP**, **Glow**) are fast in both directions, and they pay for it with less expressive layers. Pick the cheap direction to match the job: maximum-likelihood training on data needs $x\to z$; variational inference and synthesis need $z\to x$.

> **Forward and reverse KL behave differently:** Maximum likelihood on data minimises $\mathrm{KL}(p_\text{data}\,\|\,p_X)$, which is mass-covering: the model is punished for missing any mode. Fitting a known unnormalised target $e^{-U}$, as in the code above, minimises $\mathrm{KL}(q\,\|\,p)=\mathbb E_q[\log q+U]+\log Z$. That objective is mode-seeking, and a weak flow can drop one lobe of a two-lobed target without a large penalty. Check whether your flow is trained on samples or on an energy before interpreting its failures.

> **Dequantise, and include every Jacobian:** Without dequantisation, a flow trained on 8-bit images can put spikes on the integer grid and report a meaningless, ever-improving likelihood. Every preprocessing step that changes volume, such as scaling to $[0,1)$, the logit transform, or centring with scaling, needs its log-determinant in the reported number. A model that is uniform over pixel values scores exactly 8 bits/dim, which is a useful sanity check.

> **Same dimension, always:** A bijection cannot reduce dimension, so a flow's latent for a $256\times256\times3$ image has 196,608 numbers, and the model must spend capacity on noise-level detail. Multi-scale factor-out (**RealNVP**) only splits where those dimensions are modelled. Latent-variable models with a decoder, such as the **Variational autoencoder (VAE)** and **Latent diffusion models**, give up the exact likelihood in exchange for compression.

**Strengths**

- Exact log-likelihood: direct maximum-likelihood training, model comparison in bits/dim, density evaluation for anomaly scores
- Exact inverse: every data point has a latent code, and encoding is lossless
- Coupling flows sample in one parallel pass; training is stable (no adversary, no posterior)
- Modular: any invertible layer with a tractable Jacobian can be stacked

**Limitations**

- Architectural restrictions (invertibility, cheap determinant) cost expressivity; many layers are needed
- Latent dimension equals data dimension: large memory, no compression
- Sample quality and likelihood trail diffusion and autoregressive models on images
- A bijection keeps topology: mapping a Gaussian onto disconnected modes needs very steep, near-singular maps

**Question:** A flow layer maps a neighbourhood of x to a region of z with 4 times the volume (|det ∂f/∂x| = 4). How does the density p_X(x) relate to p_Z(z)?

- A. p_X(x) = p_Z(z) / 4
- B. p_X(x) = 4 · p_Z(z)
- C. p_X(x) = p_Z(z), densities are preserved by bijections
- D. p_X(x) = p_Z(z) + log 4

**Answer:** B. p_X(x) = 4 · p_Z(z) — $p_X(x)=p_Z(f(x))\,|\det\partial f/\partial x|=4\,p_Z(z)$. The same probability mass sits in an $x$-region 4 times smaller than its $z$-image, so the density in $x$-space is 4 times higher. In logs, $\log p_X=\log p_Z+\log4$; the last option adds $\log4$ to a density, not to a log-density.

**Question:** You need to evaluate log p(x) on millions of external data points quickly, and sampling speed does not matter. Which design fits best?

- A. Inverse autoregressive flow (IAF)
- B. Masked autoregressive flow (MAF)
- C. Planar flow
- D. A GAN with an inverted generator

**Answer:** B. Masked autoregressive flow (MAF) — MAF's conditioners read $x$, so $z=f(x)$ and $\log|\det|=-\sum\alpha_i(x_{<i})$ come out of one parallel pass. Its sampling is sequential, but that does not matter here. IAF is the reverse: the density of an external point needs $D$ sequential steps. A planar flow has no closed-form inverse, and a GAN has no density.

**Key points**

- Change of variables: $\log p_X(x)=\log p_Z(f(x))+\log|\det\partial f/\partial x^\top|$, with $f$ invertible and $\dim z=\dim x$.
- Composition: $f=f_K\circ\dots\circ f_1$ gives $\log p_X(x)=\log p_Z(h_K)+\sum_k\log|\det J_k|$; each layer is built for an $O(D)$ determinant.
- Planar flow $z+u\tanh(w^\top z+b)$: $\det=1+u^\top\psi$ by the matrix determinant lemma; invertible iff $w^\top u\ge-1$.
- Coupling layers (NICE, RealNVP, Glow) have triangular Jacobians and are cheap in both directions; MAF is fast for density, IAF for sampling.
- Continuous flows replace $\log|\det|$ by $-\int\operatorname{tr}(\partial v/\partial z)\,dt$; flow matching trains such flows without simulation.
- Report likelihoods on dequantised data in bits/dim $=(-\log p_Y(y)+D\ln256)/(D\ln2)$; 8 is the uniform baseline.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
