RealNVP

RealNVP (Dinh, Sohl-Dickstein and Bengio, 2017) builds an invertible image model from affine coupling layers y = x⊙exp(s) + t whose Jacobian is triangular, so log|det| is just Σs; checkerboard and channel masks, squeezing and a multi-scale factor-out make it work on images.

14 min read1147 wordscore0 viewsupdated 2026-10-12

Definition

RealNVP (real-valued non-volume preserving transformations) is the normalizing flow of Laurent Dinh, Jascha Sohl-Dickstein and Samy Bengio, "Density estimation using Real NVP" (ICLR 2017). It stacks affine coupling layers: a binary mask splits the input into a part xcx_c that is copied and a part xux_u that is scaled and shifted, yu=xu⊙exp⁡(s(xc))+t(xc)y_u=x_u\odot\exp(s(x_c))+t(x_c), with ss and tt arbitrary deep networks. The Jacobian is triangular, so log⁡∣det⁡J∣=∑s(xc)\log|\det J|=\sum s(x_c), and the inverse costs one more pass of ss and tt. Masks alternate between layers, and a multi-scale architecture with squeeze and factor-out operations makes it practical for images.

A flow needs three things at once: an invertible map, a cheap Jacobian determinant, and enough flexibility to turn a Gaussian into images (see Normalizing flows). Dinh, Krueger and Bengio's NICE (2014) got the first two with additive coupling, yu=xu+m(xc)y_u=x_u+m(x_c). Its Jacobian determinant is exactly 1, so every layer preserves volume, and only a final diagonal scaling could change it. RealNVP adds a learned, input-dependent scale exp⁡(s(xc))\exp(s(x_c)). The determinant is then no longer 1, which is the "non-volume preserving" in the name. Combined with convolutional residual networks for ss and tt, batch normalization, and a multi-scale design, it produced sharp samples and exact log-likelihoods on CIFAR-10, downsampled ImageNet, LSUN and CelebA. It was the first flow competitive on natural images, and Glow keeps its coupling layer and multi-scale skeleton almost unchanged.

The affine coupling layer

y1:d=x1:d,yd+1:D=xd+1:D⊙exp⁡ ⁣(s(x1:d))+t(x1:d)y_{1:d}=x_{1:d},\qquad y_{d+1:D}=x_{d+1:D}\odot\exp\!\big(s(x_{1:d})\big)+t(x_{1:d})
x1:dx_{1:d}
the conditioning half, copied unchanged (xcx_c)
xd+1:Dx_{d+1:D}
the transformed half (xux_u), scaled elementwise and shifted
s, t:Rd→RD−ds,\ t:\mathbb R^d\to\mathbb R^{D-d}
log-scale and translation; any network, never inverted or differentiated through a determinant
⊙\odot
elementwise product
∂y∂x⊤=(Id0∂yd+1:D∂x1:d⊤diag(exp⁡[s(x1:d)])),log⁡∣det⁡∂y∂x⊤∣=∑js(x1:d)j\frac{\partial y}{\partial x^\top}=\begin{pmatrix}I_d & 0\\[2pt] \dfrac{\partial y_{d+1:D}}{\partial x_{1:d}^\top} & \mathrm{diag}\big(\exp[s(x_{1:d})]\big)\end{pmatrix},\qquad \log\Big|\det\frac{\partial y}{\partial x^\top}\Big|=\sum_{j}s(x_{1:d})_j
IdI_d
the copied half depends only on itself
00
the copied half does not read xux_u: the matrix is block lower triangular
∂yu/∂xc\partial y_u/\partial x_c
dense and complicated (it contains the Jacobians of ss and tt), but it does not enter the determinant
diag(exp⁡s)\mathrm{diag}(\exp s)
the determinant of a triangular matrix is the product of its diagonal, exp⁡(∑jsj)\exp(\sum_j s_j)
x1:d=y1:d,xd+1:D=(yd+1:D−t(y1:d))⊙exp⁡ ⁣(−s(y1:d))x_{1:d}=y_{1:d},\qquad x_{d+1:D}=\big(y_{d+1:D}-t(y_{1:d})\big)\odot\exp\!\big(-s(y_{1:d})\big)
t(y1:d)t(y_{1:d})
since y1:d=x1:dy_{1:d}=x_{1:d}, the inverse can recompute exactly the same ss and tt

This is the whole trick. Inverting the layer never requires inverting ss or tt, and the log-determinant never requires their Jacobians. So ss and tt can be deep convolutional ResNets with batch normalization, and the layer stays exactly invertible and costs O(D)O(D) for log⁡∣det⁡J∣\log|\det J| instead of O(D3)O(D^3). Both directions cost one evaluation of the networks. That makes sampling as fast as density evaluation, unlike the autoregressive flows of Normalizing flows.

Masks: checkerboard and channel-wise

For images the split is written with a binary mask bb of the input's shape:

y=b⊙x+(1−b)⊙(x⊙exp⁡ ⁣(s(b⊙x))+t(b⊙x))y=b\odot x+(1-b)\odot\Big(x\odot\exp\!\big(s(b\odot x)\big)+t(b\odot x)\Big)
bb
1 on the conditioning entries, 0 on the updated ones
checkerboard\textit{checkerboard}
bij=1b_{ij}=1 where i+ji+j is even (spatial alternation); s,ts,t see each pixel's neighbours
channel-wise\textit{channel-wise}
b=1b=1 on the first half of the channels; s,ts,t see the other channels at every position

A single coupling leaves half of the dimensions untouched, so layers alternate the mask (b→1−bb\to1-b). After two layers every dimension has been updated, and after three every dimension has been updated as a function of every other within the receptive field. Composition keeps everything tractable: Jacobians multiply, so log-determinants add.

Squeeze and the multi-scale architecture

The squeeze operation reshapes an s×s×cs\times s\times c tensor into s2×s2×4c\tfrac s2\times\tfrac s2\times 4c: each 2×22\times2 spatial block becomes four channels at one position. It is a permutation of coordinates, so its log-determinant is 0, and it lets channel-wise masks mix information that was spatially adjacent. One scale of RealNVP is:

One scale of the multi-scale architecture (Dinh et al. 2017, Section 3.6)

  1. 1Three coupling layers with alternating checkerboard masks on the s×s×cs\times s\times c input.
  2. 2Squeeze to s2×s2×4c\tfrac s2\times\tfrac s2\times 4c.
  3. 3Three coupling layers with alternating channel-wise masks.
  4. 4Factor out half of the channels: z(i)z^{(i)} goes straight to the latent vector and is modelled by the Gaussian prior; the other half, h(i)h^{(i)}, is the input of the next scale.
  5. 5Repeat on h(i)h^{(i)}. The final scale uses four coupling layers with alternating checkerboard masks, and its output joins the factored-out latents: z=(z(1),…,z(L))z=(z^{(1)},\dots,z^{(L)}) has exactly DD dimensions.
log⁡pX(x)=∑ilog⁡N(z(i);0,I)+∑klog⁡∣det⁡∂fk∂hk−1⊤∣\log p_X(x)=\sum_{i}\log\mathcal N\big(z^{(i)};0,I\big)+\sum_{k}\log\Big|\det\frac{\partial f_k}{\partial h_{k-1}^\top}\Big|
z(i)z^{(i)}
the latents factored out at scale ii plus the final output; together a bijective image of xx
fkf_k
every coupling (and batch-norm) layer; squeeze and split contribute 0

Factoring out has three effects. Each scale halves the dimensions that later, deeper layers must process, which saves computation and memory. The objective is distributed across the network: early latents receive a training signal after only a few layers, which the authors compare to deep supervision. And the latents form a hierarchy: early-factored z(1)z^{(1)} carries fine local detail, while the last scale carries global structure. In the paper's conceptual-compression figure, keeping the higher-level latents of an image and resampling the lower-level ones preserves its global structure and changes the details.

Batch normalization and preprocessing

RealNVP uses deep residual networks with batch normalization and weight normalization for ss and tt, and also applies batch normalization to the outputs of coupling layers as a flow layer. As a map, x↦(x−μ~)/σ~2+ϵx\mapsto(x-\tilde\mu)/\sqrt{\tilde\sigma^2+\epsilon} is a diagonal rescaling with log-determinant −12∑ilog⁡(σ~i2+ϵ)-\tfrac12\sum_i\log(\tilde\sigma_i^2+\epsilon). Pixels are dequantised with uniform noise and passed through a logit transform before the flow, and the Jacobian of that preprocessing is included when likelihoods are reported in bits per dimension. The prior is an isotropic unit Gaussian.

Setup in the paper

Paper
Dinh, Sohl-Dickstein, Bengio, "Density estimation using Real NVP", ICLR 2017 (arXiv 1605.08803)
Layer
affine coupling yu=xu⊙es(xc)+t(xc)y_u=x_u\odot e^{s(x_c)}+t(x_c); ss, tt are convolutional ResNets
Masks
checkerboard before each squeeze, channel-wise after; masks alternate between consecutive couplings
Scale block
3 checkerboard couplings → squeeze → 3 channel-wise couplings → factor out half; final scale: 4 checkerboard couplings
Normalization
batch normalization and weight normalization in s and t; batch normalization also used as a flow layer
Data
CIFAR-10, downsampled ImageNet 32×32 and 64×64, LSUN (bedroom, tower, church outdoor), CelebA; dequantised and logit-transformed
Prior
N(0,I)\mathcal N(0,I) over all factored-out latents
stage 1/13 · coupling 1: checkerboard□ x_c kept · ■ x_u ← x_u⊙exp(s(x_c)) + t(x_c) · shade = value1×8×8 (one channel)∂y/∂x: 64×64, order [x_c | x_u]0Idiag(eˢ)∂y_u/∂x_cthis layer log|det J| = Σ s−5.674by elimination on dense J−5.674cumulative Σ log|det|−5.674dims active / factored out64 / 0max |x − f⁻¹(f(x))| so far8.9e-16s = α·tanh(m), t = 0.5·m, m = mean of conditioning neighbours · α = 0.50det of a block-triangular J = product of its diagonal = exp(Σ s): no dense determinant neededlog p(x) = log N(z₁) + log N(z₂) + Σₖ log|det Jₖ| · z₁ (32 dims) skips the last 4 couplingstotal log|det| through all 10 couplings = −35.980 at α = 0.50squeeze (a permutation) and factor-out (a split) add 0 to log|det|
step
fig.A toy 1×8×8 image (64 numbers) passes through a two-scale RealNVP in the paper's layout: 3 checkerboard couplings, squeeze to 4×4×4, 3 channel-wise couplings, factor out channels 2–3 (32 dimensions go to z₁), then 4 checkerboard couplings on the remaining 2×4×4. Step through the 14 stages. Blue cells are copied, orange cells are scaled and shifted. On the right is the dense Jacobian of the current coupling, with rows and columns reordered as [x_c | x_u]. The upper-right block is exactly zero, the upper-left is I, and the diagonal of the lower-right block is exp(s). Only 176 of its 4,096 entries are non-zero in coupling 1. Here s = α·tanh(m) and t = 0.5·m are fixed toy functions of the mean m of the conditioning neighbours (no training). At α = 0.5, coupling 1 has log|det| = Σs = −5.674, and Gaussian elimination on the full 64×64 matrix gives the same −5.674. The total over all 10 couplings is −35.980. Set α = 0 to get NICE's additive coupling: every log-determinant is exactly 0, but the map is still non-trivial because t still shifts. Squeeze (a permutation) and factor-out (a split) never contribute. Inverting the whole stack reproduces the input to within 1.1·10⁻¹⁴ at α = 0.5 (3.2·10⁻¹⁴ at α = 1.5): it is exact up to floating point.

Worked example: one coupling on four numbers

Take x=(0.5, −1.0, 1.2, 0.3)x=(0.5,\,-1.0,\,1.2,\,0.3) with mask b=(1,1,0,0)b=(1,1,0,0), so xc=(0.5,−1.0)x_c=(0.5,-1.0) and xu=(1.2,0.3)x_u=(1.2,0.3). Let s(xc)=tanh⁡(Axc)s(x_c)=\tanh(Ax_c) with A=(0.80.4−0.60.2)A=\begin{pmatrix}0.8&0.4\\-0.6&0.2\end{pmatrix} and t(xc)=Bxct(x_c)=Bx_c with B=(100.5−0.5)B=\begin{pmatrix}1&0\\0.5&-0.5\end{pmatrix}.

quantitydim 3dim 4
AxcAx_c0.0000−0.5000
s=tanh⁡(Axc)s=\tanh(Ax_c)0.0000−0.4621
exp⁡(s)\exp(s)1.00000.6300
t=Bxct=Bx_c0.50000.7500
yu=xu⊙es+ty_u=x_u\odot e^{s}+t1.70000.9390
Forward pass (computed in PyTorch, float64)

Determinant, inverse and likelihood

  1. 1y=(0.5, −1.0, 1.7, 0.9390)y=(0.5,\,-1.0,\,1.7,\,0.9390). The first two entries are copied.
  2. 2Full Jacobian (by autograd): rows (1,0,0,0)(1,0,0,0), (0,1,0,0)(0,1,0,0), (1.96, 0.48, 1, 0)(1.96,\,0.48,\,1,\,0), (0.4108, −0.4703, 0, 0.6300)(0.4108,\,-0.4703,\,0,\,0.6300). It is lower triangular, so det⁡J=1⋅1⋅1⋅0.6300=e0+(−0.4621)\det J=1\cdot1\cdot1\cdot0.6300=e^{0+(-0.4621)}. The dense det⁡\det gives the same 0.6300, and the off-diagonal entries never mattered.
  3. 3Inverse: ss and tt are recomputed from yc=xcy_c=x_c, and xu=(yu−t)⊙e−s=((1.7−0.5)⋅1, (0.9390−0.75)/0.6300)=(1.2, 0.3)x_u=(y_u-t)\odot e^{-s}=((1.7-0.5)\cdot1,\ (0.9390-0.75)/0.6300)=(1.2,\,0.3). Exact.
  4. 4As a one-layer flow with z=yz=y: log⁡N(z;0,I)=−12(0.25+1+2.89+0.8817)−2log⁡2π=−6.1866\log\mathcal N(z;0,I)=-\tfrac12(0.25+1+2.89+0.8817)-2\log 2\pi=-6.1866, and log⁡p(x)=−6.1866+(−0.4621)=−6.6487\log p(x)=-6.1866+(-0.4621)=-6.6487 nats.

Results

datasetPixelRNNRealNVP
CIFAR-103.003.49
ImageNet 32×323.864.28
ImageNet 64×643.633.98
LSUN bedroom2.72
LSUN tower2.81
LSUN church outdoor3.08
CelebA3.02
Test negative log-likelihood in bits/dim (lower is better), Dinh et al. 2017, Table 1

RealNVP did not beat autoregressive PixelRNN on likelihood. PixelRNN, however, needs one network evaluation per subpixel to sample a 32×32×3 image (3,072 sequential steps). RealNVP samples in one parallel pass and also provides an exact inverse, so every image has a latent code that can be interpolated or edited. Glow later improved all of these numbers (for example CIFAR-10 3.35 bits/dim) with the same coupling layer.

python
1import torch, torch.nn as nn, math
2
3def checkerboard(c, h, w, parity):
4 m = (torch.arange(h)[:, None] + torch.arange(w)[None, :]) % 2 == parity
5 return m.float().expand(1, c, h, w)
6
7def channel_mask(c, h, w, parity):
8 m = torch.zeros(1, c, h, w); m[:, : c // 2] = 1
9 return m if parity == 0 else 1 - m
10
11class AffineCoupling(nn.Module):
12 """y = b*x + (1-b)*(x*exp(s(b*x)) + t(b*x)); log|det J| = sum of (1-b)*s."""
13 def __init__(self, mask, hidden=64):
14 super().__init__()
15 self.register_buffer("b", mask)
16 c = mask.size(1)
17 self.net = nn.Sequential(nn.Conv2d(c, hidden, 3, padding=1), nn.ReLU(),
18 nn.Conv2d(hidden, hidden, 3, padding=1), nn.ReLU(),
19 nn.Conv2d(hidden, 2 * c, 3, padding=1))
20 nn.init.zeros_(self.net[-1].weight); nn.init.zeros_(self.net[-1].bias) # start at identity
21 self.scale = nn.Parameter(torch.ones(1, c, 1, 1))
22
23 def st(self, xb):
24 s, t = self.net(xb).chunk(2, dim=1)
25 s = self.scale * torch.tanh(s) # bounded log-scale for stability
26 return s * (1 - self.b), t * (1 - self.b)
27
28 def forward(self, x): # data -> latent
29 s, t = self.st(self.b * x)
30 y = self.b * x + (1 - self.b) * (x * torch.exp(s) + t)
31 return y, s.flatten(1).sum(1)
32
33 def inverse(self, y): # latent -> data
34 s, t = self.st(self.b * y) # b*y == b*x
35 return self.b * y + (1 - self.b) * ((y - t) * torch.exp(-s))
36
37def squeeze(x): # (N,C,H,W) -> (N,4C,H/2,W/2)
38 N, C, H, W = x.shape
39 x = x.view(N, C, H // 2, 2, W // 2, 2).permute(0, 1, 3, 5, 2, 4)
40 return x.reshape(N, 4 * C, H // 2, W // 2)
41
42class Scale(nn.Module):
43 """3 checkerboard couplings, squeeze, 3 channel couplings, then factor out half."""
44 def __init__(self, c, h):
45 super().__init__()
46 self.pre = nn.ModuleList(AffineCoupling(checkerboard(c, h, h, p)) for p in (0, 1, 0))
47 self.post = nn.ModuleList(AffineCoupling(channel_mask(4 * c, h // 2, h // 2, p)) for p in (0, 1, 0))
48
49 def forward(self, x):
50 ld = 0
51 for f in self.pre:
52 x, l = f(x); ld = ld + l
53 x = squeeze(x)
54 for f in self.post:
55 x, l = f(x); ld = ld + l
56 x, z = x.chunk(2, dim=1) # z is factored out
57 return x, z, ld
58
59class RealNVP(nn.Module):
60 def __init__(self, c=3, h=32, scales=2):
61 super().__init__()
62 self.scales = nn.ModuleList()
63 for _ in range(scales):
64 self.scales.append(Scale(c, h)); c, h = 2 * c, h // 2
65 self.final = nn.ModuleList(AffineCoupling(checkerboard(c, h, h, p)) for p in (0, 1, 0, 1))
66
67 def forward(self, x):
68 zs, ld = [], 0
69 for sc in self.scales:
70 x, z, l = sc(x); zs.append(z); ld = ld + l
71 for f in self.final:
72 x, l = f(x); ld = ld + l
73 zs.append(x)
74 logpz = sum((-0.5 * z ** 2 - 0.5 * math.log(2 * math.pi)).flatten(1).sum(1) for z in zs)
75 return zs, logpz + ld # log p(x) by change of variables
76
77def bits_per_dim(x_uint8, model, alpha=0.05):
78 """Dequantise, logit-transform, and score in bits per (8-bit) subpixel."""
79 D = x_uint8[0].numel()
80 u = (x_uint8.float() + torch.rand_like(x_uint8.float())) / 256 # in (0,1)
81 p = alpha + (1 - 2 * alpha) * u
82 y = torch.log(p) - torch.log1p(-p) # logit
83 _, logp_y = model(y)
84 # Jacobian of x/256 -> y, plus the 1/256 of the rescaling from integers
85 ld = (math.log(1 - 2 * alpha) - torch.log(p) - torch.log1p(-p)).flatten(1).sum(1) - D * math.log(256)
86 return -(logp_y + ld) / (D * math.log(2))
87
88torch.manual_seed(0)
89model = RealNVP()
90x = torch.randint(0, 256, (4, 3, 32, 32))
91zs, _ = model(torch.randn(4, 3, 32, 32))
92print([tuple(z.shape) for z in zs]) # [(4, 6, 16, 16), (4, 12, 8, 8), (4, 12, 8, 8)]
93print(sum(z[0].numel() for z in zs)) # 3072 = 3*32*32: nothing lost
94b = bits_per_dim(x, model)
95print(b.shape) # torch.Size([4])
96f = model.final[0]; v = torch.randn(2, 12, 8, 8)
97with torch.no_grad():
98 for p in f.net.parameters(): p.normal_(std=0.1) # a non-trivial coupling
99y, _ = f(v)
100print((f.inverse(y) - v).abs().max() < 1e-5) # tensor(True): exact inverse
A two-scale RealNVP for 3×32×32 images: masked affine coupling, squeeze, factor-out, and bits/dim with dequantisation and the logit transform

Use when

  • +Exact log-likelihood and exact inverse; sampling and density evaluation each take one parallel pass
  • +log⁡∣det⁡J∣=∑s\log|\det J|=\sum s costs O(D)O(D), regardless of how deep ss and tt are
  • +Multi-scale factor-out saves compute and gives a coarse-to-fine latent hierarchy
  • +Meaningful latent space: interpolations and resampling of individual scales

Avoid when

  • −Each layer transforms only half the dimensions; many layers are needed for expressivity
  • −Fixed masks constrain which variables can interact in a layer; the order of dimensions is fixed by hand
  • −Worse likelihood than autoregressive models (3.49 vs 3.00 bits/dim on CIFAR-10) and less sharp samples than GANs
  • −Latent has the same dimension as the data: no compression, high memory use; unbounded exp⁡(s)\exp(s) needs care

Check yourself

Why can s and t in a RealNVP coupling layer be arbitrary, non-invertible neural networks?

Check yourself

A coupling layer on a 6-dimensional input copies 3 coordinates and outputs s = (0.2, −0.5, 0.1) for the other 3. What is log|det J|?

Remember

  • ✓Affine coupling: yc=xcy_c=x_c, yu=xu⊙exp⁡(s(xc))+t(xc)y_u=x_u\odot\exp(s(x_c))+t(x_c); inverse xu=(yu−t(yc))⊙exp⁡(−s(yc))x_u=(y_u-t(y_c))\odot\exp(-s(y_c)).
  • ✓The Jacobian is block triangular, so log⁡∣det⁡J∣=∑js(xc)j\log|\det J|=\sum_j s(x_c)_j: O(D)O(D), with ss, tt arbitrary networks.
  • ✓Masks alternate (checkerboard, then channel-wise after a squeeze from s×s×c to s/2×s/2×4c) so every dimension gets transformed.
  • ✓Multi-scale: per scale 3 checkerboard couplings, squeeze, 3 channel couplings, factor out half to the Gaussian prior.
  • ✓CIFAR-10: 3.49 bits/dim (PixelRNN 3.00), but sampling in one parallel pass with an exact inverse.
← back to Deep Learning roadmap