Definition
RealNVP (real-valued non-volume preserving transformations) is the normalizing flow of Laurent Dinh, Jascha Sohl-Dickstein and Samy Bengio, "Density estimation using Real NVP" (ICLR 2017). It stacks affine coupling layers: a binary mask splits the input into a part that is copied and a part that is scaled and shifted, , with and arbitrary deep networks. The Jacobian is triangular, so , and the inverse costs one more pass of and . Masks alternate between layers, and a multi-scale architecture with squeeze and factor-out operations makes it practical for images.
A flow needs three things at once: an invertible map, a cheap Jacobian determinant, and enough flexibility to turn a Gaussian into images (see Normalizing flows). Dinh, Krueger and Bengio's NICE (2014) got the first two with additive coupling, . Its Jacobian determinant is exactly 1, so every layer preserves volume, and only a final diagonal scaling could change it. RealNVP adds a learned, input-dependent scale . The determinant is then no longer 1, which is the "non-volume preserving" in the name. Combined with convolutional residual networks for and , batch normalization, and a multi-scale design, it produced sharp samples and exact log-likelihoods on CIFAR-10, downsampled ImageNet, LSUN and CelebA. It was the first flow competitive on natural images, and Glow keeps its coupling layer and multi-scale skeleton almost unchanged.
The affine coupling layer
- the conditioning half, copied unchanged ()
- the transformed half (), scaled elementwise and shifted
- log-scale and translation; any network, never inverted or differentiated through a determinant
- elementwise product
- the copied half depends only on itself
- the copied half does not read : the matrix is block lower triangular
- dense and complicated (it contains the Jacobians of and ), but it does not enter the determinant
- the determinant of a triangular matrix is the product of its diagonal,
- since , the inverse can recompute exactly the same and
This is the whole trick. Inverting the layer never requires inverting or , and the log-determinant never requires their Jacobians. So and can be deep convolutional ResNets with batch normalization, and the layer stays exactly invertible and costs for instead of . Both directions cost one evaluation of the networks. That makes sampling as fast as density evaluation, unlike the autoregressive flows of Normalizing flows.
Masks: checkerboard and channel-wise
For images the split is written with a binary mask of the input's shape:
- 1 on the conditioning entries, 0 on the updated ones
- where is even (spatial alternation); see each pixel's neighbours
- on the first half of the channels; see the other channels at every position
A single coupling leaves half of the dimensions untouched, so layers alternate the mask (). After two layers every dimension has been updated, and after three every dimension has been updated as a function of every other within the receptive field. Composition keeps everything tractable: Jacobians multiply, so log-determinants add.
Squeeze and the multi-scale architecture
The squeeze operation reshapes an tensor into : each spatial block becomes four channels at one position. It is a permutation of coordinates, so its log-determinant is 0, and it lets channel-wise masks mix information that was spatially adjacent. One scale of RealNVP is:
One scale of the multi-scale architecture (Dinh et al. 2017, Section 3.6)
- 1Three coupling layers with alternating checkerboard masks on the input.
- 2Squeeze to .
- 3Three coupling layers with alternating channel-wise masks.
- 4Factor out half of the channels: goes straight to the latent vector and is modelled by the Gaussian prior; the other half, , is the input of the next scale.
- 5Repeat on . The final scale uses four coupling layers with alternating checkerboard masks, and its output joins the factored-out latents: has exactly dimensions.
- the latents factored out at scale plus the final output; together a bijective image of
- every coupling (and batch-norm) layer; squeeze and split contribute 0
Factoring out has three effects. Each scale halves the dimensions that later, deeper layers must process, which saves computation and memory. The objective is distributed across the network: early latents receive a training signal after only a few layers, which the authors compare to deep supervision. And the latents form a hierarchy: early-factored carries fine local detail, while the last scale carries global structure. In the paper's conceptual-compression figure, keeping the higher-level latents of an image and resampling the lower-level ones preserves its global structure and changes the details.
Batch normalization and preprocessing
RealNVP uses deep residual networks with batch normalization and weight normalization for and , and also applies batch normalization to the outputs of coupling layers as a flow layer. As a map, is a diagonal rescaling with log-determinant . Pixels are dequantised with uniform noise and passed through a logit transform before the flow, and the Jacobian of that preprocessing is included when likelihoods are reported in bits per dimension. The prior is an isotropic unit Gaussian.
Setup in the paper
- Paper
- Dinh, Sohl-Dickstein, Bengio, "Density estimation using Real NVP", ICLR 2017 (arXiv 1605.08803)
- Layer
- affine coupling ; , are convolutional ResNets
- Masks
- checkerboard before each squeeze, channel-wise after; masks alternate between consecutive couplings
- Scale block
- 3 checkerboard couplings → squeeze → 3 channel-wise couplings → factor out half; final scale: 4 checkerboard couplings
- Normalization
- batch normalization and weight normalization in s and t; batch normalization also used as a flow layer
- Data
- CIFAR-10, downsampled ImageNet 32×32 and 64×64, LSUN (bedroom, tower, church outdoor), CelebA; dequantised and logit-transformed
- Prior
- over all factored-out latents
Worked example: one coupling on four numbers
Take with mask , so and . Let with and with .
| quantity | dim 3 | dim 4 |
|---|---|---|
| 0.0000 | −0.5000 | |
| 0.0000 | −0.4621 | |
| 1.0000 | 0.6300 | |
| 0.5000 | 0.7500 | |
| 1.7000 | 0.9390 |
Determinant, inverse and likelihood
- 1. The first two entries are copied.
- 2Full Jacobian (by autograd): rows , , , . It is lower triangular, so . The dense gives the same 0.6300, and the off-diagonal entries never mattered.
- 3Inverse: and are recomputed from , and . Exact.
- 4As a one-layer flow with : , and nats.
Results
| dataset | PixelRNN | RealNVP |
|---|---|---|
| CIFAR-10 | 3.00 | 3.49 |
| ImageNet 32×32 | 3.86 | 4.28 |
| ImageNet 64×64 | 3.63 | 3.98 |
| LSUN bedroom | 2.72 | |
| LSUN tower | 2.81 | |
| LSUN church outdoor | 3.08 | |
| CelebA | 3.02 |
RealNVP did not beat autoregressive PixelRNN on likelihood. PixelRNN, however, needs one network evaluation per subpixel to sample a 32×32×3 image (3,072 sequential steps). RealNVP samples in one parallel pass and also provides an exact inverse, so every image has a latent code that can be interpolated or edited. Glow later improved all of these numbers (for example CIFAR-10 3.35 bits/dim) with the same coupling layer.
1import torch, torch.nn as nn, math23def checkerboard(c, h, w, parity):4 m = (torch.arange(h)[:, None] + torch.arange(w)[None, :]) % 2 == parity5 return m.float().expand(1, c, h, w)67def channel_mask(c, h, w, parity):8 m = torch.zeros(1, c, h, w); m[:, : c // 2] = 19 return m if parity == 0 else 1 - m1011class AffineCoupling(nn.Module):12 """y = b*x + (1-b)*(x*exp(s(b*x)) + t(b*x)); log|det J| = sum of (1-b)*s."""13 def __init__(self, mask, hidden=64):14 super().__init__()15 self.register_buffer("b", mask)16 c = mask.size(1)17 self.net = nn.Sequential(nn.Conv2d(c, hidden, 3, padding=1), nn.ReLU(),18 nn.Conv2d(hidden, hidden, 3, padding=1), nn.ReLU(),19 nn.Conv2d(hidden, 2 * c, 3, padding=1))20 nn.init.zeros_(self.net[-1].weight); nn.init.zeros_(self.net[-1].bias) # start at identity21 self.scale = nn.Parameter(torch.ones(1, c, 1, 1))2223 def st(self, xb):24 s, t = self.net(xb).chunk(2, dim=1)25 s = self.scale * torch.tanh(s) # bounded log-scale for stability26 return s * (1 - self.b), t * (1 - self.b)2728 def forward(self, x): # data -> latent29 s, t = self.st(self.b * x)30 y = self.b * x + (1 - self.b) * (x * torch.exp(s) + t)31 return y, s.flatten(1).sum(1)3233 def inverse(self, y): # latent -> data34 s, t = self.st(self.b * y) # b*y == b*x35 return self.b * y + (1 - self.b) * ((y - t) * torch.exp(-s))3637def squeeze(x): # (N,C,H,W) -> (N,4C,H/2,W/2)38 N, C, H, W = x.shape39 x = x.view(N, C, H // 2, 2, W // 2, 2).permute(0, 1, 3, 5, 2, 4)40 return x.reshape(N, 4 * C, H // 2, W // 2)4142class Scale(nn.Module):43 """3 checkerboard couplings, squeeze, 3 channel couplings, then factor out half."""44 def __init__(self, c, h):45 super().__init__()46 self.pre = nn.ModuleList(AffineCoupling(checkerboard(c, h, h, p)) for p in (0, 1, 0))47 self.post = nn.ModuleList(AffineCoupling(channel_mask(4 * c, h // 2, h // 2, p)) for p in (0, 1, 0))4849 def forward(self, x):50 ld = 051 for f in self.pre:52 x, l = f(x); ld = ld + l53 x = squeeze(x)54 for f in self.post:55 x, l = f(x); ld = ld + l56 x, z = x.chunk(2, dim=1) # z is factored out57 return x, z, ld5859class RealNVP(nn.Module):60 def __init__(self, c=3, h=32, scales=2):61 super().__init__()62 self.scales = nn.ModuleList()63 for _ in range(scales):64 self.scales.append(Scale(c, h)); c, h = 2 * c, h // 265 self.final = nn.ModuleList(AffineCoupling(checkerboard(c, h, h, p)) for p in (0, 1, 0, 1))6667 def forward(self, x):68 zs, ld = [], 069 for sc in self.scales:70 x, z, l = sc(x); zs.append(z); ld = ld + l71 for f in self.final:72 x, l = f(x); ld = ld + l73 zs.append(x)74 logpz = sum((-0.5 * z ** 2 - 0.5 * math.log(2 * math.pi)).flatten(1).sum(1) for z in zs)75 return zs, logpz + ld # log p(x) by change of variables7677def bits_per_dim(x_uint8, model, alpha=0.05):78 """Dequantise, logit-transform, and score in bits per (8-bit) subpixel."""79 D = x_uint8[0].numel()80 u = (x_uint8.float() + torch.rand_like(x_uint8.float())) / 256 # in (0,1)81 p = alpha + (1 - 2 * alpha) * u82 y = torch.log(p) - torch.log1p(-p) # logit83 _, logp_y = model(y)84 # Jacobian of x/256 -> y, plus the 1/256 of the rescaling from integers85 ld = (math.log(1 - 2 * alpha) - torch.log(p) - torch.log1p(-p)).flatten(1).sum(1) - D * math.log(256)86 return -(logp_y + ld) / (D * math.log(2))8788torch.manual_seed(0)89model = RealNVP()90x = torch.randint(0, 256, (4, 3, 32, 32))91zs, _ = model(torch.randn(4, 3, 32, 32))92print([tuple(z.shape) for z in zs]) # [(4, 6, 16, 16), (4, 12, 8, 8), (4, 12, 8, 8)]93print(sum(z[0].numel() for z in zs)) # 3072 = 3*32*32: nothing lost94b = bits_per_dim(x, model)95print(b.shape) # torch.Size([4])96f = model.final[0]; v = torch.randn(2, 12, 8, 8)97with torch.no_grad():98 for p in f.net.parameters(): p.normal_(std=0.1) # a non-trivial coupling99y, _ = f(v)100print((f.inverse(y) - v).abs().max() < 1e-5) # tensor(True): exact inverseUse when
- +Exact log-likelihood and exact inverse; sampling and density evaluation each take one parallel pass
- + costs , regardless of how deep and are
- +Multi-scale factor-out saves compute and gives a coarse-to-fine latent hierarchy
- +Meaningful latent space: interpolations and resampling of individual scales
Avoid when
- −Each layer transforms only half the dimensions; many layers are needed for expressivity
- −Fixed masks constrain which variables can interact in a layer; the order of dimensions is fixed by hand
- −Worse likelihood than autoregressive models (3.49 vs 3.00 bits/dim on CIFAR-10) and less sharp samples than GANs
- −Latent has the same dimension as the data: no compression, high memory use; unbounded needs care
Check yourself
Why can s and t in a RealNVP coupling layer be arbitrary, non-invertible neural networks?
Check yourself
A coupling layer on a 6-dimensional input copies 3 coordinates and outputs s = (0.2, −0.5, 0.1) for the other 3. What is log|det J|?
Remember
- ✓Affine coupling: , ; inverse .
- ✓The Jacobian is block triangular, so : , with , arbitrary networks.
- ✓Masks alternate (checkerboard, then channel-wise after a squeeze from s×s×c to s/2×s/2×4c) so every dimension gets transformed.
- ✓Multi-scale: per scale 3 checkerboard couplings, squeeze, 3 channel couplings, factor out half to the Gaussian prior.
- ✓CIFAR-10: 3.49 bits/dim (PixelRNN 3.00), but sampling in one parallel pass with an exact inverse.