# RealNVP

> RealNVP (Dinh, Sohl-Dickstein and Bengio, 2017) builds an invertible image model from affine coupling layers y = x⊙exp(s) + t whose Jacobian is triangular, so log|det| is just Σs; checkerboard and channel masks, squeezing and a multi-scale factor-out make it work on images.

Source: https://www.mlfoundry.in/learn/realnvp  
Section: Deep Learning › Generative Deep Learning Algorithms  
Level: core · 14 min read · updated 2026-10-12

**RealNVP:** (**real-valued non-volume preserving** transformations) is the normalizing flow of Laurent Dinh, Jascha Sohl-Dickstein and Samy Bengio, "Density estimation using Real NVP" (ICLR 2017). It stacks **affine coupling layers**: a binary mask splits the input into a part $x_c$ that is copied and a part $x_u$ that is scaled and shifted, $y_u=x_u\odot\exp(s(x_c))+t(x_c)$, with $s$ and $t$ arbitrary deep networks. The Jacobian is triangular, so $\log|\det J|=\sum s(x_c)$, and the inverse costs one more pass of $s$ and $t$. Masks alternate between layers, and a multi-scale architecture with **squeeze** and **factor-out** operations makes it practical for images.

A flow needs three things at once: an invertible map, a cheap Jacobian determinant, and enough flexibility to turn a Gaussian into images (see **Normalizing flows**). Dinh, Krueger and Bengio's NICE (2014) got the first two with **additive** coupling, $y_u=x_u+m(x_c)$. Its Jacobian determinant is exactly 1, so every layer preserves volume, and only a final diagonal scaling could change it. RealNVP adds a learned, input-dependent scale $\exp(s(x_c))$. The determinant is then no longer 1, which is the "non-volume preserving" in the name. Combined with convolutional residual networks for $s$ and $t$, batch normalization, and a multi-scale design, it produced sharp samples and exact log-likelihoods on CIFAR-10, downsampled ImageNet, LSUN and CelebA. It was the first flow competitive on natural images, and **Glow** keeps its coupling layer and multi-scale skeleton almost unchanged.

## The affine coupling layer

$$
y_{1:d}=x_{1:d},\qquad y_{d+1:D}=x_{d+1:D}\odot\exp\!\big(s(x_{1:d})\big)+t(x_{1:d})
$$

- $x_{1:d}$: the conditioning half, copied unchanged ($x_c$)
- $x_{d+1:D}$: the transformed half ($x_u$), scaled elementwise and shifted
- $s,\ t:\mathbb R^d\to\mathbb R^{D-d}$: log-scale and translation; any network, never inverted or differentiated through a determinant
- $\odot$: elementwise product

$$
\frac{\partial y}{\partial x^\top}=\begin{pmatrix}I_d & 0\\[2pt] \dfrac{\partial y_{d+1:D}}{\partial x_{1:d}^\top} & \mathrm{diag}\big(\exp[s(x_{1:d})]\big)\end{pmatrix},\qquad \log\Big|\det\frac{\partial y}{\partial x^\top}\Big|=\sum_{j}s(x_{1:d})_j
$$

- $I_d$: the copied half depends only on itself
- $0$: the copied half does not read $x_u$: the matrix is block lower triangular
- $\partial y_u/\partial x_c$: dense and complicated (it contains the Jacobians of $s$ and $t$), but it does not enter the determinant
- $\mathrm{diag}(\exp s)$: the determinant of a triangular matrix is the product of its diagonal, $\exp(\sum_j s_j)$

$$
x_{1:d}=y_{1:d},\qquad x_{d+1:D}=\big(y_{d+1:D}-t(y_{1:d})\big)\odot\exp\!\big(-s(y_{1:d})\big)
$$

- $t(y_{1:d})$: since $y_{1:d}=x_{1:d}$, the inverse can recompute exactly the same $s$ and $t$

This is the whole trick. Inverting the layer never requires inverting $s$ or $t$, and the log-determinant never requires their Jacobians. So $s$ and $t$ can be deep convolutional ResNets with batch normalization, and the layer stays exactly invertible and costs $O(D)$ for $\log|\det J|$ instead of $O(D^3)$. Both directions cost one evaluation of the networks. That makes sampling as fast as density evaluation, unlike the autoregressive flows of **Normalizing flows**.

## Masks: checkerboard and channel-wise

For images the split is written with a binary mask $b$ of the input's shape:

$$
y=b\odot x+(1-b)\odot\Big(x\odot\exp\!\big(s(b\odot x)\big)+t(b\odot x)\Big)
$$

- $b$: 1 on the conditioning entries, 0 on the updated ones
- $\textit{checkerboard}$: $b_{ij}=1$ where $i+j$ is even (spatial alternation); $s,t$ see each pixel's neighbours
- $\textit{channel-wise}$: $b=1$ on the first half of the channels; $s,t$ see the other channels at every position

A single coupling leaves half of the dimensions untouched, so layers alternate the mask ($b\to1-b$). After two layers every dimension has been updated, and after three every dimension has been updated as a function of every other within the receptive field. Composition keeps everything tractable: Jacobians multiply, so log-determinants add.

## Squeeze and the multi-scale architecture

The **squeeze** operation reshapes an $s\times s\times c$ tensor into $\tfrac s2\times\tfrac s2\times 4c$: each $2\times2$ spatial block becomes four channels at one position. It is a permutation of coordinates, so its log-determinant is 0, and it lets channel-wise masks mix information that was spatially adjacent. One **scale** of RealNVP is:

**One scale of the multi-scale architecture (Dinh et al. 2017, Section 3.6)**

1. Three coupling layers with alternating **checkerboard** masks on the $s\times s\times c$ input.
2. **Squeeze** to $\tfrac s2\times\tfrac s2\times 4c$.
3. Three coupling layers with alternating **channel-wise** masks.
4. **Factor out** half of the channels: $z^{(i)}$ goes straight to the latent vector and is modelled by the Gaussian prior; the other half, $h^{(i)}$, is the input of the next scale.
5. Repeat on $h^{(i)}$. The final scale uses four coupling layers with alternating checkerboard masks, and its output joins the factored-out latents: $z=(z^{(1)},\dots,z^{(L)})$ has exactly $D$ dimensions.

$$
\log p_X(x)=\sum_{i}\log\mathcal N\big(z^{(i)};0,I\big)+\sum_{k}\log\Big|\det\frac{\partial f_k}{\partial h_{k-1}^\top}\Big|
$$

- $z^{(i)}$: the latents factored out at scale $i$ plus the final output; together a bijective image of $x$
- $f_k$: every coupling (and batch-norm) layer; squeeze and split contribute 0

Factoring out has three effects. Each scale halves the dimensions that later, deeper layers must process, which saves computation and memory. The objective is distributed across the network: early latents receive a training signal after only a few layers, which the authors compare to deep supervision. And the latents form a hierarchy: early-factored $z^{(1)}$ carries fine local detail, while the last scale carries global structure. In the paper's conceptual-compression figure, keeping the higher-level latents of an image and resampling the lower-level ones preserves its global structure and changes the details.

## Batch normalization and preprocessing

RealNVP uses deep residual networks with batch normalization and weight normalization for $s$ and $t$, and also applies batch normalization to the outputs of coupling layers as a flow layer. As a map, $x\mapsto(x-\tilde\mu)/\sqrt{\tilde\sigma^2+\epsilon}$ is a diagonal rescaling with log-determinant $-\tfrac12\sum_i\log(\tilde\sigma_i^2+\epsilon)$. Pixels are dequantised with uniform noise and passed through a logit transform before the flow, and the Jacobian of that preprocessing is included when likelihoods are reported in bits per dimension. The prior is an isotropic unit Gaussian.

**Setup in the paper**

- **Paper:** Dinh, Sohl-Dickstein, Bengio, "Density estimation using Real NVP", ICLR 2017 (arXiv 1605.08803)
- **Layer:** affine coupling $y_u=x_u\odot e^{s(x_c)}+t(x_c)$; $s$, $t$ are convolutional ResNets
- **Masks:** checkerboard before each squeeze, channel-wise after; masks alternate between consecutive couplings
- **Scale block:** 3 checkerboard couplings → squeeze → 3 channel-wise couplings → factor out half; final scale: 4 checkerboard couplings
- **Normalization:** batch normalization and weight normalization in s and t; batch normalization also used as a flow layer
- **Data:** CIFAR-10, downsampled ImageNet 32×32 and 64×64, LSUN (bedroom, tower, church outdoor), CelebA; dequantised and logit-transformed
- **Prior:** $\mathcal N(0,I)$ over all factored-out latents

> Figure: A toy 1×8×8 image (64 numbers) passes through a two-scale RealNVP in the paper's layout: 3 checkerboard couplings, squeeze to 4×4×4, 3 channel-wise couplings, factor out channels 2–3 (32 dimensions go to z₁), then 4 checkerboard couplings on the remaining 2×4×4. Step through the 14 stages. Blue cells are copied, orange cells are scaled and shifted. On the right is the dense Jacobian of the current coupling, with rows and columns reordered as [x_c | x_u]. The upper-right block is exactly zero, the upper-left is I, and the diagonal of the lower-right block is exp(s). Only 176 of its 4,096 entries are non-zero in coupling 1. Here s = α·tanh(m) and t = 0.5·m are fixed toy functions of the mean m of the conditioning neighbours (no training). At α = 0.5, coupling 1 has log|det| = Σs = −5.674, and Gaussian elimination on the full 64×64 matrix gives the same −5.674. The total over all 10 couplings is −35.980. Set α = 0 to get NICE's additive coupling: every log-determinant is exactly 0, but the map is still non-trivial because t still shifts. Squeeze (a permutation) and factor-out (a split) never contribute. Inverting the whole stack reproduces the input to within 1.1·10⁻¹⁴ at α = 0.5 (3.2·10⁻¹⁴ at α = 1.5): it is exact up to floating point.

## Worked example: one coupling on four numbers

Take $x=(0.5,\,-1.0,\,1.2,\,0.3)$ with mask $b=(1,1,0,0)$, so $x_c=(0.5,-1.0)$ and $x_u=(1.2,0.3)$. Let $s(x_c)=\tanh(Ax_c)$ with $A=\begin{pmatrix}0.8&0.4\\-0.6&0.2\end{pmatrix}$ and $t(x_c)=Bx_c$ with $B=\begin{pmatrix}1&0\\0.5&-0.5\end{pmatrix}$.

Forward pass (computed in PyTorch, float64)

| quantity | dim 3 | dim 4 |
| --- | --- | --- |
| $Ax_c$ | 0.0000 | −0.5000 |
| $s=\tanh(Ax_c)$ | 0.0000 | −0.4621 |
| $\exp(s)$ | 1.0000 | 0.6300 |
| $t=Bx_c$ | 0.5000 | 0.7500 |
| $y_u=x_u\odot e^{s}+t$ | 1.7000 | 0.9390 |

**Determinant, inverse and likelihood**

1. $y=(0.5,\,-1.0,\,1.7,\,0.9390)$. The first two entries are copied.
2. Full Jacobian (by autograd): rows $(1,0,0,0)$, $(0,1,0,0)$, $(1.96,\,0.48,\,1,\,0)$, $(0.4108,\,-0.4703,\,0,\,0.6300)$. It is lower triangular, so $\det J=1\cdot1\cdot1\cdot0.6300=e^{0+(-0.4621)}$. The dense $\det$ gives the same 0.6300, and the off-diagonal entries never mattered.
3. Inverse: $s$ and $t$ are recomputed from $y_c=x_c$, and $x_u=(y_u-t)\odot e^{-s}=((1.7-0.5)\cdot1,\ (0.9390-0.75)/0.6300)=(1.2,\,0.3)$. Exact.
4. As a one-layer flow with $z=y$: $\log\mathcal N(z;0,I)=-\tfrac12(0.25+1+2.89+0.8817)-2\log 2\pi=-6.1866$, and $\log p(x)=-6.1866+(-0.4621)=-6.6487$ nats.

## Results

Test negative log-likelihood in bits/dim (lower is better), Dinh et al. 2017, Table 1

| dataset | PixelRNN | RealNVP |
| --- | --- | --- |
| CIFAR-10 | 3.00 | 3.49 |
| ImageNet 32×32 | 3.86 | 4.28 |
| ImageNet 64×64 | 3.63 | 3.98 |
| LSUN bedroom |  | 2.72 |
| LSUN tower |  | 2.81 |
| LSUN church outdoor |  | 3.08 |
| CelebA |  | 3.02 |

RealNVP did not beat autoregressive PixelRNN on likelihood. PixelRNN, however, needs one network evaluation per subpixel to sample a 32×32×3 image (3,072 sequential steps). RealNVP samples in one parallel pass and also provides an exact inverse, so every image has a latent code that can be interpolated or edited. **Glow** later improved all of these numbers (for example CIFAR-10 3.35 bits/dim) with the same coupling layer.

```python
import torch, torch.nn as nn, math

def checkerboard(c, h, w, parity):
    m = (torch.arange(h)[:, None] + torch.arange(w)[None, :]) % 2 == parity
    return m.float().expand(1, c, h, w)

def channel_mask(c, h, w, parity):
    m = torch.zeros(1, c, h, w); m[:, : c // 2] = 1
    return m if parity == 0 else 1 - m

class AffineCoupling(nn.Module):
    """y = b*x + (1-b)*(x*exp(s(b*x)) + t(b*x)); log|det J| = sum of (1-b)*s."""
    def __init__(self, mask, hidden=64):
        super().__init__()
        self.register_buffer("b", mask)
        c = mask.size(1)
        self.net = nn.Sequential(nn.Conv2d(c, hidden, 3, padding=1), nn.ReLU(),
                                 nn.Conv2d(hidden, hidden, 3, padding=1), nn.ReLU(),
                                 nn.Conv2d(hidden, 2 * c, 3, padding=1))
        nn.init.zeros_(self.net[-1].weight); nn.init.zeros_(self.net[-1].bias)   # start at identity
        self.scale = nn.Parameter(torch.ones(1, c, 1, 1))

    def st(self, xb):
        s, t = self.net(xb).chunk(2, dim=1)
        s = self.scale * torch.tanh(s)                    # bounded log-scale for stability
        return s * (1 - self.b), t * (1 - self.b)

    def forward(self, x):                                  # data -> latent
        s, t = self.st(self.b * x)
        y = self.b * x + (1 - self.b) * (x * torch.exp(s) + t)
        return y, s.flatten(1).sum(1)

    def inverse(self, y):                                  # latent -> data
        s, t = self.st(self.b * y)                         # b*y == b*x
        return self.b * y + (1 - self.b) * ((y - t) * torch.exp(-s))

def squeeze(x):                                            # (N,C,H,W) -> (N,4C,H/2,W/2)
    N, C, H, W = x.shape
    x = x.view(N, C, H // 2, 2, W // 2, 2).permute(0, 1, 3, 5, 2, 4)
    return x.reshape(N, 4 * C, H // 2, W // 2)

class Scale(nn.Module):
    """3 checkerboard couplings, squeeze, 3 channel couplings, then factor out half."""
    def __init__(self, c, h):
        super().__init__()
        self.pre = nn.ModuleList(AffineCoupling(checkerboard(c, h, h, p)) for p in (0, 1, 0))
        self.post = nn.ModuleList(AffineCoupling(channel_mask(4 * c, h // 2, h // 2, p)) for p in (0, 1, 0))

    def forward(self, x):
        ld = 0
        for f in self.pre:
            x, l = f(x); ld = ld + l
        x = squeeze(x)
        for f in self.post:
            x, l = f(x); ld = ld + l
        x, z = x.chunk(2, dim=1)                           # z is factored out
        return x, z, ld

class RealNVP(nn.Module):
    def __init__(self, c=3, h=32, scales=2):
        super().__init__()
        self.scales = nn.ModuleList()
        for _ in range(scales):
            self.scales.append(Scale(c, h)); c, h = 2 * c, h // 2
        self.final = nn.ModuleList(AffineCoupling(checkerboard(c, h, h, p)) for p in (0, 1, 0, 1))

    def forward(self, x):
        zs, ld = [], 0
        for sc in self.scales:
            x, z, l = sc(x); zs.append(z); ld = ld + l
        for f in self.final:
            x, l = f(x); ld = ld + l
        zs.append(x)
        logpz = sum((-0.5 * z ** 2 - 0.5 * math.log(2 * math.pi)).flatten(1).sum(1) for z in zs)
        return zs, logpz + ld                              # log p(x) by change of variables

def bits_per_dim(x_uint8, model, alpha=0.05):
    """Dequantise, logit-transform, and score in bits per (8-bit) subpixel."""
    D = x_uint8[0].numel()
    u = (x_uint8.float() + torch.rand_like(x_uint8.float())) / 256       # in (0,1)
    p = alpha + (1 - 2 * alpha) * u
    y = torch.log(p) - torch.log1p(-p)                                  # logit
    _, logp_y = model(y)
    # Jacobian of x/256 -> y, plus the 1/256 of the rescaling from integers
    ld = (math.log(1 - 2 * alpha) - torch.log(p) - torch.log1p(-p)).flatten(1).sum(1) - D * math.log(256)
    return -(logp_y + ld) / (D * math.log(2))

torch.manual_seed(0)
model = RealNVP()
x = torch.randint(0, 256, (4, 3, 32, 32))
zs, _ = model(torch.randn(4, 3, 32, 32))
print([tuple(z.shape) for z in zs])          # [(4, 6, 16, 16), (4, 12, 8, 8), (4, 12, 8, 8)]
print(sum(z[0].numel() for z in zs))          # 3072 = 3*32*32: nothing lost
b = bits_per_dim(x, model)
print(b.shape)                                # torch.Size([4])
f = model.final[0]; v = torch.randn(2, 12, 8, 8)
with torch.no_grad():
    for p in f.net.parameters(): p.normal_(std=0.1)  # a non-trivial coupling
y, _ = f(v)
print((f.inverse(y) - v).abs().max() < 1e-5)  # tensor(True): exact inverse
```

A two-scale RealNVP for 3×32×32 images: masked affine coupling, squeeze, factor-out, and bits/dim with dequantisation and the logit transform

> **Unbounded scales blow up:** $\exp(s)$ is unbounded. A network that outputs $s=20$ on some input multiplies by $5\cdot10^8$, and the inverse divides by it. Training then diverges or produces NaNs. Common fixes: zero-initialise the last layer of $s$ and $t$ so every coupling starts as the identity, and bound the log-scale, for example $s=\lambda\tanh(\cdot)$ with a learned $\lambda$ (as in the code) or Glow's sigmoid parameterisation. Watch $\sum s$: a large negative value inflates the likelihood by squeezing data into a small volume.

> **Do not forget the preprocessing Jacobian:** If you report bits/dim, the dequantisation scale $1/256$ and the logit transform are part of the model. Their log-determinants must be added, as in `bits_per_dim` above. Omitting them yields numbers that are not comparable with the literature and can even be negative. A quick sanity check: a model that ignores the data and outputs uniform pixels scores exactly 8 bits/dim.

> **Why the triangle matters:** For a general $D\times D$ Jacobian, $\log|\det|$ costs $O(D^3)$: about $\tfrac23D^3\approx2\cdot10^{10}$ flops of Gaussian elimination for one 3,072-dimensional CIFAR image, per layer. The coupling layer arranges, by construction, that the copied half never reads the transformed half. The upper-right block is zero, and the determinant is the product of $D-d$ diagonal entries. All the expressivity sits in the off-diagonal block, which the determinant ignores. The figure shows this structure directly.

> **Batch norm makes the likelihood batch-dependent:** With batch statistics, the density of one image depends on the other images in its batch. The paper's flow-layer batch norm uses moving averages of the batch statistics. At evaluation you need fixed statistics for $\log p(x)$ to be a function of $x$ alone. **Glow** replaced this layer with actnorm, which has data-initialised but then fixed per-channel parameters.

**Strengths**

- Exact log-likelihood and exact inverse; sampling and density evaluation each take one parallel pass
- $\log|\det J|=\sum s$ costs $O(D)$, regardless of how deep $s$ and $t$ are
- Multi-scale factor-out saves compute and gives a coarse-to-fine latent hierarchy
- Meaningful latent space: interpolations and resampling of individual scales

**Limitations**

- Each layer transforms only half the dimensions; many layers are needed for expressivity
- Fixed masks constrain which variables can interact in a layer; the order of dimensions is fixed by hand
- Worse likelihood than autoregressive models (3.49 vs 3.00 bits/dim on CIFAR-10) and less sharp samples than GANs
- Latent has the same dimension as the data: no compression, high memory use; unbounded $\exp(s)$ needs care

**Question:** Why can s and t in a RealNVP coupling layer be arbitrary, non-invertible neural networks?

- A. Because they are trained with a reconstruction loss
- B. Because the inverse recomputes s and t from the copied half, which is identical in x and y, and the determinant depends only on the values of s
- C. Because the Jacobian of s and t is always diagonal
- D. Because the coupling layer is approximately invertible, which is enough

**Answer:** B. Because the inverse recomputes s and t from the copied half, which is identical in x and y, and the determinant depends only on the values of s — $y_c=x_c$, so the inverse evaluates the same $s(y_c)=s(x_c)$ and $t(y_c)$ and solves the elementwise affine map for $x_u$. The Jacobian is block triangular; $\partial s/\partial x_c$ sits in the off-diagonal block and never enters $\det J=\exp(\sum s)$. The inverse is exact, not approximate.

**Question:** A coupling layer on a 6-dimensional input copies 3 coordinates and outputs s = (0.2, −0.5, 0.1) for the other 3. What is log|det J|?

- A. −0.2
- B. 0.2
- C. exp(−0.2) ≈ 0.819
- D. It requires the full 6×6 Jacobian

**Answer:** A. −0.2 — $\log|\det J|=\sum_j s_j=0.2-0.5+0.1=-0.2$; the determinant itself is $e^{-0.2}\approx0.819$. The layer contracts volume locally, so it raises the density of $x$ relative to $z$ by that factor in $\log p(x)=\log p(z)+\log|\det J|$.

**Key points**

- Affine coupling: $y_c=x_c$, $y_u=x_u\odot\exp(s(x_c))+t(x_c)$; inverse $x_u=(y_u-t(y_c))\odot\exp(-s(y_c))$.
- The Jacobian is block triangular, so $\log|\det J|=\sum_j s(x_c)_j$: $O(D)$, with $s$, $t$ arbitrary networks.
- Masks alternate (checkerboard, then channel-wise after a squeeze from s×s×c to s/2×s/2×4c) so every dimension gets transformed.
- Multi-scale: per scale 3 checkerboard couplings, squeeze, 3 channel couplings, factor out half to the Gaussian prior.
- CIFAR-10: 3.49 bits/dim (PixelRNN 3.00), but sampling in one parallel pass with an exact inverse.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
