# Glow

> Glow (Kingma and Dhariwal, 2018) is a RealNVP-style flow whose step is actnorm, then a learned invertible 1×1 convolution, then affine coupling; an LU-parameterised 1×1 convolution has log-determinant h·w·Σlog|sᵢ| at O(c) cost, and it replaces fixed channel permutations.

Source: https://www.mlfoundry.in/learn/glow  
Section: Deep Learning › Generative Deep Learning Algorithms  
Level: advanced · 14 min read · updated 2026-10-12

**Glow:** The normalizing flow of Diederik P. Kingma and Prafulla Dhariwal, "Glow: Generative Flow with Invertible 1×1 Convolutions" (NeurIPS 2018). One **step of flow** has three invertible layers: **actnorm**, a per-channel affine map with data-initialised parameters; an **invertible 1×1 convolution**, a learned $c\times c$ matrix applied at every pixel, which generalises the fixed channel permutation between couplings; and an **affine coupling layer**. Steps are stacked $K$ deep inside each of $L$ levels of the multi-scale architecture of **RealNVP**. With the 1×1 weight parameterised by its LU decomposition, every log-determinant costs $O(c)$ per layer. Glow improved bits/dim on standard benchmarks and generated $256\times256$ faces.

**RealNVP** showed that affine couplings with a triangular Jacobian could model natural images (see **Normalizing flows** for the general framework). Two parts of its design were fixed by hand. Between couplings the channels were reversed or swapped so that the copied half would be updated next. Batch normalization was used as a flow layer, which makes the density depend on the minibatch and degrades with the tiny per-GPU batches that large images force. Glow replaced the permutation with a learned linear map, $W$, that can mix channels arbitrarily, and replaced batch norm with actnorm. On CIFAR-10 the authors compared reversing, random shuffling and the learned 1×1 convolution, each with additive and with affine coupling. With both coupling types, the 1×1 convolution reached a lower negative log-likelihood and converged faster. The paper's demonstrations were faces at $256\times256$ from CelebA-HQ, smooth latent interpolations, and attribute manipulation by moving along directions in $z$-space. All of these came from a model with an exact likelihood and one-pass sampling.

## One step of flow

The three layers of a Glow step for an h×w×c tensor (Kingma & Dhariwal 2018, Table 1)

| layer | forward | inverse | log-determinant |
| --- | --- | --- | --- |
| actnorm | $y_{i,j}=s\odot x_{i,j}+b$ | $x_{i,j}=(y_{i,j}-b)/s$ | $h\cdot w\cdot\sum_c\log\|s_c\|$ |
| invertible 1×1 conv | $y_{i,j}=W x_{i,j}$ | $x_{i,j}=W^{-1}y_{i,j}$ | $h\cdot w\cdot\log\|\det W\|$ |
| affine coupling | $x_a,x_b=\text{split}(x)$; $(\log s,t)=\text{NN}(x_b)$; $y_a=s\odot x_a+t$; $y_b=x_b$ | $x_a=(y_a-t)/s$; $x_b=y_b$ | $\sum\log\|s\|$ |

Here $x_{i,j}\in\mathbb R^c$ is the channel vector at pixel $(i,j)$. Actnorm and the 1×1 convolution use the same parameters at every pixel, so their log-determinants are a per-pixel constant times $h\cdot w$, independent of the input. Only the coupling has a data-dependent determinant.

## Actnorm

Actnorm is a per-channel scale $s$ and bias $b$, like batch norm with the statistics frozen into parameters. Glow uses **data-dependent initialisation**. On the first minibatch, $b$ and $s$ are set so that the post-actnorm activations of every channel have zero mean and unit variance. After that they are ordinary trainable parameters, and the density of an image no longer depends on what else is in its batch. That matters for large images, where memory limits the minibatch to one image per GPU. The layer is also an inexpensive normaliser for the very deep stacks Glow uses.

## The invertible 1×1 convolution

$$
\log\Big|\det\frac{\partial\,\mathrm{conv2D}(h;W)}{\partial h}\Big|=h\cdot w\cdot\log|\det W|,\qquad W=P\,L\,\big(U+\operatorname{diag}(s)\big)
$$

- $W\in\mathbb R^{c\times c}$: a 1×1 convolution is a matrix multiply of every pixel's channel vector; initialised as a random rotation ($\log|\det W|=0$)
- $P$: a permutation matrix, fixed after initialisation
- $L$: lower triangular with ones on the diagonal (learned)
- $U$: upper triangular with zeros on the diagonal (learned)
- $s$: a learned vector; $\log|\det W|=\sum_i\log|s_i|$

A permutation is a special case of $W$, so the learned convolution can only do better than the reverse or shuffle it replaces. Computing $\det W$ directly costs $O(c^3)$. The paper notes this is often comparable to the convolution itself, which costs $O(h\cdot w\cdot c^2)$. In the LU form, $\det P=\pm1$, $\det L=1$ and $\det(U+\operatorname{diag}s)=\prod_i s_i$, so the log-determinant is a sum of $c$ numbers. The inverse $W^{-1}$ is needed only for sampling, and it can be computed once per sampling run. To initialise, sample a random rotation, take its LU decomposition to get $P$, $L$, $U$ and $s$, and then optimise $L$, $U$ and $s$, keeping the signs of $s$ fixed.

## Affine coupling, Glow style

The coupling layer is RealNVP's with three details. The split is along channels only, since checkerboard masks are no longer needed once $W$ mixes channels. The network $\text{NN}$ is a shallow convolutional stack: a $3\times3$ convolution and a $1\times1$ convolution with 512 channels and ReLU, then a final $3\times3$ convolution. That last convolution is **initialised to zero**, so every coupling starts as the identity, which the authors found helps train very deep flows. The released code also bounds the scale as $s=\text{sigmoid}(\tilde h+2)$ instead of $\exp$, which keeps $s\in(0,1)$ with $s\approx0.88$ at initialisation. With $s\equiv1$ the layer becomes NICE's **additive coupling**, with determinant 1.

## Multi-scale architecture

**One level of Glow, repeated L times (Kingma & Dhariwal 2018, Figure 2)**

1. **Squeeze**: $h\times w\times c\to\tfrac h2\times\tfrac w2\times4c$, a permutation with log-determinant 0.
2. **K steps of flow**: actnorm $\to$ 1×1 conv $\to$ affine coupling, $K$ times.
3. **Split**: factor out half of the channels as latent $z_i$; the other half goes to the next level. The last level keeps everything.
4. The log-likelihood is $\sum_i\log p(z_i)$ plus the sum of all $3KL$ per-layer log-determinants. As in RealNVP, the factored-out latents together have exactly as many dimensions as the image.

**Setup in the paper**

- **Paper:** Kingma, Dhariwal, "Glow: Generative Flow with Invertible 1×1 Convolutions", NeurIPS 2018 (arXiv 1807.03039)
- **Step of flow:** actnorm → invertible 1×1 convolution → affine coupling
- **Coupling network:** 3×3 conv → 1×1 conv (512 channels, ReLU) → zero-initialised 3×3 conv
- **Architecture:** multi-scale: L levels of squeeze → K steps → split
- **Data:** CIFAR-10, ImageNet 32×32 and 64×64, LSUN, CelebA-HQ 256×256 (30,000 images, 5-bit)
- **Ablation:** reverse vs. shuffle vs. 1×1 conv, each with additive and affine coupling, on CIFAR-10: 1×1 conv gives the lowest NLL
- **Sampling:** reduced temperature, e.g. $T=0.7$, for the face samples

## Temperature sampling

$$
z\sim\mathcal N(0,T^2I),\quad x=g(z)\qquad\Longrightarrow\qquad p_T(x)\propto p_X(x)^{1/T^2}\ \ \text{if}\ \log|\det\partial g/\partial z|\ \text{is constant}
$$

- $T<1$: samples concentrate on the high-density regions; tails, which are usually artefacts, disappear
- $\text{constant }\log|\det|$: true for additive coupling with actnorm and 1×1 convolutions; approximately true otherwise

Maximum likelihood is mass-covering, so a flow puts some density on implausible images, and samples at $T=1$ show it. Shrinking the prior's standard deviation trades diversity for fidelity. Glow's face samples are drawn at $T=0.7$, and the paper shows quality and diversity changing as $T$ goes from 0 to 1. The same latent space supports interpolation between two encoded faces and semantic edits. Average $z$ over images with an attribute, such as smiling, and over images without it, and add a multiple of the difference to the encoding of a new face.

> Figure: A hand-set (untrained) Glow with c = 2 channels and K = 3 steps. Each point of the plane is the channel vector of one pixel. Step through the 9 sublayers in the sampling direction: for k = 3, 2, 1 it applies coupling⁻¹, then 1×1 conv⁻¹, then actnorm⁻¹. Dots are coloured by |z|/T. The right panel is the exact density p(x) = N(f(x)) · |det ∂f/∂x| from one forward pass, and it integrates to 1.000 over the box. The readouts give each sublayer's log-determinant in the density direction. Actnorm is Σ log s: 0.050, −0.100 and 0.900 for steps 1, 2 and 3. The 1×1 convolution is Σ log|sᵢ| from its LU factors: 0.095, −0.003 and 0. For step 2, W = P·L·(U + diag s) with s = (1.05, −0.95), and det W = +0.9975 because P swaps the channels. The couplings have E[log σ] = −0.082, −0.128 and −0.018 at T = 1. The total is E[log|det ∂z/∂x|] = 0.714, and f(f⁻¹(z)) reproduces z to within 2.4·10⁻¹⁵. Lower the temperature: the mean log p(x) of the samples rises from −2.118 at T = 1 to −1.610 at T = 0.7 and −1.371 at T = 0.5, as the samples collapse onto the ridge of the crescent. The share of samples beyond 2T in z-space stays at 92/700, because every z is just rescaled by T.

## Worked example: actnorm and an LU 1×1 convolution on one pixel

Take $c=3$ channels. On the initialisation batch the channel means are $(3.0,\,-1.0,\,0.5)$ and the standard deviations are $(2.0,\,0.25,\,1.0)$. Let the 1×1 weight have $P=I$, $L=\begin{pmatrix}1&0&0\\0.5&1&0\\-0.2&0.3&1\end{pmatrix}$, strictly upper $U$ entries $(u_{12},u_{13},u_{23})=(0.4,-0.1,0.6)$, and $s=(1.2,\,-0.8,\,0.5)$.

One pixel x = (5, −0.5, 1.5) through actnorm and the 1×1 convolution (computed in Python)

| quantity | value |
| --- | --- |
| actnorm $b=-\mu$, $s_\text{an}=1/\sigma$ | b = (−3, 1, −0.5), s = (0.5, 4, 1) |
| $a=s_\text{an}\odot(x+b)$ | (1.0, 2.0, 1.0) |
| actnorm $\sum\log s_\text{an}$ | −0.6931 + 1.3863 + 0 = 0.6931 |
| $W=PL(U+\operatorname{diag}s)$ | [[1.2, 0.4, −0.1], [0.6, −0.6, 0.55], [−0.24, −0.32, 0.7]] |
| $y=Wa$ | (1.9, −0.05, −0.18) |
| $\sum\log\|s_i\|$ | 0.1823 − 0.2231 − 0.6931 = −0.7340 |
| $\log\|\det W\|$ by cofactor expansion | log 0.48 = −0.7340 (det W = −0.48) |

**From one pixel to a feature map**

1. Actnorm here uses the per-pixel parameterisation $s\odot(x+b)$ of the released code. The paper's form $s\odot x+b'$ is the same map with $b'=s\odot b$.
2. The LU route needs 3 logarithms. The dense determinant needs the full matrix, and the two agree to all printed digits.
3. On a $16\times16$ map these per-pixel constants are multiplied by $h\cdot w=256$: actnorm contributes $256\times0.6931=177.4$ nats and the 1×1 convolution $256\times(-0.7340)=-187.9$. The step's two linear layers together give $-10.45$ nats, whatever the image.
4. Channel counts grow with depth. In a 6-level model of $256\times256\times3$ images, the last level has $c=384$ channels on a $4\times4$ map. There a dense determinant ($\approx\tfrac23c^3\approx3.8\cdot10^7$ flops) costs more than the convolution itself ($h\,w\,c^2\approx2.4\cdot10^6$ multiply-adds). The LU form needs 384 logarithms.

```python
import torch, torch.nn as nn, math

class ActNorm(nn.Module):
    """y = s * (x + b) per channel; s, b initialised from the first batch, then trained."""
    def __init__(self, c):
        super().__init__()
        self.b = nn.Parameter(torch.zeros(1, c, 1, 1))
        self.logs = nn.Parameter(torch.zeros(1, c, 1, 1))
        self.register_buffer("ready", torch.tensor(False))

    def forward(self, x):
        if not self.ready:                              # data-dependent initialisation
            with torch.no_grad():
                self.b.copy_(-x.mean((0, 2, 3), keepdim=True))
                self.logs.copy_(-x.std((0, 2, 3), keepdim=True).add(1e-6).log())
                self.ready.fill_(True)
        h, w = x.shape[2:]
        return (x + self.b) * self.logs.exp(), h * w * self.logs.sum()

    def inverse(self, y):
        return y * (-self.logs).exp() - self.b

class InvConv1x1(nn.Module):
    """W = P L (U + diag(sign * exp(log_s))); log|det| = h*w*sum(log_s): O(c), not O(c^3)."""
    def __init__(self, c):
        super().__init__()
        q, _ = torch.linalg.qr(torch.randn(c, c))       # random rotation
        P, L, U = torch.linalg.lu(q)
        s = torch.diagonal(U)
        self.register_buffer("P", P)
        self.register_buffer("sign", torch.sign(s))
        self.register_buffer("lmask", torch.tril(torch.ones(c, c), -1))
        self.L = nn.Parameter(L)
        self.U = nn.Parameter(torch.triu(U, 1))
        self.log_s = nn.Parameter(s.abs().log())

    def weight(self):
        c = self.log_s.numel()
        L = self.L * self.lmask + torch.eye(c)
        U = self.U * self.lmask.T + torch.diag(self.sign * self.log_s.exp())
        return self.P @ L @ U

    def forward(self, x):
        h, w = x.shape[2:]
        return torch.einsum("ij,njhw->nihw", self.weight(), x), h * w * self.log_s.sum()

    def inverse(self, y):
        return torch.einsum("ij,njhw->nihw", torch.linalg.inv(self.weight()), y)

class Coupling(nn.Module):
    """Affine coupling; NN = conv3x3 -> conv1x1 -> zero-initialised conv3x3."""
    def __init__(self, c, hidden=512):
        super().__init__()
        self.nn = nn.Sequential(nn.Conv2d(c // 2, hidden, 3, padding=1), nn.ReLU(),
                                nn.Conv2d(hidden, hidden, 1), nn.ReLU(),
                                nn.Conv2d(hidden, c, 3, padding=1))
        nn.init.zeros_(self.nn[-1].weight); nn.init.zeros_(self.nn[-1].bias)

    def st(self, xa):
        h = self.nn(xa)
        return torch.sigmoid(h[:, 1::2] + 2), h[:, 0::2]   # scale in (0, 1), shift

    def forward(self, x):
        xa, xb = x.chunk(2, 1)
        s, t = self.st(xa)
        return torch.cat([xa, (xb + t) * s], 1), s.log().flatten(1).sum(1)

    def inverse(self, y):
        ya, yb = y.chunk(2, 1)
        s, t = self.st(ya)
        return torch.cat([ya, yb / s - t], 1)

class FlowStep(nn.Module):
    def __init__(self, c, hidden=512):
        super().__init__()
        self.layers = nn.ModuleList([ActNorm(c), InvConv1x1(c), Coupling(c, hidden)])

    def forward(self, x):
        total = 0
        for f in self.layers:
            x, ld = f(x); total = total + ld
        return x, total

    def inverse(self, y):
        for f in reversed(self.layers):
            y = f.inverse(y)
        return y

def squeeze(x):                                         # (N,C,H,W) -> (N,4C,H/2,W/2)
    N, C, H, W = x.shape
    return x.view(N, C, H // 2, 2, W // 2, 2).permute(0, 1, 3, 5, 2, 4).reshape(N, 4 * C, H // 2, W // 2)

torch.manual_seed(0)
x = squeeze(torch.randn(16, 3, 32, 32))                 # first level of Glow on CIFAR-sized input
step = FlowStep(12)
y, ld = step(x)                                         # first call initialises actnorm
print(y.shape, ld.shape)                                # torch.Size([16, 12, 16, 16]) torch.Size([16])
z, _ = step.layers[0](x)
print(z.mean().abs() < 1e-5, (z.std((0, 2, 3)) - 1).abs().max() < 1e-3)  # tensor(True) tensor(True)

# log-det of the whole step against the brute-force Jacobian on a tiny 1x4x2x2 input (16 dims)
small = FlowStep(4, hidden=8); small(torch.randn(8, 4, 2, 2))   # initialise actnorm
with torch.no_grad():
    for p in small.layers[2].nn[-1].parameters(): p.normal_(std=0.3)  # a non-trivial coupling
v = torch.randn(1, 4, 2, 2)
J = torch.autograd.functional.jacobian(lambda a: small(a.view(1, 4, 2, 2))[0].flatten(), v.flatten())
print(J.shape)                                          # torch.Size([16, 16])
print(torch.allclose(torch.linalg.slogdet(J)[1], small(v)[1][0], atol=1e-4))  # True
print((small.inverse(small(v)[0]) - v).abs().max() < 1e-5)                     # tensor(True)

def sample(step, shape, T=0.7):                         # temperature: z ~ N(0, T^2 I)
    return step.inverse(T * torch.randn(shape))

print(sample(step, (4, 12, 16, 16)).shape)              # torch.Size([4, 12, 16, 16])
```

A Glow step of flow (actnorm with data-dependent init, LU-parameterised 1×1 convolution, affine coupling), checked against the brute-force Jacobian

Test negative log-likelihood in bits/dim (lower is better), Kingma & Dhariwal 2018, Table 2

| dataset | RealNVP | Glow |
| --- | --- | --- |
| CIFAR-10 | 3.49 | 3.35 |
| ImageNet 32×32 | 4.28 | 4.09 |
| ImageNet 64×64 | 3.98 | 3.81 |

The gains come from the same coupling layer with a better mixing layer, better normalisation and more depth. Autoregressive models still had lower bits/dim on these benchmarks, but they sample sequentially. Glow samples a whole image in one pass, and its exact encoder supports interpolation and attribute edits that autoregressive models do not offer.

> **A permutation is a 1×1 convolution with frozen weights:** Reversing or shuffling channels between couplings is multiplication by a permutation matrix at every pixel. Glow learns that matrix. The ordering, and any rotation of the channel basis, becomes a parameter, so the model chooses which mixtures of channels the next coupling copies and which it transforms. The viz uses $c=2$, where the only permutations are "keep" and "swap", and the learned $W$ can also rotate and shear.

> **Initialise actnorm on real data, once:** Data-dependent initialisation must see a representative batch of real, preprocessed data before training starts. Running it on random noise, re-running it after loading a checkpoint, or initialising on a batch of one image gives badly scaled layers. In multi-GPU training, initialise on every replica with the same batch, or broadcast the parameters. Large data-initialised scales also mean large actnorm log-determinants, so check $\sum\log|s|$ when likelihoods look odd.

> **Keep W invertible:** An unconstrained $W$ can drift towards singular, where $\log|\det W|\to-\infty$ and $W^{-1}$ is unstable. The LU form makes $\det W$ explicit. Watch for $|s_i|$ collapsing towards 0, and keep the signs of $s$ fixed, since a sign change would have to pass through $\det W=0$. For sampling, compute $W^{-1}$ in float64 or at least once outside the training graph.

> **Temperature is exact only for volume-constant flows:** Scaling $z$ by $T$ samples from $p_X^{1/T^2}$ (renormalised) only if $\log|\det\partial g/\partial z|$ does not depend on $z$. That holds for additive coupling with actnorm and 1×1 convolutions, whose log-determinants are constants. With affine coupling it is a heuristic, though it works well in practice.

**Strengths**

- Learned channel mixing; cheap determinant: log|det W| = Σ log|sᵢ| with the LU parameterisation
- Actnorm removes batch-norm's dependence on batch statistics and works with a batch size of 1
- Exact likelihood, exact encoder and one-pass sampling: interpolation and attribute manipulation in z
- Better bits/dim than RealNVP on every benchmark reported

**Limitations**

- Very deep and wide: large memory and compute (the latent and every activation have the image's dimension)
- Still behind autoregressive models on bits/dim, and behind GANs and later diffusion models on sample quality
- Samples at T = 1 contain artefacts; temperature trades diversity for quality
- $W^{-1}$ and actnorm initialisation add numerical edge cases

**Question:** A Glow 1×1 convolution with c = 4 acts on a 32×32 feature map. Its LU parameters have s = (2, 0.5, −1, 1). What is its log-determinant?

- A. 0
- B. log 2 ≈ 0.693
- C. 1024 · log 2 ≈ 709.8
- D. It needs the 4096 × 4096 Jacobian

**Answer:** A. 0 — $\log|\det W|=\log2+\log0.5+\log1+\log1=0$, and multiplying by $h\cdot w=1024$ still gives 0. The signs of $s_i$ affect $\det W$ but not $|\det W|$. The full Jacobian is block-diagonal with 1024 copies of $W$, so it never has to be formed.

**Question:** Why does Glow initialise actnorm from data rather than at s = 1, b = 0?

- A. So that the post-actnorm activations start with zero mean and unit variance per channel, which keeps a very deep flow well-scaled from the first step
- B. Because actnorm must be recomputed from every batch, like batch normalization
- C. Because otherwise the log-determinant would not be computable
- D. To make the 1×1 convolution orthogonal

**Answer:** A. So that the post-actnorm activations start with zero mean and unit variance per channel, which keeps a very deep flow well-scaled from the first step — The data are used once, at initialisation. Afterwards $s$ and $b$ are ordinary parameters, so unlike batch norm the density of an image does not depend on its batch, and batches of one image work. The log-determinant $h\cdot w\sum\log|s|$ is computable for any $s$.

**Key points**

- A Glow step is actnorm ($h\,w\sum\log|s|$) $\to$ invertible 1×1 conv ($h\,w\log|\det W|$) $\to$ affine coupling ($\sum\log|s(x_b)|$).
- LU parameterisation $W=PL(U+\operatorname{diag}s)$ gives $\log|\det W|=\sum_i\log|s_i|$: $O(c)$ instead of $O(c^3)$.
- The learned 1×1 convolution generalises the fixed channel reverse/shuffle between couplings and lowered NLL in the paper's ablation.
- Actnorm: per-channel scale and bias initialised so that the first batch has zero mean and unit variance, then trained; it works with batch size 1.
- Multi-scale as RealNVP: L levels of squeeze → K steps → split; CIFAR-10 3.35 bits/dim versus RealNVP's 3.49.
- Temperature sampling $z\sim\mathcal N(0,T^2I)$ with $T\approx0.7$ trades diversity for sample quality.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
