Glow

Glow (Kingma and Dhariwal, 2018) is a RealNVP-style flow whose step is actnorm, then a learned invertible 1×1 convolution, then affine coupling; an LU-parameterised 1×1 convolution has log-determinant h·w·Σlog|sᵢ| at O(c) cost, and it replaces fixed channel permutations.

14 min read1331 wordsadvanced0 viewsupdated 2026-10-12

Definition

Glow The normalizing flow of Diederik P. Kingma and Prafulla Dhariwal, "Glow: Generative Flow with Invertible 1×1 Convolutions" (NeurIPS 2018). One step of flow has three invertible layers: actnorm, a per-channel affine map with data-initialised parameters; an invertible 1×1 convolution, a learned c×cc\times c matrix applied at every pixel, which generalises the fixed channel permutation between couplings; and an affine coupling layer. Steps are stacked KK deep inside each of LL levels of the multi-scale architecture of RealNVP. With the 1×1 weight parameterised by its LU decomposition, every log-determinant costs O(c)O(c) per layer. Glow improved bits/dim on standard benchmarks and generated 256×256256\times256 faces.

RealNVP showed that affine couplings with a triangular Jacobian could model natural images (see Normalizing flows for the general framework). Two parts of its design were fixed by hand. Between couplings the channels were reversed or swapped so that the copied half would be updated next. Batch normalization was used as a flow layer, which makes the density depend on the minibatch and degrades with the tiny per-GPU batches that large images force. Glow replaced the permutation with a learned linear map, WW, that can mix channels arbitrarily, and replaced batch norm with actnorm. On CIFAR-10 the authors compared reversing, random shuffling and the learned 1×1 convolution, each with additive and with affine coupling. With both coupling types, the 1×1 convolution reached a lower negative log-likelihood and converged faster. The paper's demonstrations were faces at 256×256256\times256 from CelebA-HQ, smooth latent interpolations, and attribute manipulation by moving along directions in zz-space. All of these came from a model with an exact likelihood and one-pass sampling.

One step of flow

layerforwardinverselog-determinant
actnormyi,j=s⊙xi,j+by_{i,j}=s\odot x_{i,j}+bxi,j=(yi,j−b)/sx_{i,j}=(y_{i,j}-b)/sh⋅w⋅∑clog⁡∣sc∣h\cdot w\cdot\sum_c\log|s_c|
invertible 1×1 convyi,j=Wxi,jy_{i,j}=W x_{i,j}xi,j=W−1yi,jx_{i,j}=W^{-1}y_{i,j}h⋅w⋅log⁡∣det⁡W∣h\cdot w\cdot\log|\det W|
affine couplingxa,xb=split(x)x_a,x_b=\text{split}(x); (log⁡s,t)=NN(xb)(\log s,t)=\text{NN}(x_b); ya=s⊙xa+ty_a=s\odot x_a+t; yb=xby_b=x_bxa=(ya−t)/sx_a=(y_a-t)/s; xb=ybx_b=y_b∑log⁡∣s∣\sum\log|s|
The three layers of a Glow step for an h×w×c tensor (Kingma & Dhariwal 2018, Table 1)

Here xi,j∈Rcx_{i,j}\in\mathbb R^c is the channel vector at pixel (i,j)(i,j). Actnorm and the 1×1 convolution use the same parameters at every pixel, so their log-determinants are a per-pixel constant times h⋅wh\cdot w, independent of the input. Only the coupling has a data-dependent determinant.

Actnorm

Actnorm is a per-channel scale ss and bias bb, like batch norm with the statistics frozen into parameters. Glow uses data-dependent initialisation. On the first minibatch, bb and ss are set so that the post-actnorm activations of every channel have zero mean and unit variance. After that they are ordinary trainable parameters, and the density of an image no longer depends on what else is in its batch. That matters for large images, where memory limits the minibatch to one image per GPU. The layer is also an inexpensive normaliser for the very deep stacks Glow uses.

The invertible 1×1 convolution

log⁡∣det⁡∂ conv2D(h;W)∂h∣=h⋅w⋅log⁡∣det⁡W∣,W=P L (U+diag⁡(s))\log\Big|\det\frac{\partial\,\mathrm{conv2D}(h;W)}{\partial h}\Big|=h\cdot w\cdot\log|\det W|,\qquad W=P\,L\,\big(U+\operatorname{diag}(s)\big)
W∈Rc×cW\in\mathbb R^{c\times c}
a 1×1 convolution is a matrix multiply of every pixel's channel vector; initialised as a random rotation (log⁡∣det⁡W∣=0\log|\det W|=0)
PP
a permutation matrix, fixed after initialisation
LL
lower triangular with ones on the diagonal (learned)
UU
upper triangular with zeros on the diagonal (learned)
ss
a learned vector; log⁡∣det⁡W∣=∑ilog⁡∣si∣\log|\det W|=\sum_i\log|s_i|

A permutation is a special case of WW, so the learned convolution can only do better than the reverse or shuffle it replaces. Computing det⁡W\det W directly costs O(c3)O(c^3). The paper notes this is often comparable to the convolution itself, which costs O(h⋅w⋅c2)O(h\cdot w\cdot c^2). In the LU form, det⁡P=±1\det P=\pm1, det⁡L=1\det L=1 and det⁡(U+diag⁡s)=∏isi\det(U+\operatorname{diag}s)=\prod_i s_i, so the log-determinant is a sum of cc numbers. The inverse W−1W^{-1} is needed only for sampling, and it can be computed once per sampling run. To initialise, sample a random rotation, take its LU decomposition to get PP, LL, UU and ss, and then optimise LL, UU and ss, keeping the signs of ss fixed.

Affine coupling, Glow style

The coupling layer is RealNVP's with three details. The split is along channels only, since checkerboard masks are no longer needed once WW mixes channels. The network NN\text{NN} is a shallow convolutional stack: a 3×33\times3 convolution and a 1×11\times1 convolution with 512 channels and ReLU, then a final 3×33\times3 convolution. That last convolution is initialised to zero, so every coupling starts as the identity, which the authors found helps train very deep flows. The released code also bounds the scale as s=sigmoid(h~+2)s=\text{sigmoid}(\tilde h+2) instead of exp⁡\exp, which keeps s∈(0,1)s\in(0,1) with s≈0.88s\approx0.88 at initialisation. With s≡1s\equiv1 the layer becomes NICE's additive coupling, with determinant 1.

Multi-scale architecture

One level of Glow, repeated L times (Kingma & Dhariwal 2018, Figure 2)

  1. 1Squeeze: h×w×c→h2×w2×4ch\times w\times c\to\tfrac h2\times\tfrac w2\times4c, a permutation with log-determinant 0.
  2. 2K steps of flow: actnorm →\to 1×1 conv →\to affine coupling, KK times.
  3. 3Split: factor out half of the channels as latent ziz_i; the other half goes to the next level. The last level keeps everything.
  4. 4The log-likelihood is ∑ilog⁡p(zi)\sum_i\log p(z_i) plus the sum of all 3KL3KL per-layer log-determinants. As in RealNVP, the factored-out latents together have exactly as many dimensions as the image.

Setup in the paper

Paper
Kingma, Dhariwal, "Glow: Generative Flow with Invertible 1×1 Convolutions", NeurIPS 2018 (arXiv 1807.03039)
Step of flow
actnorm → invertible 1×1 convolution → affine coupling
Coupling network
3×3 conv → 1×1 conv (512 channels, ReLU) → zero-initialised 3×3 conv
Architecture
multi-scale: L levels of squeeze → K steps → split
Data
CIFAR-10, ImageNet 32×32 and 64×64, LSUN, CelebA-HQ 256×256 (30,000 images, 5-bit)
Ablation
reverse vs. shuffle vs. 1×1 conv, each with additive and affine coupling, on CIFAR-10: 1×1 conv gives the lowest NLL
Sampling
reduced temperature, e.g. T=0.7T=0.7, for the face samples

Temperature sampling

z∼N(0,T2I),x=g(z)⟹pT(x)∝pX(x)1/T2  if log⁡∣det⁡∂g/∂z∣ is constantz\sim\mathcal N(0,T^2I),\quad x=g(z)\qquad\Longrightarrow\qquad p_T(x)\propto p_X(x)^{1/T^2}\ \ \text{if}\ \log|\det\partial g/\partial z|\ \text{is constant}
T<1T<1
samples concentrate on the high-density regions; tails, which are usually artefacts, disappear
constant log⁡∣det⁡∣\text{constant }\log|\det|
true for additive coupling with actnorm and 1×1 convolutions; approximately true otherwise

Maximum likelihood is mass-covering, so a flow puts some density on implausible images, and samples at T=1T=1 show it. Shrinking the prior's standard deviation trades diversity for fidelity. Glow's face samples are drawn at T=0.7T=0.7, and the paper shows quality and diversity changing as TT goes from 0 to 1. The same latent space supports interpolation between two encoded faces and semantic edits. Average zz over images with an attribute, such as smiling, and over images without it, and add a multiple of the difference to the encoding of a new face.

sampling x = f⁻¹(z) · stage 9/9 · step 1: actnorm⁻¹700 samples, colour = |z|/T: <1 1–2 >2exact p(x) (violet) + final samples−44−44inside the box: 700/700∫ p(x) dx over the box = 1.000this sublayer (per pixel)log|det| = Σ log|s_c| = 0.050W of step 1 = P·L·(U + diag s)[1.00 −0.20; 0.20 1.06], s = (1.00, 1.10), det W = 1.100E[log|det ∂z/∂x|], all 9 sublayers0.714mean log p(x) of samples at T = 1.00−2.118 (T = 1: −2.118)max |f(f⁻¹(z)) − z|2.4e-15image of h×w×c: conv log-det = h·w·Σ log|sᵢ|, O(c) with LU instead of O(c³) for det WT < 1 shrinks z ~ N(0, T²I): samples move to the high-density ridge, tails vanish
step
fig.A hand-set (untrained) Glow with c = 2 channels and K = 3 steps. Each point of the plane is the channel vector of one pixel. Step through the 9 sublayers in the sampling direction: for k = 3, 2, 1 it applies coupling⁻¹, then 1×1 conv⁻¹, then actnorm⁻¹. Dots are coloured by |z|/T. The right panel is the exact density p(x) = N(f(x)) · |det ∂f/∂x| from one forward pass, and it integrates to 1.000 over the box. The readouts give each sublayer's log-determinant in the density direction. Actnorm is Σ log s: 0.050, −0.100 and 0.900 for steps 1, 2 and 3. The 1×1 convolution is Σ log|sᵢ| from its LU factors: 0.095, −0.003 and 0. For step 2, W = P·L·(U + diag s) with s = (1.05, −0.95), and det W = +0.9975 because P swaps the channels. The couplings have E[log σ] = −0.082, −0.128 and −0.018 at T = 1. The total is E[log|det ∂z/∂x|] = 0.714, and f(f⁻¹(z)) reproduces z to within 2.4·10⁻¹⁵. Lower the temperature: the mean log p(x) of the samples rises from −2.118 at T = 1 to −1.610 at T = 0.7 and −1.371 at T = 0.5, as the samples collapse onto the ridge of the crescent. The share of samples beyond 2T in z-space stays at 92/700, because every z is just rescaled by T.

Worked example: actnorm and an LU 1×1 convolution on one pixel

Take c=3c=3 channels. On the initialisation batch the channel means are (3.0, −1.0, 0.5)(3.0,\,-1.0,\,0.5) and the standard deviations are (2.0, 0.25, 1.0)(2.0,\,0.25,\,1.0). Let the 1×1 weight have P=IP=I, L=(1000.510−0.20.31)L=\begin{pmatrix}1&0&0\\0.5&1&0\\-0.2&0.3&1\end{pmatrix}, strictly upper UU entries (u12,u13,u23)=(0.4,−0.1,0.6)(u_{12},u_{13},u_{23})=(0.4,-0.1,0.6), and s=(1.2, −0.8, 0.5)s=(1.2,\,-0.8,\,0.5).

quantityvalue
actnorm b=−μb=-\mu, san=1/σs_\text{an}=1/\sigmab = (−3, 1, −0.5), s = (0.5, 4, 1)
a=san⊙(x+b)a=s_\text{an}\odot(x+b)(1.0, 2.0, 1.0)
actnorm ∑log⁡san\sum\log s_\text{an}−0.6931 + 1.3863 + 0 = 0.6931
W=PL(U+diag⁡s)W=PL(U+\operatorname{diag}s)[[1.2, 0.4, −0.1], [0.6, −0.6, 0.55], [−0.24, −0.32, 0.7]]
y=Way=Wa(1.9, −0.05, −0.18)
∑log⁡∣si∣\sum\log|s_i|0.1823 − 0.2231 − 0.6931 = −0.7340
log⁡∣det⁡W∣\log|\det W| by cofactor expansionlog 0.48 = −0.7340 (det W = −0.48)
One pixel x = (5, −0.5, 1.5) through actnorm and the 1×1 convolution (computed in Python)

From one pixel to a feature map

  1. 1Actnorm here uses the per-pixel parameterisation s⊙(x+b)s\odot(x+b) of the released code. The paper's form s⊙x+b′s\odot x+b' is the same map with b′=s⊙bb'=s\odot b.
  2. 2The LU route needs 3 logarithms. The dense determinant needs the full matrix, and the two agree to all printed digits.
  3. 3On a 16×1616\times16 map these per-pixel constants are multiplied by h⋅w=256h\cdot w=256: actnorm contributes 256×0.6931=177.4256\times0.6931=177.4 nats and the 1×1 convolution 256×(−0.7340)=−187.9256\times(-0.7340)=-187.9. The step's two linear layers together give −10.45-10.45 nats, whatever the image.
  4. 4Channel counts grow with depth. In a 6-level model of 256×256×3256\times256\times3 images, the last level has c=384c=384 channels on a 4×44\times4 map. There a dense determinant (≈23c3≈3.8⋅107\approx\tfrac23c^3\approx3.8\cdot10^7 flops) costs more than the convolution itself (h w c2≈2.4⋅106h\,w\,c^2\approx2.4\cdot10^6 multiply-adds). The LU form needs 384 logarithms.
python
1import torch, torch.nn as nn, math
2
3class ActNorm(nn.Module):
4 """y = s * (x + b) per channel; s, b initialised from the first batch, then trained."""
5 def __init__(self, c):
6 super().__init__()
7 self.b = nn.Parameter(torch.zeros(1, c, 1, 1))
8 self.logs = nn.Parameter(torch.zeros(1, c, 1, 1))
9 self.register_buffer("ready", torch.tensor(False))
10
11 def forward(self, x):
12 if not self.ready: # data-dependent initialisation
13 with torch.no_grad():
14 self.b.copy_(-x.mean((0, 2, 3), keepdim=True))
15 self.logs.copy_(-x.std((0, 2, 3), keepdim=True).add(1e-6).log())
16 self.ready.fill_(True)
17 h, w = x.shape[2:]
18 return (x + self.b) * self.logs.exp(), h * w * self.logs.sum()
19
20 def inverse(self, y):
21 return y * (-self.logs).exp() - self.b
22
23class InvConv1x1(nn.Module):
24 """W = P L (U + diag(sign * exp(log_s))); log|det| = h*w*sum(log_s): O(c), not O(c^3)."""
25 def __init__(self, c):
26 super().__init__()
27 q, _ = torch.linalg.qr(torch.randn(c, c)) # random rotation
28 P, L, U = torch.linalg.lu(q)
29 s = torch.diagonal(U)
30 self.register_buffer("P", P)
31 self.register_buffer("sign", torch.sign(s))
32 self.register_buffer("lmask", torch.tril(torch.ones(c, c), -1))
33 self.L = nn.Parameter(L)
34 self.U = nn.Parameter(torch.triu(U, 1))
35 self.log_s = nn.Parameter(s.abs().log())
36
37 def weight(self):
38 c = self.log_s.numel()
39 L = self.L * self.lmask + torch.eye(c)
40 U = self.U * self.lmask.T + torch.diag(self.sign * self.log_s.exp())
41 return self.P @ L @ U
42
43 def forward(self, x):
44 h, w = x.shape[2:]
45 return torch.einsum("ij,njhw->nihw", self.weight(), x), h * w * self.log_s.sum()
46
47 def inverse(self, y):
48 return torch.einsum("ij,njhw->nihw", torch.linalg.inv(self.weight()), y)
49
50class Coupling(nn.Module):
51 """Affine coupling; NN = conv3x3 -> conv1x1 -> zero-initialised conv3x3."""
52 def __init__(self, c, hidden=512):
53 super().__init__()
54 self.nn = nn.Sequential(nn.Conv2d(c // 2, hidden, 3, padding=1), nn.ReLU(),
55 nn.Conv2d(hidden, hidden, 1), nn.ReLU(),
56 nn.Conv2d(hidden, c, 3, padding=1))
57 nn.init.zeros_(self.nn[-1].weight); nn.init.zeros_(self.nn[-1].bias)
58
59 def st(self, xa):
60 h = self.nn(xa)
61 return torch.sigmoid(h[:, 1::2] + 2), h[:, 0::2] # scale in (0, 1), shift
62
63 def forward(self, x):
64 xa, xb = x.chunk(2, 1)
65 s, t = self.st(xa)
66 return torch.cat([xa, (xb + t) * s], 1), s.log().flatten(1).sum(1)
67
68 def inverse(self, y):
69 ya, yb = y.chunk(2, 1)
70 s, t = self.st(ya)
71 return torch.cat([ya, yb / s - t], 1)
72
73class FlowStep(nn.Module):
74 def __init__(self, c, hidden=512):
75 super().__init__()
76 self.layers = nn.ModuleList([ActNorm(c), InvConv1x1(c), Coupling(c, hidden)])
77
78 def forward(self, x):
79 total = 0
80 for f in self.layers:
81 x, ld = f(x); total = total + ld
82 return x, total
83
84 def inverse(self, y):
85 for f in reversed(self.layers):
86 y = f.inverse(y)
87 return y
88
89def squeeze(x): # (N,C,H,W) -> (N,4C,H/2,W/2)
90 N, C, H, W = x.shape
91 return x.view(N, C, H // 2, 2, W // 2, 2).permute(0, 1, 3, 5, 2, 4).reshape(N, 4 * C, H // 2, W // 2)
92
93torch.manual_seed(0)
94x = squeeze(torch.randn(16, 3, 32, 32)) # first level of Glow on CIFAR-sized input
95step = FlowStep(12)
96y, ld = step(x) # first call initialises actnorm
97print(y.shape, ld.shape) # torch.Size([16, 12, 16, 16]) torch.Size([16])
98z, _ = step.layers[0](x)
99print(z.mean().abs() < 1e-5, (z.std((0, 2, 3)) - 1).abs().max() < 1e-3) # tensor(True) tensor(True)
100
101# log-det of the whole step against the brute-force Jacobian on a tiny 1x4x2x2 input (16 dims)
102small = FlowStep(4, hidden=8); small(torch.randn(8, 4, 2, 2)) # initialise actnorm
103with torch.no_grad():
104 for p in small.layers[2].nn[-1].parameters(): p.normal_(std=0.3) # a non-trivial coupling
105v = torch.randn(1, 4, 2, 2)
106J = torch.autograd.functional.jacobian(lambda a: small(a.view(1, 4, 2, 2))[0].flatten(), v.flatten())
107print(J.shape) # torch.Size([16, 16])
108print(torch.allclose(torch.linalg.slogdet(J)[1], small(v)[1][0], atol=1e-4)) # True
109print((small.inverse(small(v)[0]) - v).abs().max() < 1e-5) # tensor(True)
110
111def sample(step, shape, T=0.7): # temperature: z ~ N(0, T^2 I)
112 return step.inverse(T * torch.randn(shape))
113
114print(sample(step, (4, 12, 16, 16)).shape) # torch.Size([4, 12, 16, 16])
A Glow step of flow (actnorm with data-dependent init, LU-parameterised 1×1 convolution, affine coupling), checked against the brute-force Jacobian
datasetRealNVPGlow
CIFAR-103.493.35
ImageNet 32×324.284.09
ImageNet 64×643.983.81
Test negative log-likelihood in bits/dim (lower is better), Kingma & Dhariwal 2018, Table 2

The gains come from the same coupling layer with a better mixing layer, better normalisation and more depth. Autoregressive models still had lower bits/dim on these benchmarks, but they sample sequentially. Glow samples a whole image in one pass, and its exact encoder supports interpolation and attribute edits that autoregressive models do not offer.

Use when

  • +Learned channel mixing; cheap determinant: log|det W| = Σ log|sᵢ| with the LU parameterisation
  • +Actnorm removes batch-norm's dependence on batch statistics and works with a batch size of 1
  • +Exact likelihood, exact encoder and one-pass sampling: interpolation and attribute manipulation in z
  • +Better bits/dim than RealNVP on every benchmark reported

Avoid when

  • −Very deep and wide: large memory and compute (the latent and every activation have the image's dimension)
  • −Still behind autoregressive models on bits/dim, and behind GANs and later diffusion models on sample quality
  • −Samples at T = 1 contain artefacts; temperature trades diversity for quality
  • −W−1W^{-1} and actnorm initialisation add numerical edge cases

Check yourself

A Glow 1×1 convolution with c = 4 acts on a 32×32 feature map. Its LU parameters have s = (2, 0.5, −1, 1). What is its log-determinant?

Check yourself

Why does Glow initialise actnorm from data rather than at s = 1, b = 0?

Remember

  • ✓A Glow step is actnorm (h w∑log⁡∣s∣h\,w\sum\log|s|) →\to invertible 1×1 conv (h wlog⁡∣det⁡W∣h\,w\log|\det W|) →\to affine coupling (∑log⁡∣s(xb)∣\sum\log|s(x_b)|).
  • ✓LU parameterisation W=PL(U+diag⁡s)W=PL(U+\operatorname{diag}s) gives log⁡∣det⁡W∣=∑ilog⁡∣si∣\log|\det W|=\sum_i\log|s_i|: O(c)O(c) instead of O(c3)O(c^3).
  • ✓The learned 1×1 convolution generalises the fixed channel reverse/shuffle between couplings and lowered NLL in the paper's ablation.
  • ✓Actnorm: per-channel scale and bias initialised so that the first batch has zero mean and unit variance, then trained; it works with batch size 1.
  • ✓Multi-scale as RealNVP: L levels of squeeze → K steps → split; CIFAR-10 3.35 bits/dim versus RealNVP's 3.49.
  • ✓Temperature sampling z∼N(0,T2I)z\sim\mathcal N(0,T^2I) with T≈0.7T\approx0.7 trades diversity for sample quality.
← back to Deep Learning roadmap