Definition
Glow The normalizing flow of Diederik P. Kingma and Prafulla Dhariwal, "Glow: Generative Flow with Invertible 1×1 Convolutions" (NeurIPS 2018). One step of flow has three invertible layers: actnorm, a per-channel affine map with data-initialised parameters; an invertible 1×1 convolution, a learned matrix applied at every pixel, which generalises the fixed channel permutation between couplings; and an affine coupling layer. Steps are stacked deep inside each of levels of the multi-scale architecture of RealNVP. With the 1×1 weight parameterised by its LU decomposition, every log-determinant costs per layer. Glow improved bits/dim on standard benchmarks and generated faces.
RealNVP showed that affine couplings with a triangular Jacobian could model natural images (see Normalizing flows for the general framework). Two parts of its design were fixed by hand. Between couplings the channels were reversed or swapped so that the copied half would be updated next. Batch normalization was used as a flow layer, which makes the density depend on the minibatch and degrades with the tiny per-GPU batches that large images force. Glow replaced the permutation with a learned linear map, , that can mix channels arbitrarily, and replaced batch norm with actnorm. On CIFAR-10 the authors compared reversing, random shuffling and the learned 1×1 convolution, each with additive and with affine coupling. With both coupling types, the 1×1 convolution reached a lower negative log-likelihood and converged faster. The paper's demonstrations were faces at from CelebA-HQ, smooth latent interpolations, and attribute manipulation by moving along directions in -space. All of these came from a model with an exact likelihood and one-pass sampling.
One step of flow
| layer | forward | inverse | log-determinant |
|---|---|---|---|
| actnorm | |||
| invertible 1×1 conv | |||
| affine coupling | ; ; ; | ; |
Here is the channel vector at pixel . Actnorm and the 1×1 convolution use the same parameters at every pixel, so their log-determinants are a per-pixel constant times , independent of the input. Only the coupling has a data-dependent determinant.
Actnorm
Actnorm is a per-channel scale and bias , like batch norm with the statistics frozen into parameters. Glow uses data-dependent initialisation. On the first minibatch, and are set so that the post-actnorm activations of every channel have zero mean and unit variance. After that they are ordinary trainable parameters, and the density of an image no longer depends on what else is in its batch. That matters for large images, where memory limits the minibatch to one image per GPU. The layer is also an inexpensive normaliser for the very deep stacks Glow uses.
The invertible 1×1 convolution
- a 1×1 convolution is a matrix multiply of every pixel's channel vector; initialised as a random rotation ()
- a permutation matrix, fixed after initialisation
- lower triangular with ones on the diagonal (learned)
- upper triangular with zeros on the diagonal (learned)
- a learned vector;
A permutation is a special case of , so the learned convolution can only do better than the reverse or shuffle it replaces. Computing directly costs . The paper notes this is often comparable to the convolution itself, which costs . In the LU form, , and , so the log-determinant is a sum of numbers. The inverse is needed only for sampling, and it can be computed once per sampling run. To initialise, sample a random rotation, take its LU decomposition to get , , and , and then optimise , and , keeping the signs of fixed.
Affine coupling, Glow style
The coupling layer is RealNVP's with three details. The split is along channels only, since checkerboard masks are no longer needed once mixes channels. The network is a shallow convolutional stack: a convolution and a convolution with 512 channels and ReLU, then a final convolution. That last convolution is initialised to zero, so every coupling starts as the identity, which the authors found helps train very deep flows. The released code also bounds the scale as instead of , which keeps with at initialisation. With the layer becomes NICE's additive coupling, with determinant 1.
Multi-scale architecture
One level of Glow, repeated L times (Kingma & Dhariwal 2018, Figure 2)
- 1Squeeze: , a permutation with log-determinant 0.
- 2K steps of flow: actnorm 1×1 conv affine coupling, times.
- 3Split: factor out half of the channels as latent ; the other half goes to the next level. The last level keeps everything.
- 4The log-likelihood is plus the sum of all per-layer log-determinants. As in RealNVP, the factored-out latents together have exactly as many dimensions as the image.
Setup in the paper
- Paper
- Kingma, Dhariwal, "Glow: Generative Flow with Invertible 1×1 Convolutions", NeurIPS 2018 (arXiv 1807.03039)
- Step of flow
- actnorm → invertible 1×1 convolution → affine coupling
- Coupling network
- 3×3 conv → 1×1 conv (512 channels, ReLU) → zero-initialised 3×3 conv
- Architecture
- multi-scale: L levels of squeeze → K steps → split
- Data
- CIFAR-10, ImageNet 32×32 and 64×64, LSUN, CelebA-HQ 256×256 (30,000 images, 5-bit)
- Ablation
- reverse vs. shuffle vs. 1×1 conv, each with additive and affine coupling, on CIFAR-10: 1×1 conv gives the lowest NLL
- Sampling
- reduced temperature, e.g. , for the face samples
Temperature sampling
- samples concentrate on the high-density regions; tails, which are usually artefacts, disappear
- true for additive coupling with actnorm and 1×1 convolutions; approximately true otherwise
Maximum likelihood is mass-covering, so a flow puts some density on implausible images, and samples at show it. Shrinking the prior's standard deviation trades diversity for fidelity. Glow's face samples are drawn at , and the paper shows quality and diversity changing as goes from 0 to 1. The same latent space supports interpolation between two encoded faces and semantic edits. Average over images with an attribute, such as smiling, and over images without it, and add a multiple of the difference to the encoding of a new face.
Worked example: actnorm and an LU 1×1 convolution on one pixel
Take channels. On the initialisation batch the channel means are and the standard deviations are . Let the 1×1 weight have , , strictly upper entries , and .
| quantity | value |
|---|---|
| actnorm , | b = (−3, 1, −0.5), s = (0.5, 4, 1) |
| (1.0, 2.0, 1.0) | |
| actnorm | −0.6931 + 1.3863 + 0 = 0.6931 |
| [[1.2, 0.4, −0.1], [0.6, −0.6, 0.55], [−0.24, −0.32, 0.7]] | |
| (1.9, −0.05, −0.18) | |
| 0.1823 − 0.2231 − 0.6931 = −0.7340 | |
| by cofactor expansion | log 0.48 = −0.7340 (det W = −0.48) |
From one pixel to a feature map
- 1Actnorm here uses the per-pixel parameterisation of the released code. The paper's form is the same map with .
- 2The LU route needs 3 logarithms. The dense determinant needs the full matrix, and the two agree to all printed digits.
- 3On a map these per-pixel constants are multiplied by : actnorm contributes nats and the 1×1 convolution . The step's two linear layers together give nats, whatever the image.
- 4Channel counts grow with depth. In a 6-level model of images, the last level has channels on a map. There a dense determinant ( flops) costs more than the convolution itself ( multiply-adds). The LU form needs 384 logarithms.
1import torch, torch.nn as nn, math23class ActNorm(nn.Module):4 """y = s * (x + b) per channel; s, b initialised from the first batch, then trained."""5 def __init__(self, c):6 super().__init__()7 self.b = nn.Parameter(torch.zeros(1, c, 1, 1))8 self.logs = nn.Parameter(torch.zeros(1, c, 1, 1))9 self.register_buffer("ready", torch.tensor(False))1011 def forward(self, x):12 if not self.ready: # data-dependent initialisation13 with torch.no_grad():14 self.b.copy_(-x.mean((0, 2, 3), keepdim=True))15 self.logs.copy_(-x.std((0, 2, 3), keepdim=True).add(1e-6).log())16 self.ready.fill_(True)17 h, w = x.shape[2:]18 return (x + self.b) * self.logs.exp(), h * w * self.logs.sum()1920 def inverse(self, y):21 return y * (-self.logs).exp() - self.b2223class InvConv1x1(nn.Module):24 """W = P L (U + diag(sign * exp(log_s))); log|det| = h*w*sum(log_s): O(c), not O(c^3)."""25 def __init__(self, c):26 super().__init__()27 q, _ = torch.linalg.qr(torch.randn(c, c)) # random rotation28 P, L, U = torch.linalg.lu(q)29 s = torch.diagonal(U)30 self.register_buffer("P", P)31 self.register_buffer("sign", torch.sign(s))32 self.register_buffer("lmask", torch.tril(torch.ones(c, c), -1))33 self.L = nn.Parameter(L)34 self.U = nn.Parameter(torch.triu(U, 1))35 self.log_s = nn.Parameter(s.abs().log())3637 def weight(self):38 c = self.log_s.numel()39 L = self.L * self.lmask + torch.eye(c)40 U = self.U * self.lmask.T + torch.diag(self.sign * self.log_s.exp())41 return self.P @ L @ U4243 def forward(self, x):44 h, w = x.shape[2:]45 return torch.einsum("ij,njhw->nihw", self.weight(), x), h * w * self.log_s.sum()4647 def inverse(self, y):48 return torch.einsum("ij,njhw->nihw", torch.linalg.inv(self.weight()), y)4950class Coupling(nn.Module):51 """Affine coupling; NN = conv3x3 -> conv1x1 -> zero-initialised conv3x3."""52 def __init__(self, c, hidden=512):53 super().__init__()54 self.nn = nn.Sequential(nn.Conv2d(c // 2, hidden, 3, padding=1), nn.ReLU(),55 nn.Conv2d(hidden, hidden, 1), nn.ReLU(),56 nn.Conv2d(hidden, c, 3, padding=1))57 nn.init.zeros_(self.nn[-1].weight); nn.init.zeros_(self.nn[-1].bias)5859 def st(self, xa):60 h = self.nn(xa)61 return torch.sigmoid(h[:, 1::2] + 2), h[:, 0::2] # scale in (0, 1), shift6263 def forward(self, x):64 xa, xb = x.chunk(2, 1)65 s, t = self.st(xa)66 return torch.cat([xa, (xb + t) * s], 1), s.log().flatten(1).sum(1)6768 def inverse(self, y):69 ya, yb = y.chunk(2, 1)70 s, t = self.st(ya)71 return torch.cat([ya, yb / s - t], 1)7273class FlowStep(nn.Module):74 def __init__(self, c, hidden=512):75 super().__init__()76 self.layers = nn.ModuleList([ActNorm(c), InvConv1x1(c), Coupling(c, hidden)])7778 def forward(self, x):79 total = 080 for f in self.layers:81 x, ld = f(x); total = total + ld82 return x, total8384 def inverse(self, y):85 for f in reversed(self.layers):86 y = f.inverse(y)87 return y8889def squeeze(x): # (N,C,H,W) -> (N,4C,H/2,W/2)90 N, C, H, W = x.shape91 return x.view(N, C, H // 2, 2, W // 2, 2).permute(0, 1, 3, 5, 2, 4).reshape(N, 4 * C, H // 2, W // 2)9293torch.manual_seed(0)94x = squeeze(torch.randn(16, 3, 32, 32)) # first level of Glow on CIFAR-sized input95step = FlowStep(12)96y, ld = step(x) # first call initialises actnorm97print(y.shape, ld.shape) # torch.Size([16, 12, 16, 16]) torch.Size([16])98z, _ = step.layers[0](x)99print(z.mean().abs() < 1e-5, (z.std((0, 2, 3)) - 1).abs().max() < 1e-3) # tensor(True) tensor(True)100101# log-det of the whole step against the brute-force Jacobian on a tiny 1x4x2x2 input (16 dims)102small = FlowStep(4, hidden=8); small(torch.randn(8, 4, 2, 2)) # initialise actnorm103with torch.no_grad():104 for p in small.layers[2].nn[-1].parameters(): p.normal_(std=0.3) # a non-trivial coupling105v = torch.randn(1, 4, 2, 2)106J = torch.autograd.functional.jacobian(lambda a: small(a.view(1, 4, 2, 2))[0].flatten(), v.flatten())107print(J.shape) # torch.Size([16, 16])108print(torch.allclose(torch.linalg.slogdet(J)[1], small(v)[1][0], atol=1e-4)) # True109print((small.inverse(small(v)[0]) - v).abs().max() < 1e-5) # tensor(True)110111def sample(step, shape, T=0.7): # temperature: z ~ N(0, T^2 I)112 return step.inverse(T * torch.randn(shape))113114print(sample(step, (4, 12, 16, 16)).shape) # torch.Size([4, 12, 16, 16])| dataset | RealNVP | Glow |
|---|---|---|
| CIFAR-10 | 3.49 | 3.35 |
| ImageNet 32×32 | 4.28 | 4.09 |
| ImageNet 64×64 | 3.98 | 3.81 |
The gains come from the same coupling layer with a better mixing layer, better normalisation and more depth. Autoregressive models still had lower bits/dim on these benchmarks, but they sample sequentially. Glow samples a whole image in one pass, and its exact encoder supports interpolation and attribute edits that autoregressive models do not offer.
Use when
- +Learned channel mixing; cheap determinant: log|det W| = Σ log|sᵢ| with the LU parameterisation
- +Actnorm removes batch-norm's dependence on batch statistics and works with a batch size of 1
- +Exact likelihood, exact encoder and one-pass sampling: interpolation and attribute manipulation in z
- +Better bits/dim than RealNVP on every benchmark reported
Avoid when
- −Very deep and wide: large memory and compute (the latent and every activation have the image's dimension)
- −Still behind autoregressive models on bits/dim, and behind GANs and later diffusion models on sample quality
- −Samples at T = 1 contain artefacts; temperature trades diversity for quality
- − and actnorm initialisation add numerical edge cases
Check yourself
A Glow 1×1 convolution with c = 4 acts on a 32×32 feature map. Its LU parameters have s = (2, 0.5, −1, 1). What is its log-determinant?
Check yourself
Why does Glow initialise actnorm from data rather than at s = 1, b = 0?
Remember
- ✓A Glow step is actnorm () invertible 1×1 conv () affine coupling ().
- ✓LU parameterisation gives : instead of .
- ✓The learned 1×1 convolution generalises the fixed channel reverse/shuffle between couplings and lowered NLL in the paper's ablation.
- ✓Actnorm: per-channel scale and bias initialised so that the first batch has zero mean and unit variance, then trained; it works with batch size 1.
- ✓Multi-scale as RealNVP: L levels of squeeze → K steps → split; CIFAR-10 3.35 bits/dim versus RealNVP's 3.49.
- ✓Temperature sampling with trades diversity for sample quality.