BigGAN

BigGAN (Brock, Donahue and Simonyan, 2019) scales a class-conditional GAN to batch 2048, with spectral norm in G and D, a shared class embedding and a projection discriminator. Orthogonal regularisation lets truncated latents trade variety for fidelity: ImageNet 128² IS 166.5, FID 7.4.

15 min read1374 wordsadvanced0 viewsupdated 2026-10-12

Definition

BigGAN is a class-conditional generative adversarial network for ImageNet, trained at a scale no GAN had used before: batches of 2048 images and about 160M parameters (G and D together) at 128×128. It combines a self-attention GAN backbone, the hinge loss, class-conditional batch norm, a projection discriminator and spectral normalisation in both networks with three additions: a shared class embedding, hierarchical latents (skip-z), and orthogonal regularisation. The last makes the generator amenable to the truncation trick: sampling zz from a truncated normal at test time trades sample variety for fidelity. Andrew Brock, Jeff Donahue and Karen Simonyan, "Large Scale GAN Training for High Fidelity Natural Image Synthesis" (ICLR 2019).

In 2018 the best class-conditional ImageNet model, SA-GAN (Zhang et al.), reached an Inception Score of 52.5 at 128×128. Real ImageNet training images score 233 (the validation set 166). Brock et al. asked how much of that gap is just scale. Their answer: most of it. Going from batch 256 to 2048 alone raised IS by 46%, and the full model improved the state of the art to IS 166.5 and FID 7.4. The paper also documents a cost of scale. These models eventually undergo sudden training collapse, and the paper characterises it instead of hiding it. This article covers what was scaled, the tricks that made scale pay off, and the trade-offs. The conditioning mechanisms themselves are explained in Conditional GAN (cGAN), the base game in Generative adversarial network (GAN).

The baseline they scaled

LD=E(x,y)[max⁡(0, 1−D(x,y))]+Ez,y[max⁡(0, 1+D(G(z,y),y))],LG=− Ez,y[D(G(z,y),y)],D(x,y)=ψ(ϕ(x))+eD(y)⊤ϕ(x)L_D=\mathbb E_{(x,y)}\big[\max(0,\,1-D(x,y))\big]+\mathbb E_{z,y}\big[\max(0,\,1+D(G(z,y),y))\big],\qquad L_G=-\,\mathbb E_{z,y}\big[D(G(z,y),y)\big],\qquad D(x,y)=\psi(\phi(x))+e_D(y)^\top\phi(x)
max⁡(0,1∓D)\max(0,1\mp D)
hinge loss (Lim & Ye 2017; Tran et al. 2017), inherited from SA-GAN
ϕ(x)\phi(x)
the discriminator's features after global sum pooling
eD(y)⊤ϕ(x)e_D(y)^\top\phi(x)
the projection term of Miyato & Koyama (2018): an inner product between a learned class embedding and the features (see Conditional GAN (cGAN))
ψ\psi
a linear layer giving the unconditional part of the score
CBN(h; c)=(1+γ(c))⊙h−μBσB+β(c),cℓ=[ e(y); zℓ ],z=[z0;z1;… ;z5]∈R120\mathrm{CBN}(h;\,c)=\big(1+\gamma(c)\big)\odot\frac{h-\mu_B}{\sigma_B}+\beta(c),\qquad c_\ell=\big[\,e(y);\ z_\ell\,\big],\qquad z=[z_0;z_1;\dots;z_5]\in\mathbb R^{120}
γ(c), β(c)\gamma(c),\ \beta(c)
per-sample gain and bias, linear projections of the conditioning vector (gain centred at 1, bias at 0)
e(y)∈R128e(y)\in\mathbb R^{128}
one shared class embedding for the whole generator, instead of a separate embedding table per batch-norm layer
zℓ∈R20z_\ell\in\mathbb R^{20}
skip-z: the 120-d latent at 128×128 is split into six 20-d chunks, one for the input linear layer and one for each of the five residual blocks
μB,σB\mu_B,\sigma_B
batch statistics, computed across all TPU devices (cross-replica batch norm)

Training setup (Sections 3 and Appendix B–C)

Backbone
SA-GAN ResNet with one self-attention (non-local) block, at 64×64 in the 128×128 models
Normalisation
spectral normalisation in both G and D (each weight divided by its largest singular value); cross-replica batch norm in G
Optimiser
Adam with β1=0\beta_1=0, β2=0.999\beta_2=0.999, constant learning rate 2⋅10−42\cdot10^{-4} for D and 5⋅10−55\cdot10^{-5} for G (SA-GAN's rates halved); two D steps per G step
Init and averaging
orthogonal initialisation; samples come from an exponential moving average of G's weights with decay 0.99990.9999
Hardware
Google TPU v3 Pod, 128 cores at 128×128, 256 at 256×256, 512 at 512×512; 24–48 hours for most models
Latent
z∼N(0,I)z\sim\mathcal N(0,I) during training, 120-d at 128×128, 140-d at 256, 160-d at 512 (BigGAN-deep: 128-d, not split)
BigGAN-deep
bottleneck residual blocks, two blocks per resolution, 4× deeper but fewer parameters (G 50.4M and D 34.6M at 128×128, against 70.4M and 88.0M for BigGAN)

What scale bought

batchch.params (M)sharedskip-zortho.Itr ×10³FIDIS
2566481.5SA-GAN baseline100018.6552.52
5126481.5–––100015.3058.77
10246481.5–––100014.8863.03
20486481.5–––73212.3976.85
204896173.5–––2959.5492.98
204896160.6✓––1859.1894.94
204896158.3✓✓–1528.7398.76
204896158.3✓✓✓1658.5199.31
20486471.3✓✓✓37110.4886.90
ImageNet 128×128 ablation (Brock et al. 2019, Table 1). Rows 5–9: mean over 8 seeds. Itr: iterations ×10³ before collapse, or 1000 if stable to 10⁶

Batch size did the most. An 8× larger batch raised IS by 46%, plausibly because each batch covers more modes and gives both networks better gradients. Models also reached better scores in fewer iterations, but collapsed (scores are from checkpoints saved just before collapse). Widening every layer by 50% roughly doubled the parameters and added another 21%. Doubling depth did not help at first; the later BigGAN-deep fixed that with bottleneck blocks. The shared embedding cut memory and computation and reached a given score in 37% fewer iterations. Skip-z added about 4% in performance and 18% in training speed. Row 8 is the configuration used for the main results.

The truncation trick

zi∼N(0,1)  conditioned on ∣zi∣≤τ(resample any zi with ∣zi∣>τ),Pr⁡(∣zi∣≤τ)=2Φ(τ)−1z_i\sim\mathcal N(0,1)\ \text{ conditioned on } |z_i|\le\tau\quad(\text{resample any } z_i \text{ with } |z_i|>\tau),\qquad \Pr(|z_i|\le\tau)=2\Phi(\tau)-1
τ\tau
truncation threshold, chosen after training; τ=∞\tau=\infty is the training distribution
Φ\Phi
the standard normal CDF

G is trained with z∼N(0,I)z\sim\mathcal N(0,I), but sampled with a truncated normal. Remarkably, the best scores came from this mismatch. As τ\tau shrinks, each zz moves toward the mode of the latent distribution, and samples move toward the modes of G's output: more typical, cleaner images of each class, but less variety. The paper sweeps τ\tau to trace a variety–fidelity curve, like a precision–recall curve. IS does not penalise missing variety within a class, so it rises steadily as τ→0\tau\to0 (precision). FID penalises missing variety but also rewards precision, so it first improves and then degrades sharply. The trick needs a generator that behaves well on the "small-zz" region. Some large models produced saturation artifacts when fed truncated noise. StyleGAN uses a related truncation, but in its intermediate latent space W\mathcal W.

Orthogonal regularisation

Rβ(W)=β ∥W⊤W−I∥F2⟶Rβ(W)=β ∥W⊤W⊙(1−I)∥F2,β=10−4R_\beta(W)=\beta\,\big\|W^\top W-I\big\|_F^2\qquad\longrightarrow\qquad R_\beta(W)=\beta\,\big\|W^\top W\odot(\mathbf 1-I)\big\|_F^2,\qquad \beta=10^{-4}
W⊤WW^\top W
Gram matrix of the filters (dot products between every pair)
1−I\mathbf 1-I
a mask that removes the diagonal, so filter norms are not constrained
β\beta
strength, swept and set to 10⁻⁴; the penalty is applied to G

To make the whole latent space map to good images, the authors push G toward smoothness. Full orthogonal regularisation (Brock et al., 2017) is known to be too limiting. The relaxed version only penalises the pairwise cosine similarity between filters, not their norms. Across the runs of Table 1, only 16% of models without it were amenable to truncation, against 60% with it. A model counts as amenable when truncation improves its scores instead of producing artifacts.

6 classes · G(z, k) = μ_k + s·g(z) · smooth G · no truncationcolour = class k · dashed: real class, radius 2sFID (log scale) vs IS, τ from 0.04 to none0.010.113.544.555.56IS →real data IS 4.70IS (higher = sharper)4.675FID (lower = better)0.2061class accuracy80.4 %spread (real s = 0.6)0.910z_i ~ N(0, 1) cut to |z_i| ≤ τ: 100.0 % of mass kept−3−2−10123real data: IS 4.70, FID 0.0034 (sampling noise) · best FID here: 0.0050 at τ = 1.6Smaller τ: samples collapse onto the class modes; IS rises toward 6 (= C), variety is lost.Large τ: the generator's slightly heavy tails leak samples into neighbouring classes.
generator
τ (Fig. 2a)

IS uses the Bayes classifier of the real classes; FID is the Fréchet distance between Gaussians fitted per class, averaged. Both are toy 2-D stand-ins for the Inception-network scores. The slider's right end means no truncation.

fig.A toy class-conditional generator in 2-D: six classes, real class k is N(μ_k, s²I) with s = 0.6, and G(z, k) = μ_k + s·g(z). The smooth G has slightly heavy tails (g(z) = z(1 + 0.12‖z‖²)). The ill-conditioned G adds a bump that sends latents near z = 0 toward the next class. IS uses the Bayes classifier of the real classes, and FID is the per-class Fréchet distance, averaged: 2-D stand-ins for the Inception-network scores. Untruncated, the smooth G scores IS 4.675 and FID 0.206. Real data scores IS 4.70, and the generator is over-dispersed: class spread 0.910 against 0.6, class accuracy 80.4 %. Truncation first helps both scores. The best FID, 0.0050, comes at τ = 1.6 (IS 4.73). Below that FID degrades while IS keeps rising: τ = 0.5 gives IS 5.602 and FID 0.362, τ = 0.04 gives IS 5.705 (at most 6, the number of classes) and FID 0.687. Each class has collapsed to a small square, because each coordinate is truncated separately and the latent box maps to a square. The ill-conditioned G behaves the same untruncated (IS 4.596, FID 0.219) but is not amenable to truncation: its IS peaks at 5.19 near τ = 0.7 and then falls to 3.54 at τ = 0.04, while FID climbs to 1.48. The lower right panel shows what fraction of each N(0, 1) coordinate survives, 2Φ(τ) − 1.

Results

modelno truncationbest FIDIS matched to validation datamax IS
SN-GAN27.62 / 36.80
SA-GAN18.65 / 52.52
BigGAN8.7 / 98.87.7 / 126.59.6 / 166.325 / 206
BigGAN-deep5.7 / 124.56.3 / 148.17.4 / 166.525 / 253
ImageNet 128×128 FID / IS at three truncation settings (Table 2; ± over at least three seeds)

Because truncation moves a model along a curve, the paper reports three points on it. The first is the truncation with the best FID. The second is where the model's IS equals that of the validation data (166 at 128×128), a rough setting for "maximum variety at good objectness". The third is the maximum IS. The headline "IS 166.5, FID 7.4" is BigGAN-deep at the second point. The original BigGAN gives 166.3 and 9.6 there. At 256×256 and 512×512, BigGAN-deep reaches IS 232.5 / FID 8.1 and IS 241.5 / FID 11.5. On JFT-300M (292M images, 8.5K classes) the same techniques helped further, a 128-channel model was better still, and training stayed stable without heavy regularisation.

Training collapse

Training stays healthy for a long time, then fails abruptly and completely. The authors tracked the top singular values of every weight. In G, most layers are well behaved. A few, usually the first (over-complete, non-convolutional) layer, have a top singular value σ0\sigma_0 that grows throughout training and explodes at collapse. Directly regularising or clamping σ0\sigma_0 (for example W←W−max⁡(0,σ0−σclamp)v0u0⊤W\leftarrow W-\max(0,\sigma_0-\sigma_{\text{clamp}})v_0u_0^\top) stopped the growth but never prevented collapse. In D the spectra are noisy and jump at collapse, and D's loss heads to zero before a sharp rise. Penalising D's gradients with R1=γ2EpD∥∇D(x)∥2R_1=\frac\gamma2\mathbb E_{p_D}\|\nabla D(x)\|^2 at γ=10\gamma=10 made training stable but cut IS by 45%. At γ=1\gamma=1, the weakest penalty that avoided sudden collapse, IS dropped by 20%. D memorises: its accuracy on training images stays above 98% while it is 50–55% on validation images. Stability does not come from G or D alone but from their interaction. The paper's practical choice is to allow collapse and keep the checkpoint saved just before it.

Worked example: what truncation does to a 120-d latent, and one orthogonal penalty

τkept mass 2Φ(τ) − 1draws per coordinatestd of z_iRMS ‖z‖ (120-d)
∞ (training)11110.954
20.95451.0480.87969.636
10.68271.4650.53965.911
0.50.38292.6110.28393.110
0.040.031931.340.02310.253
Truncated N(0, 1) per coordinate, for the thresholds of the paper's Figure 2(a), and the RMS norm √E‖z‖² of a 120-d z (computed in Python)

Reading the table, then one regulariser

  1. 1The std of a truncated normal is 1−2τφ(τ)/(2Φ(τ)−1)\sqrt{1-2\tau\varphi(\tau)/(2\Phi(\tau)-1)}. At τ=0.5\tau=0.5 that is 0.2840.284, so a 120-d zz has RMS norm 120⋅0.284=3.11\sqrt{120}\cdot0.284=3.11. G has never seen a latent this short during training, since under N(0,I)\mathcal N(0,I) the norm concentrates near 120≈10.95\sqrt{120}\approx10.95.
  2. 2Even τ=2\tau=2 resamples 4.55%4.55\% of coordinates. The chance that a 120-d draw needs no resampling at all is 0.9545120=0.00370.9545^{120}=0.0037.
  3. 3Orthogonal penalty, filters as rows: W=(1010.510)W=\begin{pmatrix}1&0&1\\0.5&1&0\end{pmatrix}, WW⊤=(20.50.51.25)WW^\top=\begin{pmatrix}2&0.5\\0.5&1.25\end{pmatrix}. The relaxed penalty sees only the off-diagonal 0.50.5 twice: R=10−4⋅(0.52+0.52)=5⋅10−5R=10^{-4}\cdot(0.5^2+0.5^2)=5\cdot10^{-5}. The full version also charges the norms: 10−4((2−1)2+0.25+(1.25−1)2)=1.5625⋅10−410^{-4}\big((2-1)^2+0.25+(1.25-1)^2\big)=1.5625\cdot10^{-4}.
  4. 4Its gradient is ∂R/∂W=4β MW\partial R/\partial W=4\beta\,MW with MM the masked Gram matrix: 4⋅10−4(00.50.50)W=(120202)⋅10−44\cdot10^{-4}\begin{pmatrix}0&0.5\\0.5&0\end{pmatrix}W=\begin{pmatrix}1&2&0\\2&0&2\end{pmatrix}\cdot10^{-4}, the same as PyTorch autograd. Each filter is pushed away from the other's direction, and neither is shrunk toward unit norm.
python
1import torch, torch.nn as nn, torch.nn.functional as F
2from torch.nn.utils import spectral_norm as SN
3
4class CondBN(nn.Module):
5 """BatchNorm whose gain and bias are linear projections of [shared class embedding, z chunk]."""
6 def __init__(self, ch, cond_dim):
7 super().__init__()
8 self.bn = nn.BatchNorm2d(ch, affine=False)
9 self.gain, self.bias = SN(nn.Linear(cond_dim, ch, bias=False)), SN(nn.Linear(cond_dim, ch, bias=False))
10 def forward(self, x, c):
11 return self.bn(x) * (1 + self.gain(c))[:, :, None, None] + self.bias(c)[:, :, None, None] # gain centred at 1
12
13class GBlock(nn.Module):
14 def __init__(self, cin, cout, cond_dim):
15 super().__init__()
16 self.bn1, self.bn2 = CondBN(cin, cond_dim), CondBN(cout, cond_dim)
17 self.c1, self.c2 = SN(nn.Conv2d(cin, cout, 3, padding=1)), SN(nn.Conv2d(cout, cout, 3, padding=1))
18 self.skip = SN(nn.Conv2d(cin, cout, 1))
19 def forward(self, x, c):
20 h = F.interpolate(F.relu(self.bn1(x, c)), scale_factor=2)
21 h = self.c2(F.relu(self.bn2(self.c1(h), c)))
22 return h + self.skip(F.interpolate(x, scale_factor=2))
23
24class Generator(nn.Module): # 32×32 toy: 3 up-blocks, z ∈ R^80 split into 4 chunks of 20
25 def __init__(self, n_classes=10, ch=32, z_chunk=20, emb=128):
26 super().__init__()
27 self.ch, self.zc = ch, z_chunk
28 self.embed = nn.Embedding(n_classes, emb) # ONE shared embedding for every CondBN
29 self.fc = SN(nn.Linear(z_chunk + emb, 4 * 4 * 4 * ch))
30 self.blocks = nn.ModuleList([GBlock(4 * ch, 4 * ch, z_chunk + emb), GBlock(4 * ch, 2 * ch, z_chunk + emb),
31 GBlock(2 * ch, ch, z_chunk + emb)])
32 self.out = nn.Sequential(nn.BatchNorm2d(ch), nn.ReLU(), SN(nn.Conv2d(ch, 3, 3, padding=1)), nn.Tanh())
33 def forward(self, z, y):
34 zs = z.split(self.zc, dim=1) # skip-z: one chunk per stage
35 e = self.embed(y)
36 h = self.fc(torch.cat([zs[0], e], 1)).view(-1, 4 * self.ch, 4, 4)
37 for blk, zk in zip(self.blocks, zs[1:]):
38 h = blk(h, torch.cat([zk, e], 1))
39 return self.out(h)
40
41class Discriminator(nn.Module): # projection discriminator: D(x, y) = ψ(φ(x)) + embed(y)·φ(x)
42 def __init__(self, n_classes=10, ch=32):
43 super().__init__()
44 self.net = nn.Sequential(SN(nn.Conv2d(3, ch, 3, 2, 1)), nn.ReLU(), SN(nn.Conv2d(ch, 2 * ch, 3, 2, 1)), nn.ReLU(),
45 SN(nn.Conv2d(2 * ch, 4 * ch, 3, 2, 1)), nn.ReLU())
46 self.psi, self.embed = SN(nn.Linear(4 * ch, 1)), SN(nn.Embedding(n_classes, 4 * ch))
47 def forward(self, x, y):
48 h = self.net(x).sum(dim=(2, 3)) # global sum pooling
49 return self.psi(h).squeeze(1) + (self.embed(y) * h).sum(1)
50
51def ortho_reg(model, beta=1e-4): # β‖WᵀW ⊙ (1 − I)‖²_F, filters as rows of W
52 loss = 0.0
53 for name, p in model.named_parameters():
54 if p.dim() < 2 or "embed" in name: # this sketch skips biases, gains and embeddings
55 continue
56 W = p.flatten(1)
57 gram = W @ W.t()
58 loss = loss + beta * (gram * (1 - torch.eye(W.size(0)))).pow(2).sum()
59 return loss
60
61def truncated_z(n, dim, tau): # resample every coordinate with |z_i| > τ
62 z = torch.randn(n, dim)
63 while (bad := z.abs() > tau).any():
64 z[bad] = torch.randn(int(bad.sum()))
65 return z
66
67torch.manual_seed(0)
68G, D = Generator(), Discriminator()
69opt_g = torch.optim.Adam(G.parameters(), lr=5e-5, betas=(0.0, 0.999)) # BigGAN: lr_G 5e-5, lr_D 2e-4, β = (0, 0.999)
70opt_d = torch.optim.Adam(D.parameters(), lr=2e-4, betas=(0.0, 0.999))
71x, y = torch.randn(16, 3, 32, 32).clamp(-1, 1), torch.randint(0, 10, (16,)) # stand-in for a real batch
72for _ in range(2): # two D steps per G step
73 fake = G(torch.randn(16, 80), y).detach()
74 loss_d = F.relu(1 - D(x, y)).mean() + F.relu(1 + D(fake, y)).mean() # hinge loss
75 opt_d.zero_grad(); loss_d.backward(); opt_d.step()
76loss_g = -D(G(torch.randn(16, 80), y), y).mean() + ortho_reg(G)
77opt_g.zero_grad(); loss_g.backward(); opt_g.step()
78
79G.eval()
80with torch.no_grad():
81 z = truncated_z(8, 80, tau=0.5)
82 img = G(z, torch.full((8,), 3))
83print(img.shape, z.abs().max().item() <= 0.5) # torch.Size([8, 3, 32, 32]) True
BigGAN's ingredients at toy scale: shared embedding + skip-z conditional BN, projection D, hinge loss, orthogonal regularisation, truncated sampling

Use when

  • +Large batches and width give large, measurable gains (IS 52.5 → 99.3 in the ablation; 166.5 with BigGAN-deep and truncation)
  • +Truncation gives post-hoc control of fidelity versus variety without retraining
  • +Shared embeddings and skip-z reduce cost and speed up training (37% and 18% fewer iterations)
  • +Pretrained models became a standard benchmark and a source of class-conditional images

Avoid when

  • −Training collapse is not solved: early stopping is required, and regularising D enough to be stable costs 20–45% of IS
  • −Huge compute: 128–512 TPU v3 cores per model, and batch-norm statistics shared across devices
  • −Not every model is amenable to truncation, even with orthogonal regularisation (60%)
  • −Class leakage and weaker results on classes with large-scale structure (crowds) than on texture-defined ones (dogs)

Check yourself

As the truncation threshold τ is lowered from ∞ toward 0 on a model that is amenable to truncation, what happens to IS and FID?

Check yourself

What does BigGAN's version of orthogonal regularisation penalise?

Remember

  • ✓BigGAN = SA-GAN backbone + hinge loss + spectral norm in G and D + batch 2048, with a shared class embedding and skip-z feeding [e(y);zℓ][e(y);z_\ell] to every conditional batch norm, and a projection discriminator.
  • ✓Truncation trick: sample zi∼N(0,1)z_i\sim\mathcal N(0,1) restricted to ∣zi∣≤τ|z_i|\le\tau. Lower τ\tau raises IS (fidelity) and, past an optimum, worsens FID (variety). Orthogonal regularisation β∥W⊤W⊙(1−I)∥F2\beta\|W^\top W\odot(\mathbf 1-I)\|_F^2, β=10−4\beta=10^{-4}, makes G amenable to it.
  • ✓ImageNet 128²: IS 166.5 / FID 7.4 (BigGAN-deep, at the truncation that matches validation IS) vs 52.52 / 18.65 for SA-GAN; batch 256 → 2048 alone gave +46% IS.
  • ✓Training collapse: G's first-layer spectral norm explodes and D memorises its training set (>98% vs 50–55% validation accuracy); strong D regularisation stabilises training but costs 20–45% IS, so models are early-stopped.
  • ✓Optimisation: Adam β=(0,0.999)\beta=(0,0.999), lr 2⋅10−42\cdot10^{-4} (D) and 5⋅10−55\cdot10^{-5} (G), two D steps per G step, EMA of G weights with decay 0.99990.9999.
← back to Deep Learning roadmap