# BigGAN

> BigGAN (Brock, Donahue and Simonyan, 2019) scales a class-conditional GAN to batch 2048, with spectral norm in G and D, a shared class embedding and a projection discriminator. Orthogonal regularisation lets truncated latents trade variety for fidelity: ImageNet 128² IS 166.5, FID 7.4.

Source: https://www.mlfoundry.in/learn/biggan  
Section: Deep Learning › Generative Deep Learning Algorithms  
Level: advanced · 15 min read · updated 2026-10-12

**BigGAN:** is a class-conditional generative adversarial network for ImageNet, trained at a scale no GAN had used before: batches of 2048 images and about 160M parameters (G and D together) at 128×128. It combines a self-attention GAN backbone, the hinge loss, class-conditional batch norm, a projection discriminator and spectral normalisation in both networks with three additions: a **shared class embedding**, **hierarchical latents** (skip-z), and **orthogonal regularisation**. The last makes the generator amenable to the **truncation trick**: sampling $z$ from a truncated normal at test time trades sample variety for fidelity. Andrew Brock, Jeff Donahue and Karen Simonyan, "Large Scale GAN Training for High Fidelity Natural Image Synthesis" (ICLR 2019).

In 2018 the best class-conditional ImageNet model, SA-GAN (Zhang et al.), reached an Inception Score of 52.5 at 128×128. Real ImageNet training images score 233 (the validation set 166). Brock et al. asked how much of that gap is just scale. Their answer: most of it. Going from batch 256 to 2048 alone raised IS by 46%, and the full model improved the state of the art to IS 166.5 and FID 7.4. The paper also documents a cost of scale. These models eventually undergo sudden **training collapse**, and the paper characterises it instead of hiding it. This article covers what was scaled, the tricks that made scale pay off, and the trade-offs. The conditioning mechanisms themselves are explained in **Conditional GAN (cGAN)**, the base game in **Generative adversarial network (GAN)**.

## The baseline they scaled

$$
L_D=\mathbb E_{(x,y)}\big[\max(0,\,1-D(x,y))\big]+\mathbb E_{z,y}\big[\max(0,\,1+D(G(z,y),y))\big],\qquad L_G=-\,\mathbb E_{z,y}\big[D(G(z,y),y)\big],\qquad D(x,y)=\psi(\phi(x))+e_D(y)^\top\phi(x)
$$

- $\max(0,1\mp D)$: hinge loss (Lim & Ye 2017; Tran et al. 2017), inherited from SA-GAN
- $\phi(x)$: the discriminator's features after global sum pooling
- $e_D(y)^\top\phi(x)$: the **projection** term of Miyato & Koyama (2018): an inner product between a learned class embedding and the features (see **Conditional GAN (cGAN)**)
- $\psi$: a linear layer giving the unconditional part of the score

$$
\mathrm{CBN}(h;\,c)=\big(1+\gamma(c)\big)\odot\frac{h-\mu_B}{\sigma_B}+\beta(c),\qquad c_\ell=\big[\,e(y);\ z_\ell\,\big],\qquad z=[z_0;z_1;\dots;z_5]\in\mathbb R^{120}
$$

- $\gamma(c),\ \beta(c)$: per-sample gain and bias, linear projections of the conditioning vector (gain centred at 1, bias at 0)
- $e(y)\in\mathbb R^{128}$: **one shared** class embedding for the whole generator, instead of a separate embedding table per batch-norm layer
- $z_\ell\in\mathbb R^{20}$: **skip-z**: the 120-d latent at 128×128 is split into six 20-d chunks, one for the input linear layer and one for each of the five residual blocks
- $\mu_B,\sigma_B$: batch statistics, computed across all TPU devices (cross-replica batch norm)

**Training setup (Sections 3 and Appendix B–C)**

- **Backbone:** SA-GAN ResNet with one self-attention (non-local) block, at 64×64 in the 128×128 models
- **Normalisation:** spectral normalisation in both G and D (each weight divided by its largest singular value); cross-replica batch norm in G
- **Optimiser:** Adam with $\beta_1=0$, $\beta_2=0.999$, constant learning rate $2\cdot10^{-4}$ for D and $5\cdot10^{-5}$ for G (SA-GAN's rates halved); two D steps per G step
- **Init and averaging:** orthogonal initialisation; samples come from an exponential moving average of G's weights with decay $0.9999$
- **Hardware:** Google TPU v3 Pod, 128 cores at 128×128, 256 at 256×256, 512 at 512×512; 24–48 hours for most models
- **Latent:** $z\sim\mathcal N(0,I)$ during training, 120-d at 128×128, 140-d at 256, 160-d at 512 (BigGAN-deep: 128-d, not split)
- **BigGAN-deep:** bottleneck residual blocks, two blocks per resolution, 4× deeper but fewer parameters (G 50.4M and D 34.6M at 128×128, against 70.4M and 88.0M for BigGAN)

## What scale bought

ImageNet 128×128 ablation (Brock et al. 2019, Table 1). Rows 5–9: mean over 8 seeds. Itr: iterations ×10³ before collapse, or 1000 if stable to 10⁶

| batch | ch. | params (M) | shared | skip-z | ortho. | Itr ×10³ | FID | IS |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 256 | 64 | 81.5 | SA-GAN baseline |  |  | 1000 | 18.65 | 52.52 |
| 512 | 64 | 81.5 | – | – | – | 1000 | 15.30 | 58.77 |
| 1024 | 64 | 81.5 | – | – | – | 1000 | 14.88 | 63.03 |
| 2048 | 64 | 81.5 | – | – | – | 732 | 12.39 | 76.85 |
| 2048 | 96 | 173.5 | – | – | – | 295 | 9.54 | 92.98 |
| 2048 | 96 | 160.6 | ✓ | – | – | 185 | 9.18 | 94.94 |
| 2048 | 96 | 158.3 | ✓ | ✓ | – | 152 | 8.73 | 98.76 |
| 2048 | 96 | 158.3 | ✓ | ✓ | ✓ | 165 | 8.51 | 99.31 |
| 2048 | 64 | 71.3 | ✓ | ✓ | ✓ | 371 | 10.48 | 86.90 |

Batch size did the most. An 8× larger batch raised IS by 46%, plausibly because each batch covers more modes and gives both networks better gradients. Models also reached better scores in fewer iterations, but collapsed (scores are from checkpoints saved just before collapse). Widening every layer by 50% roughly doubled the parameters and added another 21%. Doubling depth did not help at first; the later BigGAN-deep fixed that with bottleneck blocks. The shared embedding cut memory and computation and reached a given score in 37% fewer iterations. Skip-z added about 4% in performance and 18% in training speed. Row 8 is the configuration used for the main results.

## The truncation trick

$$
z_i\sim\mathcal N(0,1)\ \text{ conditioned on } |z_i|\le\tau\quad(\text{resample any } z_i \text{ with } |z_i|>\tau),\qquad \Pr(|z_i|\le\tau)=2\Phi(\tau)-1
$$

- $\tau$: truncation threshold, chosen **after** training; $\tau=\infty$ is the training distribution
- $\Phi$: the standard normal CDF

G is trained with $z\sim\mathcal N(0,I)$, but sampled with a truncated normal. Remarkably, the best scores came from this mismatch. As $\tau$ shrinks, each $z$ moves toward the mode of the latent distribution, and samples move toward the modes of G's output: more typical, cleaner images of each class, but less variety. The paper sweeps $\tau$ to trace a variety–fidelity curve, like a precision–recall curve. IS does not penalise missing variety within a class, so it rises steadily as $\tau\to0$ (precision). FID penalises missing variety but also rewards precision, so it first improves and then degrades sharply. The trick needs a generator that behaves well on the "small-$z$" region. Some large models produced saturation artifacts when fed truncated noise. **StyleGAN** uses a related truncation, but in its intermediate latent space $\mathcal W$.

## Orthogonal regularisation

$$
R_\beta(W)=\beta\,\big\|W^\top W-I\big\|_F^2\qquad\longrightarrow\qquad R_\beta(W)=\beta\,\big\|W^\top W\odot(\mathbf 1-I)\big\|_F^2,\qquad \beta=10^{-4}
$$

- $W^\top W$: Gram matrix of the filters (dot products between every pair)
- $\mathbf 1-I$: a mask that removes the diagonal, so filter norms are not constrained
- $\beta$: strength, swept and set to 10⁻⁴; the penalty is applied to G

To make the whole latent space map to good images, the authors push G toward smoothness. Full orthogonal regularisation (Brock et al., 2017) is known to be too limiting. The relaxed version only penalises the pairwise cosine similarity between filters, not their norms. Across the runs of Table 1, only 16% of models without it were amenable to truncation, against 60% with it. A model counts as amenable when truncation improves its scores instead of producing artifacts.

> Figure: A toy class-conditional generator in 2-D: six classes, real class k is N(μ_k, s²I) with s = 0.6, and G(z, k) = μ_k + s·g(z). The smooth G has slightly heavy tails (g(z) = z(1 + 0.12‖z‖²)). The ill-conditioned G adds a bump that sends latents near z = 0 toward the next class. IS uses the Bayes classifier of the real classes, and FID is the per-class Fréchet distance, averaged: 2-D stand-ins for the Inception-network scores. Untruncated, the smooth G scores IS 4.675 and FID 0.206. Real data scores IS 4.70, and the generator is over-dispersed: class spread 0.910 against 0.6, class accuracy 80.4 %. Truncation first helps both scores. The best FID, 0.0050, comes at τ = 1.6 (IS 4.73). Below that FID degrades while IS keeps rising: τ = 0.5 gives IS 5.602 and FID 0.362, τ = 0.04 gives IS 5.705 (at most 6, the number of classes) and FID 0.687. Each class has collapsed to a small square, because each coordinate is truncated separately and the latent box maps to a square. The ill-conditioned G behaves the same untruncated (IS 4.596, FID 0.219) but is not amenable to truncation: its IS peaks at 5.19 near τ = 0.7 and then falls to 3.54 at τ = 0.04, while FID climbs to 1.48. The lower right panel shows what fraction of each N(0, 1) coordinate survives, 2Φ(τ) − 1.

## Results

ImageNet 128×128 FID / IS at three truncation settings (Table 2; ± over at least three seeds)

| model | no truncation | best FID | IS matched to validation data | max IS |
| --- | --- | --- | --- | --- |
| SN-GAN | 27.62 / 36.80 |  |  |  |
| SA-GAN | 18.65 / 52.52 |  |  |  |
| BigGAN | 8.7 / 98.8 | 7.7 / 126.5 | 9.6 / 166.3 | 25 / 206 |
| BigGAN-deep | 5.7 / 124.5 | 6.3 / 148.1 | 7.4 / 166.5 | 25 / 253 |

Because truncation moves a model along a curve, the paper reports three points on it. The first is the truncation with the best FID. The second is where the model's IS equals that of the validation data (166 at 128×128), a rough setting for "maximum variety at good objectness". The third is the maximum IS. The headline "IS 166.5, FID 7.4" is BigGAN-deep at the second point. The original BigGAN gives 166.3 and 9.6 there. At 256×256 and 512×512, BigGAN-deep reaches IS 232.5 / FID 8.1 and IS 241.5 / FID 11.5. On JFT-300M (292M images, 8.5K classes) the same techniques helped further, a 128-channel model was better still, and training stayed stable without heavy regularisation.

## Training collapse

Training stays healthy for a long time, then fails abruptly and completely. The authors tracked the top singular values of every weight. In G, most layers are well behaved. A few, usually the first (over-complete, non-convolutional) layer, have a top singular value $\sigma_0$ that grows throughout training and explodes at collapse. Directly regularising or clamping $\sigma_0$ (for example $W\leftarrow W-\max(0,\sigma_0-\sigma_{\text{clamp}})v_0u_0^\top$) stopped the growth but never prevented collapse. In D the spectra are noisy and jump at collapse, and D's loss heads to zero before a sharp rise. Penalising D's gradients with $R_1=\frac\gamma2\mathbb E_{p_D}\|\nabla D(x)\|^2$ at $\gamma=10$ made training stable but cut IS by 45%. At $\gamma=1$, the weakest penalty that avoided sudden collapse, IS dropped by 20%. D memorises: its accuracy on training images stays above 98% while it is 50–55% on validation images. Stability does not come from G or D alone but from their interaction. The paper's practical choice is to allow collapse and keep the checkpoint saved just before it.

## Worked example: what truncation does to a 120-d latent, and one orthogonal penalty

Truncated N(0, 1) per coordinate, for the thresholds of the paper's Figure 2(a), and the RMS norm √E‖z‖² of a 120-d z (computed in Python)

| τ | kept mass 2Φ(τ) − 1 | draws per coordinate | std of z_i | RMS ‖z‖ (120-d) |
| --- | --- | --- | --- | --- |
| ∞ (training) | 1 | 1 | 1 | 10.954 |
| 2 | 0.9545 | 1.048 | 0.8796 | 9.636 |
| 1 | 0.6827 | 1.465 | 0.5396 | 5.911 |
| 0.5 | 0.3829 | 2.611 | 0.2839 | 3.110 |
| 0.04 | 0.0319 | 31.34 | 0.0231 | 0.253 |

**Reading the table, then one regulariser**

1. The std of a truncated normal is $\sqrt{1-2\tau\varphi(\tau)/(2\Phi(\tau)-1)}$. At $\tau=0.5$ that is $0.284$, so a 120-d $z$ has RMS norm $\sqrt{120}\cdot0.284=3.11$. G has never seen a latent this short during training, since under $\mathcal N(0,I)$ the norm concentrates near $\sqrt{120}\approx10.95$.
2. Even $\tau=2$ resamples $4.55\%$ of coordinates. The chance that a 120-d draw needs no resampling at all is $0.9545^{120}=0.0037$.
3. Orthogonal penalty, filters as rows: $W=\begin{pmatrix}1&0&1\\0.5&1&0\end{pmatrix}$, $WW^\top=\begin{pmatrix}2&0.5\\0.5&1.25\end{pmatrix}$. The relaxed penalty sees only the off-diagonal $0.5$ twice: $R=10^{-4}\cdot(0.5^2+0.5^2)=5\cdot10^{-5}$. The full version also charges the norms: $10^{-4}\big((2-1)^2+0.25+(1.25-1)^2\big)=1.5625\cdot10^{-4}$.
4. Its gradient is $\partial R/\partial W=4\beta\,MW$ with $M$ the masked Gram matrix: $4\cdot10^{-4}\begin{pmatrix}0&0.5\\0.5&0\end{pmatrix}W=\begin{pmatrix}1&2&0\\2&0&2\end{pmatrix}\cdot10^{-4}$, the same as PyTorch autograd. Each filter is pushed away from the other's direction, and neither is shrunk toward unit norm.

```python
import torch, torch.nn as nn, torch.nn.functional as F
from torch.nn.utils import spectral_norm as SN

class CondBN(nn.Module):
    """BatchNorm whose gain and bias are linear projections of [shared class embedding, z chunk]."""
    def __init__(self, ch, cond_dim):
        super().__init__()
        self.bn = nn.BatchNorm2d(ch, affine=False)
        self.gain, self.bias = SN(nn.Linear(cond_dim, ch, bias=False)), SN(nn.Linear(cond_dim, ch, bias=False))
    def forward(self, x, c):
        return self.bn(x) * (1 + self.gain(c))[:, :, None, None] + self.bias(c)[:, :, None, None]  # gain centred at 1

class GBlock(nn.Module):
    def __init__(self, cin, cout, cond_dim):
        super().__init__()
        self.bn1, self.bn2 = CondBN(cin, cond_dim), CondBN(cout, cond_dim)
        self.c1, self.c2 = SN(nn.Conv2d(cin, cout, 3, padding=1)), SN(nn.Conv2d(cout, cout, 3, padding=1))
        self.skip = SN(nn.Conv2d(cin, cout, 1))
    def forward(self, x, c):
        h = F.interpolate(F.relu(self.bn1(x, c)), scale_factor=2)
        h = self.c2(F.relu(self.bn2(self.c1(h), c)))
        return h + self.skip(F.interpolate(x, scale_factor=2))

class Generator(nn.Module):                       # 32×32 toy: 3 up-blocks, z ∈ R^80 split into 4 chunks of 20
    def __init__(self, n_classes=10, ch=32, z_chunk=20, emb=128):
        super().__init__()
        self.ch, self.zc = ch, z_chunk
        self.embed = nn.Embedding(n_classes, emb)                 # ONE shared embedding for every CondBN
        self.fc = SN(nn.Linear(z_chunk + emb, 4 * 4 * 4 * ch))
        self.blocks = nn.ModuleList([GBlock(4 * ch, 4 * ch, z_chunk + emb), GBlock(4 * ch, 2 * ch, z_chunk + emb),
                                     GBlock(2 * ch, ch, z_chunk + emb)])
        self.out = nn.Sequential(nn.BatchNorm2d(ch), nn.ReLU(), SN(nn.Conv2d(ch, 3, 3, padding=1)), nn.Tanh())
    def forward(self, z, y):
        zs = z.split(self.zc, dim=1)                              # skip-z: one chunk per stage
        e = self.embed(y)
        h = self.fc(torch.cat([zs[0], e], 1)).view(-1, 4 * self.ch, 4, 4)
        for blk, zk in zip(self.blocks, zs[1:]):
            h = blk(h, torch.cat([zk, e], 1))
        return self.out(h)

class Discriminator(nn.Module):                   # projection discriminator: D(x, y) = ψ(φ(x)) + embed(y)·φ(x)
    def __init__(self, n_classes=10, ch=32):
        super().__init__()
        self.net = nn.Sequential(SN(nn.Conv2d(3, ch, 3, 2, 1)), nn.ReLU(), SN(nn.Conv2d(ch, 2 * ch, 3, 2, 1)), nn.ReLU(),
                                 SN(nn.Conv2d(2 * ch, 4 * ch, 3, 2, 1)), nn.ReLU())
        self.psi, self.embed = SN(nn.Linear(4 * ch, 1)), SN(nn.Embedding(n_classes, 4 * ch))
    def forward(self, x, y):
        h = self.net(x).sum(dim=(2, 3))                           # global sum pooling
        return self.psi(h).squeeze(1) + (self.embed(y) * h).sum(1)

def ortho_reg(model, beta=1e-4):                  # β‖WᵀW ⊙ (1 − I)‖²_F, filters as rows of W
    loss = 0.0
    for name, p in model.named_parameters():
        if p.dim() < 2 or "embed" in name:        # this sketch skips biases, gains and embeddings
            continue
        W = p.flatten(1)
        gram = W @ W.t()
        loss = loss + beta * (gram * (1 - torch.eye(W.size(0)))).pow(2).sum()
    return loss

def truncated_z(n, dim, tau):                     # resample every coordinate with |z_i| > τ
    z = torch.randn(n, dim)
    while (bad := z.abs() > tau).any():
        z[bad] = torch.randn(int(bad.sum()))
    return z

torch.manual_seed(0)
G, D = Generator(), Discriminator()
opt_g = torch.optim.Adam(G.parameters(), lr=5e-5, betas=(0.0, 0.999))    # BigGAN: lr_G 5e-5, lr_D 2e-4, β = (0, 0.999)
opt_d = torch.optim.Adam(D.parameters(), lr=2e-4, betas=(0.0, 0.999))
x, y = torch.randn(16, 3, 32, 32).clamp(-1, 1), torch.randint(0, 10, (16,))   # stand-in for a real batch
for _ in range(2):                                                         # two D steps per G step
    fake = G(torch.randn(16, 80), y).detach()
    loss_d = F.relu(1 - D(x, y)).mean() + F.relu(1 + D(fake, y)).mean()    # hinge loss
    opt_d.zero_grad(); loss_d.backward(); opt_d.step()
loss_g = -D(G(torch.randn(16, 80), y), y).mean() + ortho_reg(G)
opt_g.zero_grad(); loss_g.backward(); opt_g.step()

G.eval()
with torch.no_grad():
    z = truncated_z(8, 80, tau=0.5)
    img = G(z, torch.full((8,), 3))
print(img.shape, z.abs().max().item() <= 0.5)       # torch.Size([8, 3, 32, 32]) True
```

BigGAN's ingredients at toy scale: shared embedding + skip-z conditional BN, projection D, hinge loss, orthogonal regularisation, truncated sampling

> **Compare scores at a stated truncation:** With truncation a single model has a whole curve of (IS, FID) pairs, and its IS can exceed that of real data (the "max IS" column: 206 and 253 against 166 for the validation set). A number without its $\tau$ is not comparable. State the truncation, or report the curve. The toy figure shows the same effect: IS 5.705 at $\tau=0.04$, above the real data's 4.70, with FID at its worst.

> **Batch-norm statistics at sampling time:** Sampling with batch statistics makes outputs depend on the test batch size and device split, and the paper found this "extremely important". Running averages computed during training are also poor estimates for the EMA weights used at test time. BigGAN uses **standing statistics**: run G on about 100 batches of noise, store the aggregated means and variances, then sample with those. In the code above, `G.eval()` uses the training running averages, which is the simple version of the same idea.

> **Why truncation needs a smooth G:** Truncation moves every latent into a small region around $z=0$ that held only a few percent of the training latents. If G is well behaved only on typical latents, that region can hold artifacts, and truncation then makes every sample bad (the ill-conditioned G in the figure: IS 5.19 → 3.54, FID → 1.48). Orthogonal regularisation, like spectral norm, conditions G's Jacobian so that nearby latents give nearby images everywhere, including near the origin.

> **After BigGAN:** BigGAN set the bar for class-conditional ImageNet synthesis until **Diffusion models** with classifier guidance overtook it on FID (Dhariwal and Nichol, 2021). Their guidance scale plays the same fidelity–variety role as $\tau$. For unconditional faces, **StyleGAN** brought the style-based generator and its own truncation in $\mathcal W$.

**Strengths**

- Large batches and width give large, measurable gains (IS 52.5 → 99.3 in the ablation; 166.5 with BigGAN-deep and truncation)
- Truncation gives post-hoc control of fidelity versus variety without retraining
- Shared embeddings and skip-z reduce cost and speed up training (37% and 18% fewer iterations)
- Pretrained models became a standard benchmark and a source of class-conditional images

**Limitations**

- Training collapse is not solved: early stopping is required, and regularising D enough to be stable costs 20–45% of IS
- Huge compute: 128–512 TPU v3 cores per model, and batch-norm statistics shared across devices
- Not every model is amenable to truncation, even with orthogonal regularisation (60%)
- Class leakage and weaker results on classes with large-scale structure (crowds) than on texture-defined ones (dogs)

**Question:** As the truncation threshold τ is lowered from ∞ toward 0 on a model that is amenable to truncation, what happens to IS and FID?

- A. Both improve monotonically
- B. IS rises steadily; FID first improves, then degrades sharply as variety vanishes
- C. IS falls and FID improves
- D. Neither changes, because G was trained on N(0, I)

**Answer:** B. IS rises steadily; FID first improves, then degrades sharply as variety vanishes — IS rewards confident, class-typical samples and does not penalise missing variety within a class, so it behaves like precision. FID compares whole feature distributions, so it rewards the precision gain at first but punishes the lost variety as $\tau\to0$. The toy figure shows the same shape: best FID at $\tau=1.6$, while IS keeps rising down to $\tau=0.04$.

**Question:** What does BigGAN's version of orthogonal regularisation penalise?

- A. ‖WᵀW ⊙ (1 − I)‖²_F: only pairwise dot products between filters (the diagonal is masked out), with β = 10⁻⁴
- B. ‖WᵀW − I‖²_F: both filter norms and their pairwise dot products
- C. The largest singular value of each weight
- D. The gradient norm of D at real images

**Answer:** A. ‖WᵀW ⊙ (1 − I)‖²_F: only pairwise dot products between filters (the diagonal is masked out), with β = 10⁻⁴ — The full orthogonality penalty is known to be too limiting, so BigGAN removes the diagonal. Filters are pushed toward mutual orthogonality, but their norms are free. The largest singular value is what spectral norm controls, and the gradient norm at real images is the $R_1$ penalty the authors tested on D.

**Key points**

- BigGAN = SA-GAN backbone + hinge loss + spectral norm in G and D + batch 2048, with a shared class embedding and skip-z feeding $[e(y);z_\ell]$ to every conditional batch norm, and a projection discriminator.
- Truncation trick: sample $z_i\sim\mathcal N(0,1)$ restricted to $|z_i|\le\tau$. Lower $\tau$ raises IS (fidelity) and, past an optimum, worsens FID (variety). Orthogonal regularisation $\beta\|W^\top W\odot(\mathbf 1-I)\|_F^2$, $\beta=10^{-4}$, makes G amenable to it.
- ImageNet 128²: IS 166.5 / FID 7.4 (BigGAN-deep, at the truncation that matches validation IS) vs 52.52 / 18.65 for SA-GAN; batch 256 → 2048 alone gave +46% IS.
- Training collapse: G's first-layer spectral norm explodes and D memorises its training set (>98% vs 50–55% validation accuracy); strong D regularisation stabilises training but costs 20–45% IS, so models are early-stopped.
- Optimisation: Adam $\beta=(0,0.999)$, lr $2\cdot10^{-4}$ (D) and $5\cdot10^{-5}$ (G), two D steps per G step, EMA of G weights with decay $0.9999$.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
