Definition
Generative adversarial network (GAN) A GAN is a pair of networks trained against each other (Goodfellow et al., "Generative Adversarial Nets", NeurIPS 2014). The generator maps a noise vector to a sample ; the distribution of these samples is . The discriminator outputs the probability that came from the data rather than from . is trained to classify real against fake, and is trained to make wrong. No density is written down. is an implicit model: you can sample from it, but you cannot evaluate .
Before 2014, deep generative models were mostly trained by maximum likelihood. Boltzmann machines need Markov chains to estimate the gradient of the partition function, and the Variational autoencoder (VAE) (Kingma & Welling, 2014) maximises a lower bound on the likelihood. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville and Yoshua Bengio proposed a different signal. A classifier learns to separate real from generated samples, and the generator follows the classifier's gradient. Training needs only backpropagation through both networks: no Markov chains, no inference network, no approximate likelihood. The paper's analogy is counterfeiters (G) against the police (D), each improving until the counterfeits cannot be told from real currency.
The minimax game
- probability that is real; is the discriminator's logit
- generated sample; is drawn from a fixed simple prior (uniform or Gaussian) and fed to the input layer
- minimises binary cross-entropy with label 1 on real and 0 on fake samples, which is the same as maximising
- only affects the second term; it wants to be large
The optimal discriminator. For fixed , write and maximise inside the integral, point by point. The function peaks at . Substituting the optimum gives the generator's effective objective :
- Jensen–Shannon divergence , ; between 0 and
- the global minimum of , reached only at , where
- the optimal logit is the log density ratio
This is the paper's Theorem 1: the global minimum of the training criterion is reached if and only if . Proposition 2 adds a convergence result. If and have enough capacity, is trained to its optimum at every step, and is updated to improve , then converges to . The argument works in the space of densities, where is convex in . Real networks optimise parameters, not densities, so the guarantee does not carry over. The full derivation, and the alternative losses that came later, are in Adversarial loss.
The non-saturating generator loss
Early in training rejects the fakes easily, , and is flat there. Goodfellow et al. noted this and trained to maximise instead. The fixed point of the dynamics is unchanged, and the gradients are much stronger early on. In terms of the logit :
- at the minimax factor is 0.01, the non-saturating one 0.99, 99 times larger
- binary cross-entropy of the fakes with the real label; in code
BCEWithLogits(D(G(z)), ones)
The two losses weight fakes oppositely. Minimax pushes hardest on fakes that already fool . The non-saturating loss pushes hardest on fakes that rejects. With , the non-saturating gradient equals the gradient of (Arjovsky & Bottou, 2017). This reverse KL heavily penalises samples where there is no data and barely penalises missing modes, one explanation for mode collapse. Almost every GAN in practice uses the non-saturating loss or a later variant (hinge, Wasserstein).
Alternating training
Algorithm 1 of the paper (minibatch stochastic gradient)
- 1Repeat times: draw noise samples and data samples , and ascend . The fakes are treated as constants (
detach()in PyTorch). - 2Draw fresh noise samples and descend , or ascend for the non-saturating version. The gradient flows through into , but only 's parameters are updated.
- 3The paper used , the cheapest option, with momentum SGD. Later practice is Adam with (from Deep convolutional GAN (DCGAN)), , and sometimes separate learning rates for and (the two time-scale update rule of Heusel et al., 2017).
- 4Keeping near optimal makes 's gradient follow . A that is too strong saturates the minimax loss. A that is too weak gives a meaningless signal. The paper also warns that must not be trained too much without updating , to avoid the "Helvetica scenario", in which maps many values of to the same .
The original paper
- Venue
- NeurIPS (then NIPS) 2014; arXiv 1406.2661
- Models
- multilayer perceptrons. used a mixture of rectifier and sigmoid units; used maxout units with dropout. Noise entered only at the bottom layer of
- Datasets
- MNIST, Toronto Face Database (TFD), CIFAR-10
- Evaluation
- Gaussian Parzen-window log-likelihood of test data under 's samples: 225 ± 2 on MNIST, 2057 ± 26 on TFD. The authors note that this estimator has high variance and does not work well in high dimensions
- Stated disadvantages
- no explicit ; must stay synchronised with
Mode collapse and other failure modes
The theory assumes is optimal at every step. In practice both networks take simultaneous or alternating small steps. That is a search for a saddle point, and gradient descent on a game need not converge. Three things go wrong. Oscillation: chases 's decision boundary around the data without settling. Mode collapse: maps most to a few outputs that currently fool . When adapts, moves to another mode, because nothing in a single-sample loss rewards diversity. Vanishing gradients: separates real from fake almost perfectly. Salimans et al. (2016, "Improved Techniques for Training GANs") proposed feature matching, minibatch discrimination, historical averaging and one-sided label smoothing. Later the main fixes became new losses (Wasserstein GAN (WGAN), WGAN-GP) and Lipschitz control of (spectral normalisation).
Worked example: D*, JS and the two generator gradients
Take three points with and a generator with . The generator puts most of its mass on , where there is little data.
| x | p_data | p_g | D*(x) | logit a = log p_data/p_g | minimax |∂L/∂a| = D | NS |∂L/∂a| = 1 − D |
|---|---|---|---|---|---|---|
| x₁ | 0.45 | 0.10 | 0.8182 | 1.5041 | 0.8182 | 0.1818 |
| x₂ | 0.45 | 0.30 | 0.6000 | 0.4055 | 0.6000 | 0.4000 |
| x₃ | 0.10 | 0.60 | 0.1429 | −1.7918 | 0.1429 | 0.8571 |
Checking the theory on these numbers
- 1.
- 2 nats (the maximum is ), and , the same value as .
- 3So the discriminator's BCE loss at its optimum is . At equilibrium it would be , because costs on each half. A D loss below 1.386 means can still tell the two apart.
- 4Under , the average minimax factor is and the non-saturating one . The difference is not just size. Minimax concentrates its push on (0.8182), where fakes are already rare and good. The non-saturating loss concentrates it on (0.8571), the region most needs to leave.
Evaluating a GAN
A GAN gives no likelihood, so it is evaluated with sample statistics computed from a pretrained classifier. The Inception Score (Salimans et al., 2016) is , using the label distribution of an ImageNet Inception network. It is high when each sample is classified confidently (sharp images) and the predicted classes are spread out (diversity). It never looks at real data, and it is blind to diversity within a class. The Fréchet Inception Distance (Heusel et al., 2017) fits Gaussians to the 2048-d pool features of real and generated images and compares them:
- mean and covariance of Inception-v3 pool features of real images
- the same for generated images; lower FID is better, 0 for identical Gaussians
FID drops when mode collapse is fixed, which IS does not reliably do. Both scores are biased by the sample size: FID is usually reported on 50,000 samples, and numbers computed with different sample counts or Inception implementations are not comparable. Precision and recall for distributions (Sajjadi et al., 2018; Kynkäänniemi et al., 2019) split the error into quality and coverage, the two quantities the figure tracks as "quality" and "modes covered".
1import math, torch, torch.nn as nn2torch.manual_seed(0)34def eight_gaussians(n, sigma=0.05):5 k = torch.randint(0, 8, (n,))6 centers = torch.stack([torch.cos(k * math.pi / 4), torch.sin(k * math.pi / 4)], 1)7 return centers + sigma * torch.randn(n, 2)89def mlp(i, o, h=128):10 return nn.Sequential(nn.Linear(i, h), nn.ReLU(), nn.Linear(h, h), nn.ReLU(), nn.Linear(h, o))1112nz = 813G = mlp(nz, 2) # z -> x14D = mlp(2, 1) # x -> logit a(x); D(x) = sigmoid(a)15opt_g = torch.optim.Adam(G.parameters(), lr=2e-4, betas=(0.5, 0.999))16opt_d = torch.optim.Adam(D.parameters(), lr=2e-4, betas=(0.5, 0.999))17bce = nn.BCEWithLogitsLoss()18B, k = 256, 11920for it in range(3001):21 for _ in range(k): # discriminator: real -> 1, fake -> 022 x = eight_gaussians(B)23 fake = G(torch.randn(B, nz)).detach() # no gradient into G here24 a_real, a_fake = D(x), D(fake)25 loss_d = bce(a_real, torch.ones_like(a_real)) + bce(a_fake, torch.zeros_like(a_fake))26 opt_d.zero_grad(); loss_d.backward(); opt_d.step()27 a_fake = D(G(torch.randn(B, nz))) # generator: non-saturating loss28 loss_g = bce(a_fake, torch.ones_like(a_fake)) # = mean of -log D(G(z))29 # minimax alternative: loss_g = -bce(a_fake, torch.zeros_like(a_fake)) (= mean log(1 - D(G(z))))30 opt_g.zero_grad(); loss_g.backward(); opt_g.step()31 if it % 1000 == 0:32 print(it, f"loss_d {loss_d.item():.3f} loss_g {loss_g.item():.3f}") # loss_d near log 4 = 1.386 is not "converged"3334with torch.no_grad():35 s = G(torch.randn(2000, nz))36 print(s.shape) # torch.Size([2000, 2])37 c = torch.stack([torch.cos(torch.arange(8) * math.pi / 4), torch.sin(torch.arange(8) * math.pi / 4)], 1)38 d = torch.cdist(s, c) # (2000, 8) distance to each mode39 near = d.min(1).values < 0.15 # within 3 sigma of some mode40 print(near.float().mean().item(), torch.bincount(d.argmin(1)[near], minlength=8).tolist()) # quality, fakes per modeUse when
- +Sampling is one forward pass of G: fast, and any differentiable architecture works
- +No likelihood, no Markov chain, no inference network: training is plain backpropagation
- +Learned loss: samples are sharp because blurry averages are easy for to reject
- +Theory: with an optimal the game minimises JS, with a unique optimum at
Avoid when
- −A saddle-point problem: oscillation and divergence are common, and hyperparameters are fragile
- −Mode collapse: diversity is not directly rewarded
- −No density and no encoder, so likelihoods cannot be evaluated and real inputs cannot be mapped to without extra machinery
- −Losses do not track progress; evaluation relies on Inception-based scores with known biases
Check yourself
For a fixed generator, what is the discriminator that maximises V(D, G)?
Check yourself
Early in training D(G(z)) ≈ 0.001. Why do practitioners use the non-saturating loss −log D(G(z))?
Remember
- ✓GAN game: , with trained as a real/fake BCE classifier.
- ✓Optimal gives , minimised only at .
- ✓Non-saturating generator loss : same fixed point, gradient factor instead of , so it does not vanish when rejects the fakes.
- ✓Training alternates D steps (paper: ) with one G step. It is a saddle-point search, prone to oscillation and mode collapse.
- ✓Evaluate with sample statistics: Inception Score (sharpness and class spread) and FID (Fréchet distance of Inception features, lower is better).