Generative adversarial network (GAN)

A generator G maps noise z to samples and a discriminator D learns to tell them from real data. They play min_G max_D E[log D(x)] + E[log(1 − D(G(z)))]. For a fixed G the best D is p_data/(p_data + p_g), so G then minimises the Jensen–Shannon divergence. Its unique optimum is p_g = p_data.

15 min read1370 wordscore0 viewsupdated 2026-10-12

Definition

Generative adversarial network (GAN) A GAN is a pair of networks trained against each other (Goodfellow et al., "Generative Adversarial Nets", NeurIPS 2014). The generator GG maps a noise vector z∼p(z)z\sim p(z) to a sample G(z)G(z); the distribution of these samples is pgp_g. The discriminator DD outputs the probability D(x)D(x) that xx came from the data rather than from GG. DD is trained to classify real against fake, and GG is trained to make DD wrong. No density is written down. GG is an implicit model: you can sample from it, but you cannot evaluate pg(x)p_g(x).

Before 2014, deep generative models were mostly trained by maximum likelihood. Boltzmann machines need Markov chains to estimate the gradient of the partition function, and the Variational autoencoder (VAE) (Kingma & Welling, 2014) maximises a lower bound on the likelihood. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville and Yoshua Bengio proposed a different signal. A classifier learns to separate real from generated samples, and the generator follows the classifier's gradient. Training needs only backpropagation through both networks: no Markov chains, no inference network, no approximate likelihood. The paper's analogy is counterfeiters (G) against the police (D), each improving until the counterfeits cannot be told from real currency.

The minimax game

min⁡Gmax⁡D V(D,G)=Ex∼pdata[log⁡D(x)]+Ez∼p(z)[log⁡(1−D(G(z)))]\min_G\max_D\ V(D,G)=\mathbb E_{x\sim p_{\text{data}}}\big[\log D(x)\big]+\mathbb E_{z\sim p(z)}\big[\log\big(1-D(G(z))\big)\big]
D(x)=σ(a(x))D(x)=\sigma(a(x))
probability that xx is real; a(x)a(x) is the discriminator's logit
G(z)G(z)
generated sample; zz is drawn from a fixed simple prior (uniform or Gaussian) and fed to the input layer
max⁡D\max_D
DD minimises binary cross-entropy with label 1 on real and 0 on fake samples, which is the same as maximising VV
min⁡G\min_G
GG only affects the second term; it wants D(G(z))D(G(z)) to be large

The optimal discriminator. For fixed GG, write V=∫pdata(x)log⁡D(x)+pg(x)log⁡(1−D(x)) dxV=\int p_{\text{data}}(x)\log D(x)+p_g(x)\log(1-D(x))\,dx and maximise inside the integral, point by point. The function y↦alog⁡y+blog⁡(1−y)y\mapsto a\log y+b\log(1-y) peaks at y=a/(a+b)y=a/(a+b). Substituting the optimum gives the generator's effective objective C(G)C(G):

DG∗(x)=pdata(x)pdata(x)+pg(x),C(G)=V(DG∗,G)=−log⁡4+2 JS(pdata ∥ pg)D^*_G(x)=\frac{p_{\text{data}}(x)}{p_{\text{data}}(x)+p_g(x)},\qquad C(G)=V(D^*_G,G)=-\log4+2\,\mathrm{JS}\big(p_{\text{data}}\,\|\,p_g\big)
JS\mathrm{JS}
Jensen–Shannon divergence 12KL(pdata∥m)+12KL(pg∥m)\tfrac12\mathrm{KL}(p_{\text{data}}\|m)+\tfrac12\mathrm{KL}(p_g\|m), m=12(pdata+pg)m=\tfrac12(p_{\text{data}}+p_g); between 0 and log⁡2\log 2
−log⁡4-\log4
the global minimum of C(G)C(G), reached only at pg=pdatap_g=p_{\text{data}}, where D∗≡12D^*\equiv\tfrac12
log⁡D∗1−D∗\log\frac{D^*}{1-D^*}
the optimal logit is the log density ratio log⁡pdata(x)−log⁡pg(x)\log p_{\text{data}}(x)-\log p_g(x)

This is the paper's Theorem 1: the global minimum of the training criterion is reached if and only if pg=pdatap_g=p_{\text{data}}. Proposition 2 adds a convergence result. If GG and DD have enough capacity, DD is trained to its optimum at every step, and pgp_g is updated to improve C(G)C(G), then pgp_g converges to pdatap_{\text{data}}. The argument works in the space of densities, where CC is convex in pgp_g. Real networks optimise parameters, not densities, so the guarantee does not carry over. The full derivation, and the alternative losses that came later, are in Adversarial loss.

The non-saturating generator loss

Early in training DD rejects the fakes easily, D(G(z))≈0D(G(z))\approx0, and log⁡(1−D(G(z)))\log(1-D(G(z))) is flat there. Goodfellow et al. noted this and trained GG to maximise log⁡D(G(z))\log D(G(z)) instead. The fixed point of the dynamics is unchanged, and the gradients are much stronger early on. In terms of the logit a=a(G(z))a=a(G(z)):

LGminimax=Ezlog⁡(1−σ(a)),  ∂L∂a=−σ(a)LGNS=−Ezlog⁡σ(a),  ∂L∂a=σ(a)−1\mathcal L_G^{\text{minimax}}=\mathbb E_z\log\big(1-\sigma(a)\big),\ \ \frac{\partial\mathcal L}{\partial a}=-\sigma(a)\qquad\qquad \mathcal L_G^{\text{NS}}=-\mathbb E_z\log\sigma(a),\ \ \frac{\partial\mathcal L}{\partial a}=\sigma(a)-1
σ(a)=D(G(z))\sigma(a)=D(G(z))
at D(G(z))=0.01D(G(z))=0.01 the minimax factor is 0.01, the non-saturating one 0.99, 99 times larger
LGNS\mathcal L_G^{\text{NS}}
binary cross-entropy of the fakes with the real label; in code BCEWithLogits(D(G(z)), ones)

The two losses weight fakes oppositely. Minimax pushes hardest on fakes that already fool DD. The non-saturating loss pushes hardest on fakes that DD rejects. With D=D∗D=D^*, the non-saturating gradient equals the gradient of KL(pg∥pdata)−2 JS\mathrm{KL}(p_g\|p_{\text{data}})-2\,\mathrm{JS} (Arjovsky & Bottou, 2017). This reverse KL heavily penalises samples where there is no data and barely penalises missing modes, one explanation for mode collapse. Almost every GAN in practice uses the non-saturating loss or a later variant (hinge, Wasserstein).

Alternating training

Algorithm 1 of the paper (minibatch stochastic gradient)

  1. 1Repeat kk times: draw mm noise samples z(i)z^{(i)} and mm data samples x(i)x^{(i)}, and ascend ∇θd1m∑i[log⁡D(x(i))+log⁡(1−D(G(z(i))))]\nabla_{\theta_d}\frac1m\sum_i\big[\log D(x^{(i)})+\log(1-D(G(z^{(i)})))\big]. The fakes are treated as constants (detach() in PyTorch).
  2. 2Draw mm fresh noise samples and descend ∇θg1m∑ilog⁡(1−D(G(z(i))))\nabla_{\theta_g}\frac1m\sum_i\log(1-D(G(z^{(i)}))), or ascend log⁡D(G(z(i)))\log D(G(z^{(i)})) for the non-saturating version. The gradient flows through DD into GG, but only GG's parameters are updated.
  3. 3The paper used k=1k=1, the cheapest option, with momentum SGD. Later practice is Adam with β1=0.5\beta_1=0.5 (from Deep convolutional GAN (DCGAN)), k=1k=1, and sometimes separate learning rates for DD and GG (the two time-scale update rule of Heusel et al., 2017).
  4. 4Keeping DD near optimal makes GG's gradient follow C(G)C(G). A DD that is too strong saturates the minimax loss. A DD that is too weak gives GG a meaningless signal. The paper also warns that GG must not be trained too much without updating DD, to avoid the "Helvetica scenario", in which GG maps many values of zz to the same xx.

The original paper

Venue
NeurIPS (then NIPS) 2014; arXiv 1406.2661
Models
multilayer perceptrons. GG used a mixture of rectifier and sigmoid units; DD used maxout units with dropout. Noise entered only at the bottom layer of GG
Datasets
MNIST, Toronto Face Database (TFD), CIFAR-10
Evaluation
Gaussian Parzen-window log-likelihood of test data under GG's samples: 225 ± 2 on MNIST, 2057 ± 26 on TFD. The authors note that this estimator has high variance and does not work well in high dimensions
Stated disadvantages
no explicit pg(x)p_g(x); DD must stay synchronised with GG

Mode collapse and other failure modes

The theory assumes DD is optimal at every step. In practice both networks take simultaneous or alternating small steps. That is a search for a saddle point, and gradient descent on a game need not converge. Three things go wrong. Oscillation: GG chases DD's decision boundary around the data without settling. Mode collapse: GG maps most zz to a few outputs that currently fool DD. When DD adapts, GG moves to another mode, because nothing in a single-sample loss rewards diversity. Vanishing gradients: DD separates real from fake almost perfectly. Salimans et al. (2016, "Improved Techniques for Training GANs") proposed feature matching, minibatch discrimination, historical averaging and one-sided label smoothing. Later the main fixes became new losses (Wasserstein GAN (WGAN), WGAN-GP) and Lipschitz control of DD (spectral normalisation).

D(x) field: blue D > ½ (looks real), red D < ½ · ○ real · ● G(z)8 modes, σ = 0.05 · dashed = 3σ · 128 real, 128 fakeiteration0k (D steps per G step)3mean D(x), real0.464mean D(G(z)), fake0.482D loss (BCE)1.622G loss −log D(G(z))0.730mean |∂L_G/∂a| on fakes0.518modes covered · quality0/8 · 0.00fakes within 3σ per mode (ideal 16, covered ≥ 4)0m00m10m20m30m40m50m60m7history to 1500: modes/8 · quality · D(G(z))10D step ×3: minimise −log D(x) − log(1 − D(G(z))) (Adam, lr 0.01, β₁ = 0.5)G step: minimise −log D(G(z)) (non-saturating), ∂/∂a = σ(a) − 1a = D logit on a fake; mean |∂L_G/∂a| = 0.518 = mean (1 − D(G(z))): large while D rejects fakesat the optimum p_g = p_data the best D is ½ everywhere and the D loss is log 4 = 1.386G: z ~ N(0, I₂) → 32 tanh → R² · D: R² → 32 tanh → logit · full batch, fixed seedstraining, 5 iterations per tick
G loss
G startslr
Changing a setting restarts training from the same seeds.
fig.A real GAN, trained live, on 8 Gaussians (σ = 0.05) on a unit ring. G is a 2→32→2 tanh MLP and D a 2→32→1 tanh MLP. Full batches of 128, Adam with β₁ = 0.5, fixed seeds. The shading is D(x). Default settings: non-saturating loss, k = 3 D steps per G step, lr 0.01. At iteration 0, mean D(G(z)) = 0.482 and the D loss is 1.622. All 8 modes hold at least 4 fakes by iteration 280. By iteration 1000, 88% of fakes lie within 3σ of a mode, and D gives 0.554 to reals and 0.465 to fakes, close to the ½ of the equilibrium. Then the game slips. At iteration 1100, coverage drops to 3 modes and D(real) jumps to 0.754. G recovers to 7 modes, but mode 4 now holds 35 of the 128 fakes and mode 5 none, and this partial collapse lasts to iteration 1500. With lr 0.005, coverage reaches 8 at iteration 480 and never drops below 7 afterwards. With lr 0.02 it never exceeds 5 modes, and at iteration 1500 modes 0 and 4 hold 40 fakes each. Start G far away at (2.2, 2.2): D(G(z)) = 0.130, so the minimax generator's logit gradient is 0.130 against 0.870 for the non-saturating loss. Adam rescales steps, so with k = 3 both still cover all 8 modes (iteration 250 for minimax, 260 for non-saturating).

Worked example: D*, JS and the two generator gradients

Take three points with pdata=(0.45, 0.45, 0.10)p_{\text{data}}=(0.45,\,0.45,\,0.10) and a generator with pg=(0.10, 0.30, 0.60)p_g=(0.10,\,0.30,\,0.60). The generator puts most of its mass on x3x_3, where there is little data.

xp_datap_gD*(x)logit a = log p_data/p_gminimax |∂L/∂a| = DNS |∂L/∂a| = 1 − D
x₁0.450.100.81821.50410.81820.1818
x₂0.450.300.60000.40550.60000.4000
x₃0.100.600.1429−1.79180.14290.8571
Optimal discriminator and per-sample generator gradient factors (computed in Python)

Checking the theory on these numbers

  1. 1V(D∗,G)=∑xpdatalog⁡D∗+pglog⁡(1−D∗)=−1.0526V(D^*,G)=\sum_x p_{\text{data}}\log D^*+p_g\log(1-D^*)=-1.0526.
  2. 2JS(pdata∥pg)=0.1668\mathrm{JS}(p_{\text{data}}\|p_g)=0.1668 nats (the maximum is log⁡2=0.6931\log2=0.6931), and −log⁡4+2(0.1668)=−1.0526-\log4+2(0.1668)=-1.0526, the same value as V(D∗,G)V(D^*,G).
  3. 3So the discriminator's BCE loss at its optimum is 1.05261.0526. At equilibrium it would be log⁡4=1.3863\log4=1.3863, because D≡12D\equiv\tfrac12 costs log⁡2\log2 on each half. A D loss below 1.386 means DD can still tell the two apart.
  4. 4Under pgp_g, the average minimax factor is 0.34750.3475 and the non-saturating one 0.65250.6525. The difference is not just size. Minimax concentrates its push on x1x_1 (0.8182), where fakes are already rare and good. The non-saturating loss concentrates it on x3x_3 (0.8571), the region GG most needs to leave.

Evaluating a GAN

A GAN gives no likelihood, so it is evaluated with sample statistics computed from a pretrained classifier. The Inception Score (Salimans et al., 2016) is IS=exp⁡(Ex KL(p(y∣x) ∥ p(y)))\mathrm{IS}=\exp\big(\mathbb E_x\,\mathrm{KL}(p(y\mid x)\,\|\,p(y))\big), using the label distribution of an ImageNet Inception network. It is high when each sample is classified confidently (sharp images) and the predicted classes are spread out (diversity). It never looks at real data, and it is blind to diversity within a class. The Fréchet Inception Distance (Heusel et al., 2017) fits Gaussians to the 2048-d pool features of real and generated images and compares them:

FID=∥μr−μg∥22+Tr(Σr+Σg−2(ΣrΣg)1/2)\mathrm{FID}=\big\|\mu_r-\mu_g\big\|_2^2+\mathrm{Tr}\Big(\Sigma_r+\Sigma_g-2\big(\Sigma_r\Sigma_g\big)^{1/2}\Big)
μr,Σr\mu_r,\Sigma_r
mean and covariance of Inception-v3 pool features of real images
μg,Σg\mu_g,\Sigma_g
the same for generated images; lower FID is better, 0 for identical Gaussians

FID drops when mode collapse is fixed, which IS does not reliably do. Both scores are biased by the sample size: FID is usually reported on 50,000 samples, and numbers computed with different sample counts or Inception implementations are not comparable. Precision and recall for distributions (Sajjadi et al., 2018; Kynkäänniemi et al., 2019) split the error into quality and coverage, the two quantities the figure tracks as "quality" and "modes covered".

python
1import math, torch, torch.nn as nn
2torch.manual_seed(0)
3
4def eight_gaussians(n, sigma=0.05):
5 k = torch.randint(0, 8, (n,))
6 centers = torch.stack([torch.cos(k * math.pi / 4), torch.sin(k * math.pi / 4)], 1)
7 return centers + sigma * torch.randn(n, 2)
8
9def mlp(i, o, h=128):
10 return nn.Sequential(nn.Linear(i, h), nn.ReLU(), nn.Linear(h, h), nn.ReLU(), nn.Linear(h, o))
11
12nz = 8
13G = mlp(nz, 2) # z -> x
14D = mlp(2, 1) # x -> logit a(x); D(x) = sigmoid(a)
15opt_g = torch.optim.Adam(G.parameters(), lr=2e-4, betas=(0.5, 0.999))
16opt_d = torch.optim.Adam(D.parameters(), lr=2e-4, betas=(0.5, 0.999))
17bce = nn.BCEWithLogitsLoss()
18B, k = 256, 1
19
20for it in range(3001):
21 for _ in range(k): # discriminator: real -> 1, fake -> 0
22 x = eight_gaussians(B)
23 fake = G(torch.randn(B, nz)).detach() # no gradient into G here
24 a_real, a_fake = D(x), D(fake)
25 loss_d = bce(a_real, torch.ones_like(a_real)) + bce(a_fake, torch.zeros_like(a_fake))
26 opt_d.zero_grad(); loss_d.backward(); opt_d.step()
27 a_fake = D(G(torch.randn(B, nz))) # generator: non-saturating loss
28 loss_g = bce(a_fake, torch.ones_like(a_fake)) # = mean of -log D(G(z))
29 # minimax alternative: loss_g = -bce(a_fake, torch.zeros_like(a_fake)) (= mean log(1 - D(G(z))))
30 opt_g.zero_grad(); loss_g.backward(); opt_g.step()
31 if it % 1000 == 0:
32 print(it, f"loss_d {loss_d.item():.3f} loss_g {loss_g.item():.3f}") # loss_d near log 4 = 1.386 is not "converged"
33
34with torch.no_grad():
35 s = G(torch.randn(2000, nz))
36 print(s.shape) # torch.Size([2000, 2])
37 c = torch.stack([torch.cos(torch.arange(8) * math.pi / 4), torch.sin(torch.arange(8) * math.pi / 4)], 1)
38 d = torch.cdist(s, c) # (2000, 8) distance to each mode
39 near = d.min(1).values < 0.15 # within 3 sigma of some mode
40 print(near.float().mean().item(), torch.bincount(d.argmin(1)[near], minlength=8).tolist()) # quality, fakes per mode
A GAN on 8 Gaussians: alternating D and G steps with the non-saturating loss

Use when

  • +Sampling is one forward pass of G: fast, and any differentiable architecture works
  • +No likelihood, no Markov chain, no inference network: training is plain backpropagation
  • +Learned loss: samples are sharp because blurry averages are easy for DD to reject
  • +Theory: with an optimal DD the game minimises JS, with a unique optimum at pg=pdatap_g=p_{\text{data}}

Avoid when

  • −A saddle-point problem: oscillation and divergence are common, and hyperparameters are fragile
  • −Mode collapse: diversity is not directly rewarded
  • −No density pg(x)p_g(x) and no encoder, so likelihoods cannot be evaluated and real inputs cannot be mapped to zz without extra machinery
  • −Losses do not track progress; evaluation relies on Inception-based scores with known biases

Check yourself

For a fixed generator, what is the discriminator that maximises V(D, G)?

Check yourself

Early in training D(G(z)) ≈ 0.001. Why do practitioners use the non-saturating loss −log D(G(z))?

Remember

  • ✓GAN game: min⁡Gmax⁡D Epdata[log⁡D(x)]+Ez[log⁡(1−D(G(z)))]\min_G\max_D\ \mathbb E_{p_{\text{data}}}[\log D(x)]+\mathbb E_{z}[\log(1-D(G(z)))], with DD trained as a real/fake BCE classifier.
  • ✓Optimal D∗=pdata/(pdata+pg)D^*=p_{\text{data}}/(p_{\text{data}}+p_g) gives C(G)=−log⁡4+2 JS(pdata∥pg)C(G)=-\log4+2\,\mathrm{JS}(p_{\text{data}}\|p_g), minimised only at pg=pdatap_g=p_{\text{data}}.
  • ✓Non-saturating generator loss −log⁡D(G(z))-\log D(G(z)): same fixed point, gradient factor 1−D1-D instead of DD, so it does not vanish when DD rejects the fakes.
  • ✓Training alternates kk D steps (paper: k=1k=1) with one G step. It is a saddle-point search, prone to oscillation and mode collapse.
  • ✓Evaluate with sample statistics: Inception Score (sharpness and class spread) and FID (Fréchet distance of Inception features, lower is better).
← back to Deep Learning roadmap