# Generative adversarial network (GAN)

> A generator G maps noise z to samples and a discriminator D learns to tell them from real data. They play min_G max_D E[log D(x)] + E[log(1 − D(G(z)))]. For a fixed G the best D is p_data/(p_data + p_g), so G then minimises the Jensen–Shannon divergence. Its unique optimum is p_g = p_data.

Source: https://www.mlfoundry.in/learn/generative-adversarial-network-gan  
Section: Deep Learning › Generative Deep Learning Algorithms  
Level: core · 15 min read · updated 2026-10-12

**Generative adversarial network (GAN):** A GAN is a pair of networks trained against each other (Goodfellow et al., "Generative Adversarial Nets", NeurIPS 2014). The **generator** $G$ maps a noise vector $z\sim p(z)$ to a sample $G(z)$; the distribution of these samples is $p_g$. The **discriminator** $D$ outputs the probability $D(x)$ that $x$ came from the data rather than from $G$. $D$ is trained to classify real against fake, and $G$ is trained to make $D$ wrong. No density is written down. $G$ is an **implicit** model: you can sample from it, but you cannot evaluate $p_g(x)$.

Before 2014, deep generative models were mostly trained by maximum likelihood. Boltzmann machines need Markov chains to estimate the gradient of the partition function, and the **Variational autoencoder (VAE)** (Kingma & Welling, 2014) maximises a lower bound on the likelihood. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville and Yoshua Bengio proposed a different signal. A classifier learns to separate real from generated samples, and the generator follows the classifier's gradient. Training needs only backpropagation through both networks: no Markov chains, no inference network, no approximate likelihood. The paper's analogy is counterfeiters (G) against the police (D), each improving until the counterfeits cannot be told from real currency.

## The minimax game

$$
\min_G\max_D\ V(D,G)=\mathbb E_{x\sim p_{\text{data}}}\big[\log D(x)\big]+\mathbb E_{z\sim p(z)}\big[\log\big(1-D(G(z))\big)\big]
$$

- $D(x)=\sigma(a(x))$: probability that $x$ is real; $a(x)$ is the discriminator's logit
- $G(z)$: generated sample; $z$ is drawn from a fixed simple prior (uniform or Gaussian) and fed to the input layer
- $\max_D$: $D$ minimises binary cross-entropy with label 1 on real and 0 on fake samples, which is the same as maximising $V$
- $\min_G$: $G$ only affects the second term; it wants $D(G(z))$ to be large

**The optimal discriminator.** For fixed $G$, write $V=\int p_{\text{data}}(x)\log D(x)+p_g(x)\log(1-D(x))\,dx$ and maximise inside the integral, point by point. The function $y\mapsto a\log y+b\log(1-y)$ peaks at $y=a/(a+b)$. Substituting the optimum gives the generator's effective objective $C(G)$:

$$
D^*_G(x)=\frac{p_{\text{data}}(x)}{p_{\text{data}}(x)+p_g(x)},\qquad C(G)=V(D^*_G,G)=-\log4+2\,\mathrm{JS}\big(p_{\text{data}}\,\|\,p_g\big)
$$

- $\mathrm{JS}$: Jensen–Shannon divergence $\tfrac12\mathrm{KL}(p_{\text{data}}\|m)+\tfrac12\mathrm{KL}(p_g\|m)$, $m=\tfrac12(p_{\text{data}}+p_g)$; between 0 and $\log 2$
- $-\log4$: the global minimum of $C(G)$, reached only at $p_g=p_{\text{data}}$, where $D^*\equiv\tfrac12$
- $\log\frac{D^*}{1-D^*}$: the optimal logit is the log density ratio $\log p_{\text{data}}(x)-\log p_g(x)$

This is the paper's Theorem 1: the global minimum of the training criterion is reached if and only if $p_g=p_{\text{data}}$. Proposition 2 adds a convergence result. If $G$ and $D$ have enough capacity, $D$ is trained to its optimum at every step, and $p_g$ is updated to improve $C(G)$, then $p_g$ converges to $p_{\text{data}}$. The argument works in the space of densities, where $C$ is convex in $p_g$. Real networks optimise parameters, not densities, so the guarantee does not carry over. The full derivation, and the alternative losses that came later, are in **Adversarial loss**.

## The non-saturating generator loss

Early in training $D$ rejects the fakes easily, $D(G(z))\approx0$, and $\log(1-D(G(z)))$ is flat there. Goodfellow et al. noted this and trained $G$ to **maximise** $\log D(G(z))$ instead. The fixed point of the dynamics is unchanged, and the gradients are much stronger early on. In terms of the logit $a=a(G(z))$:

$$
\mathcal L_G^{\text{minimax}}=\mathbb E_z\log\big(1-\sigma(a)\big),\ \ \frac{\partial\mathcal L}{\partial a}=-\sigma(a)\qquad\qquad \mathcal L_G^{\text{NS}}=-\mathbb E_z\log\sigma(a),\ \ \frac{\partial\mathcal L}{\partial a}=\sigma(a)-1
$$

- $\sigma(a)=D(G(z))$: at $D(G(z))=0.01$ the minimax factor is 0.01, the non-saturating one 0.99, 99 times larger
- $\mathcal L_G^{\text{NS}}$: binary cross-entropy of the fakes **with the real label**; in code `BCEWithLogits(D(G(z)), ones)`

The two losses weight fakes oppositely. Minimax pushes hardest on fakes that already fool $D$. The non-saturating loss pushes hardest on fakes that $D$ rejects. With $D=D^*$, the non-saturating gradient equals the gradient of $\mathrm{KL}(p_g\|p_{\text{data}})-2\,\mathrm{JS}$ (Arjovsky & Bottou, 2017). This reverse KL heavily penalises samples where there is no data and barely penalises missing modes, one explanation for mode collapse. Almost every GAN in practice uses the non-saturating loss or a later variant (hinge, Wasserstein).

## Alternating training

**Algorithm 1 of the paper (minibatch stochastic gradient)**

1. Repeat $k$ times: draw $m$ noise samples $z^{(i)}$ and $m$ data samples $x^{(i)}$, and **ascend** $\nabla_{\theta_d}\frac1m\sum_i\big[\log D(x^{(i)})+\log(1-D(G(z^{(i)})))\big]$. The fakes are treated as constants (`detach()` in PyTorch).
2. Draw $m$ fresh noise samples and **descend** $\nabla_{\theta_g}\frac1m\sum_i\log(1-D(G(z^{(i)})))$, or ascend $\log D(G(z^{(i)}))$ for the non-saturating version. The gradient flows through $D$ into $G$, but only $G$'s parameters are updated.
3. The paper used $k=1$, the cheapest option, with momentum SGD. Later practice is Adam with $\beta_1=0.5$ (from **Deep convolutional GAN (DCGAN)**), $k=1$, and sometimes separate learning rates for $D$ and $G$ (the two time-scale update rule of Heusel et al., 2017).
4. Keeping $D$ near optimal makes $G$'s gradient follow $C(G)$. A $D$ that is too strong saturates the minimax loss. A $D$ that is too weak gives $G$ a meaningless signal. The paper also warns that $G$ must not be trained too much without updating $D$, to avoid the "Helvetica scenario", in which $G$ maps many values of $z$ to the same $x$.

**The original paper**

- **Venue:** NeurIPS (then NIPS) 2014; arXiv 1406.2661
- **Models:** multilayer perceptrons. $G$ used a mixture of rectifier and sigmoid units; $D$ used maxout units with dropout. Noise entered only at the bottom layer of $G$
- **Datasets:** MNIST, Toronto Face Database (TFD), CIFAR-10
- **Evaluation:** Gaussian Parzen-window log-likelihood of test data under $G$'s samples: 225 ± 2 on MNIST, 2057 ± 26 on TFD. The authors note that this estimator has high variance and does not work well in high dimensions
- **Stated disadvantages:** no explicit $p_g(x)$; $D$ must stay synchronised with $G$

## Mode collapse and other failure modes

The theory assumes $D$ is optimal at every step. In practice both networks take simultaneous or alternating small steps. That is a search for a saddle point, and gradient descent on a game need not converge. Three things go wrong. **Oscillation**: $G$ chases $D$'s decision boundary around the data without settling. **Mode collapse**: $G$ maps most $z$ to a few outputs that currently fool $D$. When $D$ adapts, $G$ moves to another mode, because nothing in a single-sample loss rewards diversity. **Vanishing gradients**: $D$ separates real from fake almost perfectly. Salimans et al. (2016, "Improved Techniques for Training GANs") proposed feature matching, minibatch discrimination, historical averaging and one-sided label smoothing. Later the main fixes became new losses (**Wasserstein GAN (WGAN)**, **WGAN-GP**) and Lipschitz control of $D$ (spectral normalisation).

> Figure: A real GAN, trained live, on 8 Gaussians (σ = 0.05) on a unit ring. G is a 2→32→2 tanh MLP and D a 2→32→1 tanh MLP. Full batches of 128, Adam with β₁ = 0.5, fixed seeds. The shading is D(x). Default settings: non-saturating loss, k = 3 D steps per G step, lr 0.01. At iteration 0, mean D(G(z)) = 0.482 and the D loss is 1.622. All 8 modes hold at least 4 fakes by iteration 280. By iteration 1000, 88% of fakes lie within 3σ of a mode, and D gives 0.554 to reals and 0.465 to fakes, close to the ½ of the equilibrium. Then the game slips. At iteration 1100, coverage drops to 3 modes and D(real) jumps to 0.754. G recovers to 7 modes, but mode 4 now holds 35 of the 128 fakes and mode 5 none, and this partial collapse lasts to iteration 1500. With lr 0.005, coverage reaches 8 at iteration 480 and never drops below 7 afterwards. With lr 0.02 it never exceeds 5 modes, and at iteration 1500 modes 0 and 4 hold 40 fakes each. Start G far away at (2.2, 2.2): D(G(z)) = 0.130, so the minimax generator's logit gradient is 0.130 against 0.870 for the non-saturating loss. Adam rescales steps, so with k = 3 both still cover all 8 modes (iteration 250 for minimax, 260 for non-saturating).

## Worked example: D*, JS and the two generator gradients

Take three points with $p_{\text{data}}=(0.45,\,0.45,\,0.10)$ and a generator with $p_g=(0.10,\,0.30,\,0.60)$. The generator puts most of its mass on $x_3$, where there is little data.

Optimal discriminator and per-sample generator gradient factors (computed in Python)

| x | p_data | p_g | D*(x) | logit a = log p_data/p_g | minimax \|∂L/∂a\| = D | NS \|∂L/∂a\| = 1 − D |
| --- | --- | --- | --- | --- | --- | --- |
| x₁ | 0.45 | 0.10 | 0.8182 | 1.5041 | 0.8182 | 0.1818 |
| x₂ | 0.45 | 0.30 | 0.6000 | 0.4055 | 0.6000 | 0.4000 |
| x₃ | 0.10 | 0.60 | 0.1429 | −1.7918 | 0.1429 | 0.8571 |

**Checking the theory on these numbers**

1. $V(D^*,G)=\sum_x p_{\text{data}}\log D^*+p_g\log(1-D^*)=-1.0526$.
2. $\mathrm{JS}(p_{\text{data}}\|p_g)=0.1668$ nats (the maximum is $\log2=0.6931$), and $-\log4+2(0.1668)=-1.0526$, the same value as $V(D^*,G)$.
3. So the discriminator's BCE loss at its optimum is $1.0526$. At equilibrium it would be $\log4=1.3863$, because $D\equiv\tfrac12$ costs $\log2$ on each half. A D loss **below** 1.386 means $D$ can still tell the two apart.
4. Under $p_g$, the average minimax factor is $0.3475$ and the non-saturating one $0.6525$. The difference is not just size. Minimax concentrates its push on $x_1$ (0.8182), where fakes are already rare and good. The non-saturating loss concentrates it on $x_3$ (0.8571), the region $G$ most needs to leave.

## Evaluating a GAN

A GAN gives no likelihood, so it is evaluated with sample statistics computed from a pretrained classifier. The **Inception Score** (Salimans et al., 2016) is $\mathrm{IS}=\exp\big(\mathbb E_x\,\mathrm{KL}(p(y\mid x)\,\|\,p(y))\big)$, using the label distribution of an ImageNet Inception network. It is high when each sample is classified confidently (sharp images) and the predicted classes are spread out (diversity). It never looks at real data, and it is blind to diversity within a class. The **Fréchet Inception Distance** (Heusel et al., 2017) fits Gaussians to the 2048-d pool features of real and generated images and compares them:

$$
\mathrm{FID}=\big\|\mu_r-\mu_g\big\|_2^2+\mathrm{Tr}\Big(\Sigma_r+\Sigma_g-2\big(\Sigma_r\Sigma_g\big)^{1/2}\Big)
$$

- $\mu_r,\Sigma_r$: mean and covariance of Inception-v3 pool features of real images
- $\mu_g,\Sigma_g$: the same for generated images; lower FID is better, 0 for identical Gaussians

FID drops when mode collapse is fixed, which IS does not reliably do. Both scores are biased by the sample size: FID is usually reported on 50,000 samples, and numbers computed with different sample counts or Inception implementations are not comparable. Precision and recall for distributions (Sajjadi et al., 2018; Kynkäänniemi et al., 2019) split the error into quality and coverage, the two quantities the figure tracks as "quality" and "modes covered".

```python
import math, torch, torch.nn as nn
torch.manual_seed(0)

def eight_gaussians(n, sigma=0.05):
    k = torch.randint(0, 8, (n,))
    centers = torch.stack([torch.cos(k * math.pi / 4), torch.sin(k * math.pi / 4)], 1)
    return centers + sigma * torch.randn(n, 2)

def mlp(i, o, h=128):
    return nn.Sequential(nn.Linear(i, h), nn.ReLU(), nn.Linear(h, h), nn.ReLU(), nn.Linear(h, o))

nz = 8
G = mlp(nz, 2)                 # z -> x
D = mlp(2, 1)                  # x -> logit a(x); D(x) = sigmoid(a)
opt_g = torch.optim.Adam(G.parameters(), lr=2e-4, betas=(0.5, 0.999))
opt_d = torch.optim.Adam(D.parameters(), lr=2e-4, betas=(0.5, 0.999))
bce = nn.BCEWithLogitsLoss()
B, k = 256, 1

for it in range(3001):
    for _ in range(k):                                    # discriminator: real -> 1, fake -> 0
        x = eight_gaussians(B)
        fake = G(torch.randn(B, nz)).detach()             # no gradient into G here
        a_real, a_fake = D(x), D(fake)
        loss_d = bce(a_real, torch.ones_like(a_real)) + bce(a_fake, torch.zeros_like(a_fake))
        opt_d.zero_grad(); loss_d.backward(); opt_d.step()
    a_fake = D(G(torch.randn(B, nz)))                     # generator: non-saturating loss
    loss_g = bce(a_fake, torch.ones_like(a_fake))         # = mean of -log D(G(z))
    # minimax alternative: loss_g = -bce(a_fake, torch.zeros_like(a_fake))  (= mean log(1 - D(G(z))))
    opt_g.zero_grad(); loss_g.backward(); opt_g.step()
    if it % 1000 == 0:
        print(it, f"loss_d {loss_d.item():.3f}  loss_g {loss_g.item():.3f}")   # loss_d near log 4 = 1.386 is not "converged"

with torch.no_grad():
    s = G(torch.randn(2000, nz))
    print(s.shape)                                         # torch.Size([2000, 2])
    c = torch.stack([torch.cos(torch.arange(8) * math.pi / 4), torch.sin(torch.arange(8) * math.pi / 4)], 1)
    d = torch.cdist(s, c)                                  # (2000, 8) distance to each mode
    near = d.min(1).values < 0.15                          # within 3 sigma of some mode
    print(near.float().mean().item(), torch.bincount(d.argmin(1)[near], minlength=8).tolist())  # quality, fakes per mode
```

A GAN on 8 Gaussians: alternating D and G steps with the non-saturating loss

> **GAN losses are not progress bars:** $D$'s loss measures how well the current $D$ separates the current $G$ from data. It does not measure how good $G$ is. A falling D loss can mean $G$ is collapsing. A D loss stuck at $\log4\approx1.386$ can mean a balanced game or a $D$ that learned nothing. In the figure's default run the D loss falls from 1.285 at iteration 1000 to 0.777 at 1100, exactly when coverage drops from 8 modes to 3: the falling loss signals collapse, not progress. Track sample quality and diversity (FID, precision and recall, per-mode counts on toy data), and look at samples.

> **The discriminator is a learned density-ratio estimator:** $D^*$'s logit is $\log p_{\text{data}}(x)/p_g(x)$. Every GAN update moves fakes up the gradient of an estimated log density ratio. That explains why GANs produce sharp samples: a blurry average of two modes falls where the ratio is low. It also explains collapse: the ratio only tells a fake to move uphill, not to spread out. The same density-ratio view underlies f-GAN (Nowozin et al., 2016) and the conditional discriminators of **Conditional GAN (cGAN)**.

> **Where GANs went:** Convolutional architectures (**Deep convolutional GAN (DCGAN)**), conditioning (**Conditional GAN (cGAN)**, **Pix2Pix**, **CycleGAN**), better losses (**Wasserstein GAN (WGAN)**, **WGAN-GP**) and scale (**StyleGAN**, **BigGAN**) made GANs the leading image generators until about 2021. **Diffusion models** then overtook them on diversity and likelihood, while GANs remain much faster at sampling (one forward pass). Adversarial losses are still used inside other systems, for example in the autoencoder of **Latent diffusion models** and in distilled one-step samplers.

**Strengths**

- Sampling is one forward pass of G: fast, and any differentiable architecture works
- No likelihood, no Markov chain, no inference network: training is plain backpropagation
- Learned loss: samples are sharp because blurry averages are easy for $D$ to reject
- Theory: with an optimal $D$ the game minimises JS, with a unique optimum at $p_g=p_{\text{data}}$

**Limitations**

- A saddle-point problem: oscillation and divergence are common, and hyperparameters are fragile
- Mode collapse: diversity is not directly rewarded
- No density $p_g(x)$ and no encoder, so likelihoods cannot be evaluated and real inputs cannot be mapped to $z$ without extra machinery
- Losses do not track progress; evaluation relies on Inception-based scores with known biases

**Question:** For a fixed generator, what is the discriminator that maximises V(D, G)?

- A. $D^*(x)=p_{\text{data}}(x)\,/\,\big(p_{\text{data}}(x)+p_g(x)\big)$
- B. $D^*(x)=p_{\text{data}}(x)\,/\,p_g(x)$
- C. $D^*(x)=\tfrac12$ for every $x$
- D. $D^*(x)=1$ on the support of the data, 0 elsewhere

**Answer:** A. $D^*(x)=p_{\text{data}}(x)\,/\,\big(p_{\text{data}}(x)+p_g(x)\big)$ — Pointwise, $p_{\text{data}}\log D+p_g\log(1-D)$ is maximised at $D=p_{\text{data}}/(p_{\text{data}}+p_g)$. Its logit is the log density ratio. $D^*\equiv\tfrac12$ only at the optimum $p_g=p_{\text{data}}$. The density ratio itself is not a probability.

**Question:** Early in training D(G(z)) ≈ 0.001. Why do practitioners use the non-saturating loss −log D(G(z))?

- A. It has a different optimum, closer to the data
- B. Its gradient with respect to D's logit is $\sigma(a)-1\approx-0.999$, while the minimax loss gives $-\sigma(a)\approx-0.001$
- C. It makes the discriminator's job easier
- D. It removes the need to train D

**Answer:** B. Its gradient with respect to D's logit is $\sigma(a)-1\approx-0.999$, while the minimax loss gives $-\sigma(a)\approx-0.001$ — Both losses have the same fixed point. The minimax loss $\log(1-\sigma(a))$ saturates when $D$ confidently rejects the fakes, so $G$ gets almost no gradient exactly when it needs one. The non-saturating loss gives a factor near 1 there, 999 times larger at $D(G(z))=0.001$.

**Key points**

- GAN game: $\min_G\max_D\ \mathbb E_{p_{\text{data}}}[\log D(x)]+\mathbb E_{z}[\log(1-D(G(z)))]$, with $D$ trained as a real/fake BCE classifier.
- Optimal $D^*=p_{\text{data}}/(p_{\text{data}}+p_g)$ gives $C(G)=-\log4+2\,\mathrm{JS}(p_{\text{data}}\|p_g)$, minimised only at $p_g=p_{\text{data}}$.
- Non-saturating generator loss $-\log D(G(z))$: same fixed point, gradient factor $1-D$ instead of $D$, so it does not vanish when $D$ rejects the fakes.
- Training alternates $k$ D steps (paper: $k=1$) with one G step. It is a saddle-point search, prone to oscillation and mode collapse.
- Evaluate with sample statistics: Inception Score (sharpness and class spread) and FID (Fréchet distance of Inception features, lower is better).

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
