# WGAN-GP

> WGAN-GP (Gulrajani et al., 2017) replaces WGAN's weight clipping with a gradient penalty λ(‖∇D(x̂)‖₂ − 1)² on random interpolates x̂ = εx + (1 − ε)x̃ of real and fake samples. Defaults: λ = 10, 5 critic steps per generator step, Adam (lr 1e-4, β = (0, 0.9)), no batch norm in the critic.

Source: https://www.mlfoundry.in/learn/wgan-gp  
Section: Deep Learning › Generative Deep Learning Algorithms  
Level: advanced · 13 min read · updated 2026-10-12

**WGAN-GP:** (Wasserstein GAN with gradient penalty) is the **Wasserstein GAN (WGAN)** objective with the critic's Lipschitz constraint enforced softly. Instead of clipping the critic's weights, it adds the penalty $\lambda\,\mathbb E_{\hat x}\big[(\|\nabla_{\hat x}D(\hat x)\|_2-1)^2\big]$ on points $\hat x$ sampled uniformly on straight lines between real and generated samples. Introduced by Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin and Aaron Courville, "Improved Training of Wasserstein GANs" (NeurIPS 2017).

WGAN made GAN training far more robust, but it could still produce poor samples or fail to converge. Gulrajani et al. traced these failures to weight clipping, the device WGAN uses to keep the critic $K$-Lipschitz. Clipping biases the critic toward overly simple functions, and in deep critics it makes gradients explode or vanish unless $c$ is tuned carefully. They tried other weight constraints (L2-norm clipping, weight normalisation) and soft L1/L2 weight decay, and saw the same problems. Their fix constrains what the Lipschitz condition is actually about: the critic's gradient with respect to its input. With it they trained 101-layer ResNets, MLP generators and generators without normalisation, all with one set of hyperparameters.

## What the optimal critic looks like

$$
f^*=\arg\max_{\|f\|_L\le1}\ \mathbb E_{y\sim P_r}[f(y)]-\mathbb E_{x\sim P_g}[f(x)],\qquad x_t=tx+(1-t)y\ \Rightarrow\ \nabla f^*(x_t)=\frac{y-x_t}{\|y-x_t\|}\ \ \text{for } (x,y)\sim\pi
$$

- $\pi$: the optimal coupling between $P_r$ and $P_g$ (the transport plan that achieves $W$)
- $x_t$: any point on the segment between a generated $x$ and the real $y$ it is coupled with
- $\nabla f^*(x_t)$: a unit vector pointing from $x_t$ to $y$: the optimal critic rises with slope exactly 1 along the transport lines

This is Proposition 1 of the paper. Its Corollary 1: $f^*$ has gradient norm 1 almost everywhere under $P_r$ and $P_g$. A 1-Lipschitz function that is differentiable has $\|\nabla f\|\le1$ everywhere, and the optimal one sits on that bound along the lines that matter. Two consequences follow. A constraint that keeps the critic far below the bound (as clipping does) cannot represent $f^*$. And it is enough to check the gradient norm on those lines, not everywhere in input space.

## Why weight clipping fails

**Capacity underuse.** A clipped network that tries to reach its maximum slope ends up computing a very simple function. The paper trains critics to optimality on 8 Gaussians, 25 Gaussians and a Swiss roll, with the generator fixed at the data plus unit Gaussian noise. With clipping, the critic ignores the higher moments of the data distribution and learns a very simple approximation of the optimal critic. With the gradient penalty, its value surface follows the data. The paper also shows that clipping pushes the weights toward the two ends of the range, $\pm c$.

**Exploding and vanishing gradients.** In a 12-layer ReLU critic without batch norm trained on the Swiss roll, the norm of the gradient of the critic loss with respect to successive layers either grows or decays exponentially toward the input, for each of $c=10^{-1},10^{-2},10^{-3}$. The reason is simple. If a weight matrix of width $n$ has entries of magnitude $c$, it scales a random vector by roughly $c\sqrt n$, and ReLU keeps about half the units, so each layer multiplies the backward signal by about $c\sqrt{n/2}$. Twelve layers raise that factor to the 12th power. Only a narrow band of $c$ gives a gain near 1, and it depends on the width.

## The gradient penalty

$$
L=\underbrace{\mathbb E_{\tilde x\sim P_g}\big[D(\tilde x)\big]-\mathbb E_{x\sim P_r}\big[D(x)\big]}_{\text{original critic loss}}+\underbrace{\lambda\,\mathbb E_{\hat x\sim P_{\hat x}}\Big[\big(\|\nabla_{\hat x}D(\hat x)\|_2-1\big)^2\Big]}_{\text{gradient penalty}},\qquad \hat x=\epsilon x+(1-\epsilon)\tilde x,\ \ \epsilon\sim U[0,1]
$$

- $D$: the critic (the paper writes $D$ but it is a WGAN critic: no sigmoid, unbounded score)
- $\tilde x=G(z)$: a generated sample
- $\hat x$: a random point on the segment between a real sample $x$ and a generated sample $\tilde x$, with a fresh $\epsilon$ per pair
- $\lambda=10$: penalty coefficient, used in all experiments of the paper from toy tasks to large ImageNet CNNs
- $(\|\cdot\|-1)^2$: a **two-sided** penalty: it pushes the norm toward 1, not just below 1

A differentiable function is 1-Lipschitz if and only if its gradient norm is at most 1 everywhere. Enforcing that everywhere is intractable, so the penalty is a soft version on samples from $P_{\hat x}$. Sampling on straight lines between real and fake points follows Proposition 1: the optimal critic has unit slope on lines between coupled points. Random pairs are not the optimal coupling, but the authors found that this sampling is enough in practice. The penalty is two-sided. The optimal critic has norm-1 gradients almost everywhere under $P_r$ and $P_g$ and in much of the space between, so pushing toward 1 constrains it very little. In their early tests the two-sided version did slightly better. In the appendix, over 174 sampled architectures, the two-sided penalty gave the higher Inception score in 100 trials and the one-sided $\max(0,\|\nabla\|-1)^2$ in 77. The authors note that this difference is not statistically significant. Computing the penalty needs the gradient of $D$ with respect to its input inside the loss, so training back-propagates through a gradient (a "double backward").

**No batch norm in the critic.** Batch normalisation turns the critic into a function of the whole batch. The penalty, though, constrains the gradient with respect to each input on its own, so with batch norm the objective is no longer valid. The paper drops normalisation from the critic, or uses **layer normalisation**, which normalises each example independently. Batch norm in the generator is fine.

**Settings and results (Algorithm 1, Sections 4–5)**

- **Defaults:** $\lambda=10$, $n_{\text{critic}}=5$, Adam with $\alpha=10^{-4}$, $\beta_1=0$, $\beta_2=0.9$
- **Critic:** no batch norm; layer norm recommended as a drop-in replacement. With BN or LN in the discriminator, a small weight decay (10⁻³) helped slightly
- **200 random architectures:** 32×32 ImageNet, depth 4–20, varied nonlinearities and normalisation. At an Inception-score threshold of 5.0: 147 trained only with WGAN-GP, 0 only with the standard GAN, 42 with both, 11 with neither
- **LSUN bedrooms:** six hard architectures (no BN with constant filter count, 4-layer 512-unit MLP generator, no normalisation anywhere, gated nonlinearities, tanh, 101-layer ResNet) all trained with one set of hyperparameters; with DCGAN, LSGAN and clipped WGAN some of them were unstable or mode-collapsed
- **CIFAR-10 Inception score:** unsupervised ResNet 7.86 ± 0.07 (state of the art at the time); with labels 8.42 ± 0.10, second only to SGAN (8.59)
- **Speed:** converged faster and higher than weight clipping even with the same RMSProp optimiser; slower in wall-clock time than DCGAN but more stable at convergence
- **Text:** a character-level GAN on the Billion Word dataset with a 1-D CNN generator emitting softmax outputs, with no sampling step

> Figure: The paper's Figure 1 setup in miniature. The generator is fixed at the data plus N(0, I) noise, and only the critic is trained. The critic here is a linear model on 24 random Fourier features. Clipping is then solved exactly: the box maximum puts every weight at +c or −c. The penalised objective is minimised by Gauss–Newton, with a tiny ridge so the finite model stays well-posed. On 8 Gaussians with λ = 10, the mean gradient norm on the interpolates x̂ is 1.002 ± 0.105, 94.1 % of them lie in [0.8, 1.2], the penalty is 0.011, and the critic's estimate E_r f − E_g f is 0.606. That is below 1.253 = E‖noise‖, an upper bound on W₁ here because moving each point back by its noise is one valid transport plan. A weak penalty fails: at λ = 0.1 the mean norm is 2.90 and the estimate 2.75 exceeds the bound, so the critic is not 1-Lipschitz. λ = 1 gives 1.15 (49 % in band). Weight clipping at c = 0.01 gives a mean norm of 0.0146, 0 % in band, and all 24 weights at ±c. Changing c only rescales that critic, because only the signs of the feature differences survive. On the Swiss roll and 25 Gaussians, λ = 10 gives mean norms of 1.000 and 0.976. The lower panel takes a 12-layer, width-512 ReLU critic with every weight at ±c (where clipping drives weights). The backward signal changes by about c·√(512/2) = 16c per layer. For c = 0.01 that is 0.16, and the input-layer gradient is 3.3·10⁻¹⁰ of the output-layer one. For c = 0.1 it is 1.6, an explosion by 334×. For c = 0.001 it is 3.3·10⁻²². He-scaled Gaussian weights keep the ratio at 0.71.

## Worked example: one interpolate, its penalty, and its gradient

Take the critic $D(x)=w_0x_1+w_1x_2+w_2x_1^2$ with $w=(2,\,-1,\,0.5)$, so $\nabla_xD=(w_0+2w_2x_1,\ w_1)=(2+x_1,\,-1)$. One real sample $x=(1,0)$, one generated $\tilde x=(-1,2)$, $\lambda=10$. Then $D(x)=2.5$ and $D(\tilde x)=-3.5$, so the original critic loss is $D(\tilde x)-D(x)=-6.0$. Penalties along the segment (all checked with PyTorch autograd):

Interpolates x̂ = εx + (1 − ε)x̃ and their penalty terms

| ε | x̂ | ∇D(x̂) | ‖∇D(x̂)‖ | λ(‖∇D‖ − 1)² |
| --- | --- | --- | --- | --- |
| 0 (the fake point) | (−1, 2) | (1, −1) | 1.4142 | 1.7157 |
| 0.25 | (−0.5, 1.5) | (1.5, −1) | 1.8028 | 6.4445 |
| 0.5 | (0, 1) | (2, −1) | 2.2361 | 15.2786 |
| 1 (the real point) | (1, 0) | (3, −1) | 3.1623 | 46.7544 |

**The loss for ε = 0.25 and what its gradient does**

1. $L=D(\tilde x)-D(x)+\lambda(\|\nabla D(\hat x)\|-1)^2=-3.5-2.5+10\,(1.8028-1)^2=-6.0+6.4445=0.4445$.
2. Gradient of the original loss with respect to $w_0$: $\tilde x_1-x_1=-2$. Descending it makes $D$ steeper along $x_1$, which separates real from fake better.
3. Gradient of the penalty with respect to $w_0$: $2\lambda(\|g\|-1)\cdot\partial\|g\|/\partial w_0=20\cdot0.8028\cdot(1.5/1.8028)=13.359$. In total, $\partial L/\partial w=(11.359,\ -6.906,\ -13.359)$, the same as PyTorch's double backward. The penalty wins here, so the step flattens the critic.
4. At equilibrium the two forces balance. The critic is as steep as it can be (raising $\mathbb E_rD-\mathbb E_gD$) while its slope on the interpolates stays near 1. The figure shows this: 94 % of interpolates in $[0.8,1.2]$ at $\lambda=10$.

```python
import torch, torch.nn as nn

torch.manual_seed(0)

def gradient_penalty(D, real, fake, lam=10.0):
    eps = torch.rand(real.size(0), *([1] * (real.dim() - 1)))      # one ε per example, broadcast over features
    x_hat = (eps * real + (1 - eps) * fake).requires_grad_(True)
    d_hat = D(x_hat)
    grad, = torch.autograd.grad(d_hat.sum(), x_hat, create_graph=True)  # keep the graph: penalty is differentiated again
    norms = grad.flatten(1).norm(dim=1)                             # ‖∇_x̂ D(x̂)‖₂ per example
    return lam * ((norms - 1) ** 2).mean(), norms

def critic(d=2, h=256):                                             # no BatchNorm: LayerNorm keeps examples independent
    return nn.Sequential(nn.Linear(d, h), nn.LayerNorm(h), nn.LeakyReLU(0.2),
                         nn.Linear(h, h), nn.LayerNorm(h), nn.LeakyReLU(0.2), nn.Linear(h, 1))

G = nn.Sequential(nn.Linear(2, 256), nn.ReLU(), nn.Linear(256, 256), nn.ReLU(), nn.Linear(256, 2))
D = critic()
opt_d = torch.optim.Adam(D.parameters(), lr=1e-4, betas=(0.0, 0.9))   # the paper's defaults
opt_g = torch.optim.Adam(G.parameters(), lr=1e-4, betas=(0.0, 0.9))
n_critic, m = 5, 256

def real_batch(m):                                                   # 8 Gaussians, radius 2
    k = torch.randint(0, 8, (m,)).float() * (torch.pi / 4)
    return 2 * torch.stack([k.cos(), k.sin()], 1) + 0.05 * torch.randn(m, 2)

for it in range(100):
    for _ in range(n_critic):
        x, z = real_batch(m), torch.randn(m, 2)
        fake = G(z).detach()
        gp, norms = gradient_penalty(D, x, fake)
        loss_d = D(fake).mean() - D(x).mean() + gp                  # L = E D(x̃) − E D(x) + λ E(‖∇D(x̂)‖ − 1)²
        opt_d.zero_grad(); loss_d.backward(); opt_d.step()
    loss_g = -D(G(torch.randn(m, 2))).mean()
    opt_g.zero_grad(); loss_g.backward(); opt_g.step()

print(norms.shape, D(x).shape)                     # torch.Size([256]) torch.Size([256, 1])
print(f"mean ‖∇D(x̂)‖ = {norms.mean().item():.3f}")       # pulled toward 1 by the two-sided penalty
```

WGAN-GP on 8 Gaussians: penalty with create_graph=True, LayerNorm critic, the paper's defaults

> **Batch norm in the critic breaks the penalty:** With batch norm, $D(\hat x_i)$ depends on every other example in the batch, so $\nabla_{\hat x_i}D$ is not the per-example slope that the Lipschitz argument needs. Use no normalisation, layer norm, or another per-example scheme in the critic. Most DCGAN-style code puts `BatchNorm2d` in the discriminator by default, so porting it to WGAN-GP means removing or replacing those layers.

> **Forgetting create_graph:** `torch.autograd.grad(..., create_graph=True)` is required. Without it the gradient is a constant with respect to the critic's weights, and the penalty contributes nothing to `loss_d.backward()`. The training loop still runs, but the critic is unconstrained. The double backward costs roughly one extra forward and backward pass per critic step. Also use one $\epsilon$ per example, broadcast over its features, not one per pixel.

> **The constraint is only enforced where it is sampled:** The penalty acts on $P_{\hat x}$, the segments between current real and fake samples. Far from the data the critic can have any slope. That is enough because the generator only ever queries the critic near its own samples, and those move along the same segments. It also means the constraint moves with the generator. Mescheder et al. (2018) later analysed zero-centred penalties on real data only ($R_1$), which **BigGAN** tested and **StyleGAN** uses. Spectral normalisation instead enforces a global bound per layer.

> **The loss is a useful curve:** As with WGAN, the negative critic loss (the estimate of $W$) falls as the generator improves, and the paper plots it for LSUN bedrooms. On a 1000-image MNIST subset it also reveals overfitting. The critic's loss on held-out data diverges from the training loss: the critic overfits faster than the generator, so the training loss gradually rises even as the validation loss drops.

**Strengths**

- No clipping threshold to tune; $\lambda=10$ worked across toy data, CIFAR-10, LSUN and ImageNet in the paper
- Stable gradients in deep critics: trained 101-layer ResNets and architectures that failed with DCGAN, LSGAN and clipped WGAN
- Works with Adam ($\beta_1=0$, $\beta_2=0.9$); converges faster and higher than weight clipping
- The critic estimate remains a meaningful training curve and can detect overfitting

**Limitations**

- Double backpropagation: each critic step is noticeably more expensive
- No batch norm in the critic; layer norm or no normalisation instead
- The constraint is soft and holds only on the sampled interpolates; with too small $\lambda$ the critic is not Lipschitz (estimate above the true bound in the figure)
- Still $n_{\text{critic}}=5$ critic steps per generator step; slower in wall-clock time than DCGAN in the paper

**Question:** Where is the WGAN-GP penalty evaluated?

- A. On real samples only
- B. On generated samples only
- C. On x̂ = εx + (1 − ε)x̃ with ε ~ U[0, 1], random points on lines between real and generated samples
- D. On a fixed grid covering the input space

**Answer:** C. On x̂ = εx + (1 − ε)x̃ with ε ~ U[0, 1], random points on lines between real and generated samples — Proposition 1 says the optimal critic has unit gradient norm on the segments between coupled real and fake points. Enforcing the norm everywhere is intractable, so the paper samples $\hat x$ uniformly along lines between random real/fake pairs. Penalties on real data only ($R_1$) are a later, different method.

**Question:** Why does WGAN-GP remove batch normalisation from the critic?

- A. Batch norm makes the critic too slow
- B. The penalty constrains each input's gradient separately, but with batch norm each output depends on the whole batch, so the per-example gradient no longer measures the critic's Lipschitz constant
- C. Batch norm would make the critic output a probability
- D. Batch norm is incompatible with Adam

**Answer:** B. The penalty constrains each input's gradient separately, but with batch norm each output depends on the whole batch, so the per-example gradient no longer measures the critic's Lipschitz constant — Batch norm turns the critic into a map from a batch of inputs to a batch of outputs. The Lipschitz condition, and the penalty $(\|\nabla_{\hat x}D(\hat x)\|-1)^2$, are about a single input mapped to a single output. Layer norm normalises each example on its own, so it is the recommended drop-in replacement.

**Key points**

- Critic loss $\mathbb E[D(\tilde x)]-\mathbb E[D(x)]+\lambda\,\mathbb E\big[(\|\nabla_{\hat x}D(\hat x)\|_2-1)^2\big]$ with $\hat x=\epsilon x+(1-\epsilon)\tilde x$, $\epsilon\sim U[0,1]$.
- Defaults: $\lambda=10$, $n_{\text{critic}}=5$, Adam $\alpha=10^{-4}$, $\beta_1=0$, $\beta_2=0.9$; no batch norm in the critic (layer norm instead).
- The optimal 1-Lipschitz critic has gradient norm 1 along the lines between coupled real and fake points (Proposition 1). That motivates both the sampling and the two-sided penalty.
- Weight clipping underuses capacity, pushes weights to $\pm c$, and makes gradients in deep critics grow or decay by about $c\sqrt{n/2}$ per layer.
- With one set of hyperparameters it trained 101-layer ResNets, MLP generators and normalisation-free GANs; CIFAR-10 Inception score 7.86 unsupervised, 8.42 with labels.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
