Definition
WGAN-GP (Wasserstein GAN with gradient penalty) is the Wasserstein GAN (WGAN) objective with the critic's Lipschitz constraint enforced softly. Instead of clipping the critic's weights, it adds the penalty on points sampled uniformly on straight lines between real and generated samples. Introduced by Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin and Aaron Courville, "Improved Training of Wasserstein GANs" (NeurIPS 2017).
WGAN made GAN training far more robust, but it could still produce poor samples or fail to converge. Gulrajani et al. traced these failures to weight clipping, the device WGAN uses to keep the critic -Lipschitz. Clipping biases the critic toward overly simple functions, and in deep critics it makes gradients explode or vanish unless is tuned carefully. They tried other weight constraints (L2-norm clipping, weight normalisation) and soft L1/L2 weight decay, and saw the same problems. Their fix constrains what the Lipschitz condition is actually about: the critic's gradient with respect to its input. With it they trained 101-layer ResNets, MLP generators and generators without normalisation, all with one set of hyperparameters.
What the optimal critic looks like
- the optimal coupling between and (the transport plan that achieves )
- any point on the segment between a generated and the real it is coupled with
- a unit vector pointing from to : the optimal critic rises with slope exactly 1 along the transport lines
This is Proposition 1 of the paper. Its Corollary 1: has gradient norm 1 almost everywhere under and . A 1-Lipschitz function that is differentiable has everywhere, and the optimal one sits on that bound along the lines that matter. Two consequences follow. A constraint that keeps the critic far below the bound (as clipping does) cannot represent . And it is enough to check the gradient norm on those lines, not everywhere in input space.
Why weight clipping fails
Capacity underuse. A clipped network that tries to reach its maximum slope ends up computing a very simple function. The paper trains critics to optimality on 8 Gaussians, 25 Gaussians and a Swiss roll, with the generator fixed at the data plus unit Gaussian noise. With clipping, the critic ignores the higher moments of the data distribution and learns a very simple approximation of the optimal critic. With the gradient penalty, its value surface follows the data. The paper also shows that clipping pushes the weights toward the two ends of the range, .
Exploding and vanishing gradients. In a 12-layer ReLU critic without batch norm trained on the Swiss roll, the norm of the gradient of the critic loss with respect to successive layers either grows or decays exponentially toward the input, for each of . The reason is simple. If a weight matrix of width has entries of magnitude , it scales a random vector by roughly , and ReLU keeps about half the units, so each layer multiplies the backward signal by about . Twelve layers raise that factor to the 12th power. Only a narrow band of gives a gain near 1, and it depends on the width.
The gradient penalty
- the critic (the paper writes but it is a WGAN critic: no sigmoid, unbounded score)
- a generated sample
- a random point on the segment between a real sample and a generated sample , with a fresh per pair
- penalty coefficient, used in all experiments of the paper from toy tasks to large ImageNet CNNs
- a two-sided penalty: it pushes the norm toward 1, not just below 1
A differentiable function is 1-Lipschitz if and only if its gradient norm is at most 1 everywhere. Enforcing that everywhere is intractable, so the penalty is a soft version on samples from . Sampling on straight lines between real and fake points follows Proposition 1: the optimal critic has unit slope on lines between coupled points. Random pairs are not the optimal coupling, but the authors found that this sampling is enough in practice. The penalty is two-sided. The optimal critic has norm-1 gradients almost everywhere under and and in much of the space between, so pushing toward 1 constrains it very little. In their early tests the two-sided version did slightly better. In the appendix, over 174 sampled architectures, the two-sided penalty gave the higher Inception score in 100 trials and the one-sided in 77. The authors note that this difference is not statistically significant. Computing the penalty needs the gradient of with respect to its input inside the loss, so training back-propagates through a gradient (a "double backward").
No batch norm in the critic. Batch normalisation turns the critic into a function of the whole batch. The penalty, though, constrains the gradient with respect to each input on its own, so with batch norm the objective is no longer valid. The paper drops normalisation from the critic, or uses layer normalisation, which normalises each example independently. Batch norm in the generator is fine.
Settings and results (Algorithm 1, Sections 4–5)
- Defaults
- , , Adam with , ,
- Critic
- no batch norm; layer norm recommended as a drop-in replacement. With BN or LN in the discriminator, a small weight decay (10⁻³) helped slightly
- 200 random architectures
- 32×32 ImageNet, depth 4–20, varied nonlinearities and normalisation. At an Inception-score threshold of 5.0: 147 trained only with WGAN-GP, 0 only with the standard GAN, 42 with both, 11 with neither
- LSUN bedrooms
- six hard architectures (no BN with constant filter count, 4-layer 512-unit MLP generator, no normalisation anywhere, gated nonlinearities, tanh, 101-layer ResNet) all trained with one set of hyperparameters; with DCGAN, LSGAN and clipped WGAN some of them were unstable or mode-collapsed
- CIFAR-10 Inception score
- unsupervised ResNet 7.86 ± 0.07 (state of the art at the time); with labels 8.42 ± 0.10, second only to SGAN (8.59)
- Speed
- converged faster and higher than weight clipping even with the same RMSProp optimiser; slower in wall-clock time than DCGAN but more stable at convergence
- Text
- a character-level GAN on the Billion Word dataset with a 1-D CNN generator emitting softmax outputs, with no sampling step
The critic is a linear model on 24 random Fourier features, so clipping is solved exactly and the penalised objective (two-sided, λ(‖∇f(x̂)‖ − 1)² on x̂ = εx + (1 − ε)x̃) by Gauss–Newton. c sets both the clipped critic and the deep-network panel.
Worked example: one interpolate, its penalty, and its gradient
Take the critic with , so . One real sample , one generated , . Then and , so the original critic loss is . Penalties along the segment (all checked with PyTorch autograd):
| ε | x̂ | ∇D(x̂) | ‖∇D(x̂)‖ | λ(‖∇D‖ − 1)² |
|---|---|---|---|---|
| 0 (the fake point) | (−1, 2) | (1, −1) | 1.4142 | 1.7157 |
| 0.25 | (−0.5, 1.5) | (1.5, −1) | 1.8028 | 6.4445 |
| 0.5 | (0, 1) | (2, −1) | 2.2361 | 15.2786 |
| 1 (the real point) | (1, 0) | (3, −1) | 3.1623 | 46.7544 |
The loss for ε = 0.25 and what its gradient does
- 1.
- 2Gradient of the original loss with respect to : . Descending it makes steeper along , which separates real from fake better.
- 3Gradient of the penalty with respect to : . In total, , the same as PyTorch's double backward. The penalty wins here, so the step flattens the critic.
- 4At equilibrium the two forces balance. The critic is as steep as it can be (raising ) while its slope on the interpolates stays near 1. The figure shows this: 94 % of interpolates in at .
1import torch, torch.nn as nn23torch.manual_seed(0)45def gradient_penalty(D, real, fake, lam=10.0):6 eps = torch.rand(real.size(0), *([1] * (real.dim() - 1))) # one ε per example, broadcast over features7 x_hat = (eps * real + (1 - eps) * fake).requires_grad_(True)8 d_hat = D(x_hat)9 grad, = torch.autograd.grad(d_hat.sum(), x_hat, create_graph=True) # keep the graph: penalty is differentiated again10 norms = grad.flatten(1).norm(dim=1) # ‖∇_x̂ D(x̂)‖₂ per example11 return lam * ((norms - 1) ** 2).mean(), norms1213def critic(d=2, h=256): # no BatchNorm: LayerNorm keeps examples independent14 return nn.Sequential(nn.Linear(d, h), nn.LayerNorm(h), nn.LeakyReLU(0.2),15 nn.Linear(h, h), nn.LayerNorm(h), nn.LeakyReLU(0.2), nn.Linear(h, 1))1617G = nn.Sequential(nn.Linear(2, 256), nn.ReLU(), nn.Linear(256, 256), nn.ReLU(), nn.Linear(256, 2))18D = critic()19opt_d = torch.optim.Adam(D.parameters(), lr=1e-4, betas=(0.0, 0.9)) # the paper's defaults20opt_g = torch.optim.Adam(G.parameters(), lr=1e-4, betas=(0.0, 0.9))21n_critic, m = 5, 2562223def real_batch(m): # 8 Gaussians, radius 224 k = torch.randint(0, 8, (m,)).float() * (torch.pi / 4)25 return 2 * torch.stack([k.cos(), k.sin()], 1) + 0.05 * torch.randn(m, 2)2627for it in range(100):28 for _ in range(n_critic):29 x, z = real_batch(m), torch.randn(m, 2)30 fake = G(z).detach()31 gp, norms = gradient_penalty(D, x, fake)32 loss_d = D(fake).mean() - D(x).mean() + gp # L = E D(x̃) − E D(x) + λ E(‖∇D(x̂)‖ − 1)²33 opt_d.zero_grad(); loss_d.backward(); opt_d.step()34 loss_g = -D(G(torch.randn(m, 2))).mean()35 opt_g.zero_grad(); loss_g.backward(); opt_g.step()3637print(norms.shape, D(x).shape) # torch.Size([256]) torch.Size([256, 1])38print(f"mean ‖∇D(x̂)‖ = {norms.mean().item():.3f}") # pulled toward 1 by the two-sided penaltyUse when
- +No clipping threshold to tune; worked across toy data, CIFAR-10, LSUN and ImageNet in the paper
- +Stable gradients in deep critics: trained 101-layer ResNets and architectures that failed with DCGAN, LSGAN and clipped WGAN
- +Works with Adam (, ); converges faster and higher than weight clipping
- +The critic estimate remains a meaningful training curve and can detect overfitting
Avoid when
- −Double backpropagation: each critic step is noticeably more expensive
- −No batch norm in the critic; layer norm or no normalisation instead
- −The constraint is soft and holds only on the sampled interpolates; with too small the critic is not Lipschitz (estimate above the true bound in the figure)
- −Still critic steps per generator step; slower in wall-clock time than DCGAN in the paper
Check yourself
Where is the WGAN-GP penalty evaluated?
Check yourself
Why does WGAN-GP remove batch normalisation from the critic?
Remember
- ✓Critic loss with , .
- ✓Defaults: , , Adam , , ; no batch norm in the critic (layer norm instead).
- ✓The optimal 1-Lipschitz critic has gradient norm 1 along the lines between coupled real and fake points (Proposition 1). That motivates both the sampling and the two-sided penalty.
- ✓Weight clipping underuses capacity, pushes weights to , and makes gradients in deep critics grow or decay by about per layer.
- ✓With one set of hyperparameters it trained 101-layer ResNets, MLP generators and normalisation-free GANs; CIFAR-10 Inception score 7.86 unsupervised, 8.42 with labels.