WGAN-GP

WGAN-GP (Gulrajani et al., 2017) replaces WGAN's weight clipping with a gradient penalty λ(‖∇D(x̂)‖₂ − 1)² on random interpolates x̂ = εx + (1 − ε)x̃ of real and fake samples. Defaults: λ = 10, 5 critic steps per generator step, Adam (lr 1e-4, β = (0, 0.9)), no batch norm in the critic.

13 min read1220 wordsadvanced0 viewsupdated 2026-10-12

Definition

WGAN-GP (Wasserstein GAN with gradient penalty) is the Wasserstein GAN (WGAN) objective with the critic's Lipschitz constraint enforced softly. Instead of clipping the critic's weights, it adds the penalty λ Ex^[(∥∇x^D(x^)∥2−1)2]\lambda\,\mathbb E_{\hat x}\big[(\|\nabla_{\hat x}D(\hat x)\|_2-1)^2\big] on points x^\hat x sampled uniformly on straight lines between real and generated samples. Introduced by Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin and Aaron Courville, "Improved Training of Wasserstein GANs" (NeurIPS 2017).

WGAN made GAN training far more robust, but it could still produce poor samples or fail to converge. Gulrajani et al. traced these failures to weight clipping, the device WGAN uses to keep the critic KK-Lipschitz. Clipping biases the critic toward overly simple functions, and in deep critics it makes gradients explode or vanish unless cc is tuned carefully. They tried other weight constraints (L2-norm clipping, weight normalisation) and soft L1/L2 weight decay, and saw the same problems. Their fix constrains what the Lipschitz condition is actually about: the critic's gradient with respect to its input. With it they trained 101-layer ResNets, MLP generators and generators without normalisation, all with one set of hyperparameters.

What the optimal critic looks like

f∗=arg⁡max⁡∥f∥L≤1 Ey∼Pr[f(y)]−Ex∼Pg[f(x)],xt=tx+(1−t)y ⇒ ∇f∗(xt)=y−xt∥y−xt∥  for (x,y)∼πf^*=\arg\max_{\|f\|_L\le1}\ \mathbb E_{y\sim P_r}[f(y)]-\mathbb E_{x\sim P_g}[f(x)],\qquad x_t=tx+(1-t)y\ \Rightarrow\ \nabla f^*(x_t)=\frac{y-x_t}{\|y-x_t\|}\ \ \text{for } (x,y)\sim\pi
π\pi
the optimal coupling between PrP_r and PgP_g (the transport plan that achieves WW)
xtx_t
any point on the segment between a generated xx and the real yy it is coupled with
∇f∗(xt)\nabla f^*(x_t)
a unit vector pointing from xtx_t to yy: the optimal critic rises with slope exactly 1 along the transport lines

This is Proposition 1 of the paper. Its Corollary 1: f∗f^* has gradient norm 1 almost everywhere under PrP_r and PgP_g. A 1-Lipschitz function that is differentiable has ∥∇f∥≤1\|\nabla f\|\le1 everywhere, and the optimal one sits on that bound along the lines that matter. Two consequences follow. A constraint that keeps the critic far below the bound (as clipping does) cannot represent f∗f^*. And it is enough to check the gradient norm on those lines, not everywhere in input space.

Why weight clipping fails

Capacity underuse. A clipped network that tries to reach its maximum slope ends up computing a very simple function. The paper trains critics to optimality on 8 Gaussians, 25 Gaussians and a Swiss roll, with the generator fixed at the data plus unit Gaussian noise. With clipping, the critic ignores the higher moments of the data distribution and learns a very simple approximation of the optimal critic. With the gradient penalty, its value surface follows the data. The paper also shows that clipping pushes the weights toward the two ends of the range, ±c\pm c.

Exploding and vanishing gradients. In a 12-layer ReLU critic without batch norm trained on the Swiss roll, the norm of the gradient of the critic loss with respect to successive layers either grows or decays exponentially toward the input, for each of c=10−1,10−2,10−3c=10^{-1},10^{-2},10^{-3}. The reason is simple. If a weight matrix of width nn has entries of magnitude cc, it scales a random vector by roughly cnc\sqrt n, and ReLU keeps about half the units, so each layer multiplies the backward signal by about cn/2c\sqrt{n/2}. Twelve layers raise that factor to the 12th power. Only a narrow band of cc gives a gain near 1, and it depends on the width.

The gradient penalty

L=Ex~∼Pg[D(x~)]−Ex∼Pr[D(x)]⏟original critic loss+λ Ex^∼Px^[(∥∇x^D(x^)∥2−1)2]⏟gradient penalty,x^=ϵx+(1−ϵ)x~,  ϵ∼U[0,1]L=\underbrace{\mathbb E_{\tilde x\sim P_g}\big[D(\tilde x)\big]-\mathbb E_{x\sim P_r}\big[D(x)\big]}_{\text{original critic loss}}+\underbrace{\lambda\,\mathbb E_{\hat x\sim P_{\hat x}}\Big[\big(\|\nabla_{\hat x}D(\hat x)\|_2-1\big)^2\Big]}_{\text{gradient penalty}},\qquad \hat x=\epsilon x+(1-\epsilon)\tilde x,\ \ \epsilon\sim U[0,1]
DD
the critic (the paper writes DD but it is a WGAN critic: no sigmoid, unbounded score)
x~=G(z)\tilde x=G(z)
a generated sample
x^\hat x
a random point on the segment between a real sample xx and a generated sample x~\tilde x, with a fresh ϵ\epsilon per pair
λ=10\lambda=10
penalty coefficient, used in all experiments of the paper from toy tasks to large ImageNet CNNs
(∥⋅∥−1)2(\|\cdot\|-1)^2
a two-sided penalty: it pushes the norm toward 1, not just below 1

A differentiable function is 1-Lipschitz if and only if its gradient norm is at most 1 everywhere. Enforcing that everywhere is intractable, so the penalty is a soft version on samples from Px^P_{\hat x}. Sampling on straight lines between real and fake points follows Proposition 1: the optimal critic has unit slope on lines between coupled points. Random pairs are not the optimal coupling, but the authors found that this sampling is enough in practice. The penalty is two-sided. The optimal critic has norm-1 gradients almost everywhere under PrP_r and PgP_g and in much of the space between, so pushing toward 1 constrains it very little. In their early tests the two-sided version did slightly better. In the appendix, over 174 sampled architectures, the two-sided penalty gave the higher Inception score in 100 trials and the one-sided max⁡(0,∥∇∥−1)2\max(0,\|\nabla\|-1)^2 in 77. The authors note that this difference is not statistically significant. Computing the penalty needs the gradient of DD with respect to its input inside the loss, so training back-propagates through a gradient (a "double backward").

No batch norm in the critic. Batch normalisation turns the critic into a function of the whole batch. The penalty, though, constrains the gradient with respect to each input on its own, so with batch norm the objective is no longer valid. The paper drops normalisation from the critic, or uses layer normalisation, which normalises each example independently. Batch norm in the generator is fine.

Settings and results (Algorithm 1, Sections 4–5)

Defaults
λ=10\lambda=10, ncritic=5n_{\text{critic}}=5, Adam with α=10−4\alpha=10^{-4}, β1=0\beta_1=0, β2=0.9\beta_2=0.9
Critic
no batch norm; layer norm recommended as a drop-in replacement. With BN or LN in the discriminator, a small weight decay (10⁻³) helped slightly
200 random architectures
32×32 ImageNet, depth 4–20, varied nonlinearities and normalisation. At an Inception-score threshold of 5.0: 147 trained only with WGAN-GP, 0 only with the standard GAN, 42 with both, 11 with neither
LSUN bedrooms
six hard architectures (no BN with constant filter count, 4-layer 512-unit MLP generator, no normalisation anywhere, gated nonlinearities, tanh, 101-layer ResNet) all trained with one set of hyperparameters; with DCGAN, LSGAN and clipped WGAN some of them were unstable or mode-collapsed
CIFAR-10 Inception score
unsupervised ResNet 7.86 ± 0.07 (state of the art at the time); with labels 8.42 ± 0.10, second only to SGAN (8.59)
Speed
converged faster and higher than weight clipping even with the same RMSProp optimiser; slower in wall-clock time than DCGAN but more stable at convergence
Text
a character-level GAN on the Billion Word dataset with a 1-D CNN generator emitting softmax outputs, with no sampling step
8 Gaussians · P_g = P_r + N(0, I) held fixed · gradient penalty λ = 10f high · f low · ● real · ● fakex̂ by ‖∇f‖: < 0.8 · 0.8–1.2 · > 1.2E_r f − E_g f (critic's W estimate)0.606mean ‖∇f(x̂)‖ ± sd on interpolates1.002 ± 0.105interpolates with ‖∇f‖ in [0.8, 1.2]94.1 %penalty E(‖∇f(x̂)‖ − 1)²0.011weights with |a_k| = max|a|1 of 24a 1-Lipschitz f has W estimate ≤ E‖noise‖ = 1.253critic weights a_k / max|a| (max |a| = 7.006)penalty: a_k spread out; magnitudes carry information12-layer ReLU critic, width 512, all weights ±c: ‖∂f/∂h_l‖ per layer l (log scale)1e-201e-101e0input 036912 outputc = 0.001c = 0.01c = 0.1He-scaled weightsper-layer gain ≈ c·√(512/2)c = 0.01: 0.160 per layerinput / output gradient3.3e-10He-scaled weights: 0.709GP: λ = 10 pulls ‖∇f(x̂)‖ toward 1 (94 % within 0.8–1.2); estimate 0.606 vs bound 1.253.Deep clipped critic: gain 0.16 per layer → gradients vanish toward the input.
constraint
data

The critic is a linear model on 24 random Fourier features, so clipping is solved exactly and the penalised objective (two-sided, λ(‖∇f(x̂)‖ − 1)² on x̂ = εx + (1 − ε)x̃) by Gauss–Newton. c sets both the clipped critic and the deep-network panel.

fig.The paper's Figure 1 setup in miniature. The generator is fixed at the data plus N(0, I) noise, and only the critic is trained. The critic here is a linear model on 24 random Fourier features. Clipping is then solved exactly: the box maximum puts every weight at +c or −c. The penalised objective is minimised by Gauss–Newton, with a tiny ridge so the finite model stays well-posed. On 8 Gaussians with λ = 10, the mean gradient norm on the interpolates x̂ is 1.002 ± 0.105, 94.1 % of them lie in [0.8, 1.2], the penalty is 0.011, and the critic's estimate E_r f − E_g f is 0.606. That is below 1.253 = E‖noise‖, an upper bound on W₁ here because moving each point back by its noise is one valid transport plan. A weak penalty fails: at λ = 0.1 the mean norm is 2.90 and the estimate 2.75 exceeds the bound, so the critic is not 1-Lipschitz. λ = 1 gives 1.15 (49 % in band). Weight clipping at c = 0.01 gives a mean norm of 0.0146, 0 % in band, and all 24 weights at ±c. Changing c only rescales that critic, because only the signs of the feature differences survive. On the Swiss roll and 25 Gaussians, λ = 10 gives mean norms of 1.000 and 0.976. The lower panel takes a 12-layer, width-512 ReLU critic with every weight at ±c (where clipping drives weights). The backward signal changes by about c·√(512/2) = 16c per layer. For c = 0.01 that is 0.16, and the input-layer gradient is 3.3·10⁻¹⁰ of the output-layer one. For c = 0.1 it is 1.6, an explosion by 334×. For c = 0.001 it is 3.3·10⁻²². He-scaled Gaussian weights keep the ratio at 0.71.

Worked example: one interpolate, its penalty, and its gradient

Take the critic D(x)=w0x1+w1x2+w2x12D(x)=w_0x_1+w_1x_2+w_2x_1^2 with w=(2, −1, 0.5)w=(2,\,-1,\,0.5), so ∇xD=(w0+2w2x1, w1)=(2+x1, −1)\nabla_xD=(w_0+2w_2x_1,\ w_1)=(2+x_1,\,-1). One real sample x=(1,0)x=(1,0), one generated x~=(−1,2)\tilde x=(-1,2), λ=10\lambda=10. Then D(x)=2.5D(x)=2.5 and D(x~)=−3.5D(\tilde x)=-3.5, so the original critic loss is D(x~)−D(x)=−6.0D(\tilde x)-D(x)=-6.0. Penalties along the segment (all checked with PyTorch autograd):

εx̂∇D(x̂)‖∇D(x̂)‖λ(‖∇D‖ − 1)²
0 (the fake point)(−1, 2)(1, −1)1.41421.7157
0.25(−0.5, 1.5)(1.5, −1)1.80286.4445
0.5(0, 1)(2, −1)2.236115.2786
1 (the real point)(1, 0)(3, −1)3.162346.7544
Interpolates x̂ = εx + (1 − ε)x̃ and their penalty terms

The loss for ε = 0.25 and what its gradient does

  1. 1L=D(x~)−D(x)+λ(∥∇D(x^)∥−1)2=−3.5−2.5+10 (1.8028−1)2=−6.0+6.4445=0.4445L=D(\tilde x)-D(x)+\lambda(\|\nabla D(\hat x)\|-1)^2=-3.5-2.5+10\,(1.8028-1)^2=-6.0+6.4445=0.4445.
  2. 2Gradient of the original loss with respect to w0w_0: x~1−x1=−2\tilde x_1-x_1=-2. Descending it makes DD steeper along x1x_1, which separates real from fake better.
  3. 3Gradient of the penalty with respect to w0w_0: 2λ(∥g∥−1)⋅∂∥g∥/∂w0=20⋅0.8028⋅(1.5/1.8028)=13.3592\lambda(\|g\|-1)\cdot\partial\|g\|/\partial w_0=20\cdot0.8028\cdot(1.5/1.8028)=13.359. In total, ∂L/∂w=(11.359, −6.906, −13.359)\partial L/\partial w=(11.359,\ -6.906,\ -13.359), the same as PyTorch's double backward. The penalty wins here, so the step flattens the critic.
  4. 4At equilibrium the two forces balance. The critic is as steep as it can be (raising ErD−EgD\mathbb E_rD-\mathbb E_gD) while its slope on the interpolates stays near 1. The figure shows this: 94 % of interpolates in [0.8,1.2][0.8,1.2] at λ=10\lambda=10.
python
1import torch, torch.nn as nn
2
3torch.manual_seed(0)
4
5def gradient_penalty(D, real, fake, lam=10.0):
6 eps = torch.rand(real.size(0), *([1] * (real.dim() - 1))) # one ε per example, broadcast over features
7 x_hat = (eps * real + (1 - eps) * fake).requires_grad_(True)
8 d_hat = D(x_hat)
9 grad, = torch.autograd.grad(d_hat.sum(), x_hat, create_graph=True) # keep the graph: penalty is differentiated again
10 norms = grad.flatten(1).norm(dim=1) # ‖∇_x̂ D(x̂)‖₂ per example
11 return lam * ((norms - 1) ** 2).mean(), norms
12
13def critic(d=2, h=256): # no BatchNorm: LayerNorm keeps examples independent
14 return nn.Sequential(nn.Linear(d, h), nn.LayerNorm(h), nn.LeakyReLU(0.2),
15 nn.Linear(h, h), nn.LayerNorm(h), nn.LeakyReLU(0.2), nn.Linear(h, 1))
16
17G = nn.Sequential(nn.Linear(2, 256), nn.ReLU(), nn.Linear(256, 256), nn.ReLU(), nn.Linear(256, 2))
18D = critic()
19opt_d = torch.optim.Adam(D.parameters(), lr=1e-4, betas=(0.0, 0.9)) # the paper's defaults
20opt_g = torch.optim.Adam(G.parameters(), lr=1e-4, betas=(0.0, 0.9))
21n_critic, m = 5, 256
22
23def real_batch(m): # 8 Gaussians, radius 2
24 k = torch.randint(0, 8, (m,)).float() * (torch.pi / 4)
25 return 2 * torch.stack([k.cos(), k.sin()], 1) + 0.05 * torch.randn(m, 2)
26
27for it in range(100):
28 for _ in range(n_critic):
29 x, z = real_batch(m), torch.randn(m, 2)
30 fake = G(z).detach()
31 gp, norms = gradient_penalty(D, x, fake)
32 loss_d = D(fake).mean() - D(x).mean() + gp # L = E D(x̃) − E D(x) + λ E(‖∇D(x̂)‖ − 1)²
33 opt_d.zero_grad(); loss_d.backward(); opt_d.step()
34 loss_g = -D(G(torch.randn(m, 2))).mean()
35 opt_g.zero_grad(); loss_g.backward(); opt_g.step()
36
37print(norms.shape, D(x).shape) # torch.Size([256]) torch.Size([256, 1])
38print(f"mean ‖∇D(x̂)‖ = {norms.mean().item():.3f}") # pulled toward 1 by the two-sided penalty
WGAN-GP on 8 Gaussians: penalty with create_graph=True, LayerNorm critic, the paper's defaults

Use when

  • +No clipping threshold to tune; λ=10\lambda=10 worked across toy data, CIFAR-10, LSUN and ImageNet in the paper
  • +Stable gradients in deep critics: trained 101-layer ResNets and architectures that failed with DCGAN, LSGAN and clipped WGAN
  • +Works with Adam (β1=0\beta_1=0, β2=0.9\beta_2=0.9); converges faster and higher than weight clipping
  • +The critic estimate remains a meaningful training curve and can detect overfitting

Avoid when

  • −Double backpropagation: each critic step is noticeably more expensive
  • −No batch norm in the critic; layer norm or no normalisation instead
  • −The constraint is soft and holds only on the sampled interpolates; with too small λ\lambda the critic is not Lipschitz (estimate above the true bound in the figure)
  • −Still ncritic=5n_{\text{critic}}=5 critic steps per generator step; slower in wall-clock time than DCGAN in the paper

Check yourself

Where is the WGAN-GP penalty evaluated?

Check yourself

Why does WGAN-GP remove batch normalisation from the critic?

Remember

  • ✓Critic loss E[D(x~)]−E[D(x)]+λ E[(∥∇x^D(x^)∥2−1)2]\mathbb E[D(\tilde x)]-\mathbb E[D(x)]+\lambda\,\mathbb E\big[(\|\nabla_{\hat x}D(\hat x)\|_2-1)^2\big] with x^=ϵx+(1−ϵ)x~\hat x=\epsilon x+(1-\epsilon)\tilde x, ϵ∼U[0,1]\epsilon\sim U[0,1].
  • ✓Defaults: λ=10\lambda=10, ncritic=5n_{\text{critic}}=5, Adam α=10−4\alpha=10^{-4}, β1=0\beta_1=0, β2=0.9\beta_2=0.9; no batch norm in the critic (layer norm instead).
  • ✓The optimal 1-Lipschitz critic has gradient norm 1 along the lines between coupled real and fake points (Proposition 1). That motivates both the sampling and the two-sided penalty.
  • ✓Weight clipping underuses capacity, pushes weights to ±c\pm c, and makes gradients in deep critics grow or decay by about cn/2c\sqrt{n/2} per layer.
  • ✓With one set of hyperparameters it trained 101-layer ResNets, MLP generators and normalisation-free GANs; CIFAR-10 Inception score 7.86 unsupervised, 8.42 with labels.
← back to Deep Learning roadmap