Wasserstein GAN (WGAN)

WGAN (Arjovsky, Chintala and Bottou, 2017) trains the generator to minimise the Earth-Mover distance W(P_r, P_θ). A critic f_w estimates it through the Kantorovich–Rubinstein dual sup over 1-Lipschitz f of E_r f − E_θ f. The critic is kept Lipschitz by clipping its weights to [−0.01, 0.01].

14 min read1077 wordsadvanced0 viewsupdated 2026-10-12

Definition

Wasserstein GAN (WGAN) is a generative adversarial network whose generator gθg_\theta minimises the Earth-Mover (Wasserstein-1) distance W(Pr,Pθ)W(P_r,P_\theta) between the data distribution and the model distribution, instead of the Jensen–Shannon divergence of the original GAN. The distance is estimated by a critic fwf_w. The critic is a network with a real-valued output and no sigmoid, trained to maximise EPr[fw(x)]−Ez[fw(gθ(z))]\mathbb E_{P_r}[f_w(x)]-\mathbb E_{z}[f_w(g_\theta(z))] while its weights are clipped to a box so that fwf_w stays KK-Lipschitz. Introduced by Martin Arjovsky, Soumith Chintala and Léon Bottou, "Wasserstein Generative Adversarial Networks" (ICML 2017).

The minimax game of the Generative adversarial network (GAN) minimises 2 JS(Pr,Pθ)−2log⁡22\,\mathrm{JS}(P_r,P_\theta)-2\log 2 when the discriminator is optimal. In a companion paper (Arjovsky and Bottou, ICLR 2017) the same authors showed what goes wrong. Natural images lie near a low-dimensional manifold, and so do the outputs of a generator fed a low-dimensional zz. Two such manifolds almost never overlap in a set of positive measure. Then a perfect discriminator exists, JS=log⁡2\mathrm{JS}=\log 2 no matter how close the manifolds are, and the gradient reaching the generator vanishes. The −log⁡D-\log D ("non-saturating") generator loss avoids the vanishing gradient, but its updates are unstable. WGAN replaces the divergence with a distance that stays finite, continuous and informative when the supports are disjoint. It also reported two practical gains: a loss value that tracks sample quality, and no mode collapse in any of its experiments.

Why the choice of distance matters

W(Pr,Pg)=inf⁡γ∈Π(Pr,Pg) E(x,y)∼γ[∥x−y∥]W(P_r,P_g)=\inf_{\gamma\in\Pi(P_r,P_g)}\ \mathbb E_{(x,y)\sim\gamma}\big[\|x-y\|\big]
Π(Pr,Pg)\Pi(P_r,P_g)
all joint distributions (couplings, "transport plans") whose marginals are PrP_r and PgP_g
γ(x,y)\gamma(x,y)
how much mass is moved from xx to yy; the cost is mass × distance
WW
the cost of the cheapest plan that turns PrP_r into PgP_g: the Earth-Mover distance

The paper's Example 1 makes the difference concrete. Let Z∼U[0,1]Z\sim U[0,1], P0P_0 the distribution of (0,Z)(0,Z) and PθP_\theta that of (θ,Z)(\theta,Z): two parallel vertical segments. Then W(P0,Pθ)=∣θ∣W(P_0,P_\theta)=|\theta|, while JS(P0,Pθ)=log⁡2\mathrm{JS}(P_0,P_\theta)=\log 2, KL(Pθ∥P0)=+∞\mathrm{KL}(P_\theta\|P_0)=+\infty and the total variation is 11 for every θ≠0\theta\neq 0. Only WW is continuous in θ\theta, and only WW tells gradient descent which way to move. Theorem 1 of the paper generalises this. If gθg_\theta is continuous in θ\theta, so is W(Pr,Pθ)W(P_r,P_\theta). If gθg_\theta is locally Lipschitz (with a mild moment condition, which holds for feed-forward networks), WW is continuous everywhere and differentiable almost everywhere. Neither statement holds for JS or KL. Theorem 2 shows that WW induces a weaker topology: every sequence that converges in KL, JS or total variation also converges in WW, but not the other way round.

The dual: a critic instead of a coupling

W(Pr,Pθ)=sup⁡∥f∥L≤1 Ex∼Pr[f(x)]−Ex∼Pθ[f(x)]⟹max⁡w∈W Ex∼Pr[fw(x)]−Ez∼p(z)[fw(gθ(z))]W(P_r,P_\theta)=\sup_{\|f\|_L\le 1}\ \mathbb E_{x\sim P_r}[f(x)]-\mathbb E_{x\sim P_\theta}[f(x)]\qquad\Longrightarrow\qquad \max_{w\in\mathcal W}\ \mathbb E_{x\sim P_r}[f_w(x)]-\mathbb E_{z\sim p(z)}[f_w(g_\theta(z))]
∥f∥L≤1\|f\|_L\le 1
ff is 1-Lipschitz: ∣f(x)−f(y)∣≤∥x−y∥|f(x)-f(y)|\le\|x-y\| (Kantorovich–Rubinstein duality)
fwf_w
the critic: a network with a scalar output and no sigmoid; it scores, it does not classify
W\mathcal W
a compact set of weights, [−c,c]l[-c,c]^l in the paper. Every fwf_w is then KK-Lipschitz for some KK that depends only on W\mathcal W, so the maximum is K⋅W(Pr,Pθ)K\cdot W(P_r,P_\theta)
∇θW(Pr,Pθ)=− Ez∼p(z)[∇θf(gθ(z))]\nabla_\theta W(P_r,P_\theta)=-\,\mathbb E_{z\sim p(z)}\big[\nabla_\theta f(g_\theta(z))\big]
ff
a maximiser of the dual problem (Theorem 3 of the paper; it exists under the same assumptions as Theorem 1)
−f(gθ(z))-f(g_\theta(z))
the generator's loss: it pushes samples uphill on the critic, along ∇xf\nabla_x f

The infimum over couplings is intractable in high dimensions. The supremum over 1-Lipschitz functions can be approximated by gradient ascent on a network, and Theorem 3 says that back-propagating through the critic gives the true gradient of WW, as long as the critic is optimal. That changes how training is run. A GAN discriminator trained to optimality gives a useless gradient. A WGAN critic should be trained close to optimality, because only then does −E[fw(gθ(z))]-\mathbb E[f_w(g_\theta(z))] approximate WW up to the constant KK. The optimal critic is also linear-like: between a real point and the fake point it is coupled with, its slope is exactly 1 (see WGAN-GP, which builds on this property).

The algorithm

repeat ncritic times:gw=∇w[1m∑ifw(x(i))−1m∑ifw(gθ(z(i)))],w←w+α⋅RMSProp(w,gw),w←clip(w,−c,c)then:gθ=−∇θ1m∑ifw(gθ(z(i))),θ←θ−α⋅RMSProp(θ,gθ)\begin{aligned}&\text{repeat } n_{\text{critic}}\text{ times:}\quad g_w=\nabla_w\Big[\tfrac1m\textstyle\sum_i f_w(x^{(i)})-\tfrac1m\sum_i f_w(g_\theta(z^{(i)}))\Big],\quad w\leftarrow w+\alpha\cdot\mathrm{RMSProp}(w,g_w),\quad w\leftarrow\mathrm{clip}(w,-c,c)\\&\text{then:}\quad g_\theta=-\nabla_\theta\tfrac1m\textstyle\sum_i f_w(g_\theta(z^{(i)})),\quad \theta\leftarrow\theta-\alpha\cdot\mathrm{RMSProp}(\theta,g_\theta)\end{aligned}
ncritic=5n_{\text{critic}}=5
critic iterations per generator iteration
c=0.01c=0.01
clipping parameter: every weight is clamped to [−0.01, 0.01] after each critic update
α=5⋅10−5\alpha=5\cdot10^{-5}
learning rate for both networks, with RMSProp
m=64m=64
batch size

Setup in the paper (Algorithm 1 and Section 4)

Defaults
α=0.00005\alpha=0.00005, c=0.01c=0.01, m=64m=64, ncritic=5n_{\text{critic}}=5; used in all experiments
Optimiser
RMSProp. With Adam (β1>0\beta_1>0) training was sometimes unstable: when the loss blew up, the cosine between the Adam step and the gradient usually turned negative. Momentum seemed to hurt with this nonstationary critic loss
Critic
a DCGAN discriminator without the final sigmoid, so the output is an unbounded score
Data
LSUN bedrooms (64×64)
Generators tried
DCGAN; DCGAN without batch norm and with a constant number of filters; a 4-layer ReLU MLP with 512 hidden units
Reported
the critic's estimate of W falls as sample quality improves; WGAN still produced samples with the no-batch-norm generator, where the standard GAN failed to learn, and with the MLP, where the standard GAN mode-collapsed; no mode collapse was observed in any WGAN experiment

The authors call weight clipping "a clearly terrible way to enforce a Lipschitz constraint". A large cc makes the weights slow to reach their limit, so the critic is hard to train to optimality. A small cc makes gradients vanish in deep networks or networks without batch norm. They kept clipping for its simplicity and its results. WGAN-GP later traced the failures to the clipping itself and replaced it with a gradient penalty.

Pᵣ = N(0, σ²) · P_θ = N(θ, σ²) · σ = 0.1 · overlap shrinks as σ → 0−4−2024pᵣp_θ (peak-scaled)D*(x) = pᵣ/(pᵣ+p_θ)critic f*(x)−4−2024distance vs generator parameter θ: W₁/4 = |θ|/4 · JS/log 2W₁ = |θ|2.000JS (max 0.693)0.693KL(P_θ‖Pᵣ)200D* at x = θ< 1e-9|∂W/∂θ|1|∂(2JS)/∂θ|< 1e-9|∂(−log D)/∂θ|200θ after 20 steps (η = 0.05)WGAN: 1.000minimax: 2.000−log D: off plot (|θ|>4)WGAN: f* = −sign(θ)·x has slope 1 everywhere, so θ moves exactly 0.05 per step.minimax: D* ≈ 0 on the fakes and JS → log 2, so |2·∂JS/∂θ| < 1e-9 (flat loss).−log D: ∂/∂θ[KL(P_θ‖Pᵣ) − 2JS] = θ/σ² − 2∂JS/∂θ = 200; explodes for small σ.dots under the top axis: θ₀ … θ_k for WGAN, minimax, −log D (one row each).
preset

Each generator loss uses the optimal discriminator or critic for the current θ, which is the regime WGAN aims for by training the critic n_critic times per step.

fig.Example 1 of the paper in one dimension. Real data is N(0, σ²) and the generator is N(θ, σ²), a pure shift, so W₁ = |θ| exactly and the supports become disjoint as σ → 0. The top panel shows the two densities, the optimal GAN discriminator D*(x) = pᵣ/(pᵣ + p_θ), and the optimal 1-Lipschitz critic f*(x) = −sign(θ)·x. The bottom panel shows W₁ and JS as functions of θ. Start at θ = 2, σ = 0.1. JS = 0.693, which is log 2 to three digits. KL(P_θ‖Pᵣ) = 200. The optimal D* scores the fake mode at 1.4·10⁻⁸⁷, so the minimax gradient |∂(2JS)/∂θ| is below 10⁻⁹, the readout's floor. After 20 gradient steps with η = 0.05, the minimax generator is still at θ = 2.000. The −log D generator gets a gradient of 200 and overshoots: θ goes 2 → −8 → 32 → −128 and leaves the plot. The WGAN generator gets slope 1 everywhere and moves exactly 0.05 per step, reaching θ = 1.000. With overlapping supports (σ = 0.5) all three work: JS = 0.633, the minimax gradient is 0.274, and after 20 steps θ = 1.637 (minimax), 0.128 (−log D) and 1.000 (WGAN). Every loss uses the optimal D or critic for the current θ, which is the regime WGAN's n_critic loop aims for.

Worked example: W, its dual, and a generator step

Two real points {0, 4}\{0,\,4\} and two generated points {1, 2}\{1,\,2\} on a line, each with mass 12\tfrac12. In one dimension the optimal coupling matches sorted points: 0↔10\leftrightarrow1 and 4↔24\leftrightarrow2. So W=12(∣0−1∣+∣4−2∣)=1.5W=\tfrac12(|0-1|+|4-2|)=1.5. Now evaluate some critics (all values checked in Node):

critic fLipschitz constantE_r fE_g fE_r f − E_g f
|x − 1.5|120.51.5 (= W, optimal)
x121.50.5
0.01·x (linear, weight clipped to 0.01)0.010.020.0150.005
2·|x − 1.5|2 (not allowed)413.0
Critic candidates for real {0, 4} vs fake {1, 2}

What each critic tells the generator

  1. 1The dual is tight: f(x)=∣x−1.5∣f(x)=|x-1.5| is 1-Lipschitz and reaches 1.5=W1.5=W. Without the constraint, doubling ff doubles the score (last row), so the supremum would be +∞+\infty. That is why the constraint is essential.
  2. 2Generator step with the optimal critic: the loss −12∑f(x~)-\tfrac12\sum f(\tilde x) moves each fake point along f′(x~)f'(\tilde x). f′(1)=−1f'(1)=-1 and f′(2)=+1f'(2)=+1, so with step η=0.1\eta=0.1 the fakes become {0.9, 2.1}\{0.9,\,2.1\}. Each moves toward its coupled real point, and WW drops to 12(0.9+1.9)=1.4\tfrac12(0.9+1.9)=1.4, by exactly η\eta.
  3. 3JS sees nothing: {1,2}\{1,2\} and {0.9,2.1}\{0.9,2.1\} are both disjoint from {0,4}\{0,4\}, so JS=log⁡2\mathrm{JS}=\log 2 before and after the step.
  4. 4A clipped linear critic f=wxf=wx, ∣w∣≤0.01|w|\le0.01, picks w=0.01w=0.01 (real mean 2 > fake mean 1.5). It pushes both fakes right by 0.1×0.010.1\times0.01 to {1.001, 2.001}\{1.001,\,2.001\}, and WW stays at 1.51.5: one point moves closer, the other moves away. A critic with too little capacity gives a biased gradient. This is the failure that WGAN-GP measures on deep clipped critics.
python
1import torch, torch.nn as nn
2
3torch.manual_seed(0)
4def mlp(i, o, h=128):
5 return nn.Sequential(nn.Linear(i, h), nn.ReLU(), nn.Linear(h, h), nn.ReLU(), nn.Linear(h, o))
6
7def real_batch(m): # 8 Gaussians on a circle of radius 2
8 k = torch.randint(0, 8, (m,))
9 ang = k.float() * (2 * torch.pi / 8)
10 return 2 * torch.stack([ang.cos(), ang.sin()], 1) + 0.05 * torch.randn(m, 2)
11
12G, f = mlp(2, 2), mlp(2, 1) # generator g_θ and critic f_w (no sigmoid)
13opt_g = torch.optim.RMSprop(G.parameters(), lr=5e-5)
14opt_f = torch.optim.RMSprop(f.parameters(), lr=5e-5)
15c, n_critic, m = 0.01, 5, 64 # the paper's defaults
16
17for it in range(200):
18 for _ in range(n_critic):
19 x, z = real_batch(m), torch.randn(m, 2)
20 loss_f = -(f(x).mean() - f(G(z).detach()).mean()) # ascend E_r f − E_g f
21 opt_f.zero_grad(); loss_f.backward(); opt_f.step()
22 with torch.no_grad():
23 for p in f.parameters():
24 p.clamp_(-c, c) # weight clipping keeps f K-Lipschitz
25 z = torch.randn(m, 2)
26 loss_g = -f(G(z)).mean() # generator: descend −E_g f
27 opt_g.zero_grad(); loss_g.backward(); opt_g.step()
28
29print(f(x).shape, G(z).shape) # torch.Size([64, 1]) torch.Size([64, 2])
30w_est = -loss_f.item() # ≈ K · W(P_r, P_θ) for an unknown K
31print(f"critic estimate {w_est:.5f}")
32print(max(p.abs().max().item() for p in f.parameters()) <= c) # True
WGAN with weight clipping on a 2-D toy (8 Gaussians), the paper's defaults

Use when

  • +WW is continuous and almost everywhere differentiable in θ\theta even when supports are disjoint, so the generator always gets a gradient
  • +The critic can be trained to optimality; no careful balancing of D and G
  • +The critic's estimate decreases as samples improve, so it can be used to monitor training and compare hyperparameters within a run
  • +Robust to the generator architecture: it trained a generator without batch norm and an MLP generator, which the standard GAN could not; no mode collapse observed in the paper

Avoid when

  • −Weight clipping is a crude Lipschitz constraint: c is hard to set, deep clipped critics get vanishing or exploding gradients and learn overly simple functions
  • −ncritic=5n_{\text{critic}}=5 critic steps per generator step make each iteration several times slower
  • −Slow convergence with the small RMSProp learning rate; momentum-based optimisers were unstable with clipping
  • −The loss is K⋅WK\cdot W with an unknown KK, and only for a near-optimal critic

Check yourself

P₀ is uniform on the segment {0} × [0, 1] and P_θ on {θ} × [0, 1]. For θ ≠ 0, which quantity changes with θ?

Check yourself

Why does the WGAN critic have no sigmoid and why are its weights clipped?

Remember

  • ✓W(Pr,Pθ)=sup⁡∥f∥L≤1EPrf−EPθfW(P_r,P_\theta)=\sup_{\|f\|_L\le1}\mathbb E_{P_r}f-\mathbb E_{P_\theta}f (Kantorovich–Rubinstein). A critic fwf_w approximates the sup, and ∇θW=−Ez∇θf(gθ(z))\nabla_\theta W=-\mathbb E_z\nabla_\theta f(g_\theta(z)).
  • ✓WGAN defaults: weights clipped to [−0.01,0.01][-0.01,0.01], ncritic=5n_{\text{critic}}=5 critic steps per generator step, RMSProp with α=5⋅10−5\alpha=5\cdot10^{-5}, batch 64.
  • ✓For disjoint supports JS is stuck at log⁡2\log 2 and KL is infinite, while WW is continuous and differentiable almost everywhere in θ\theta (Theorem 1).
  • ✓Train the critic close to optimality: it then gives the right gradient and does not saturate, and its estimate falls as sample quality improves.
  • ✓Weight clipping is crude: the estimate is K⋅WK\cdot W for an unknown KK, and deep clipped critics suffer vanishing or exploding gradients. WGAN-GP replaced it.
← back to Deep Learning roadmap