Definition
Wasserstein GAN (WGAN) is a generative adversarial network whose generator minimises the Earth-Mover (Wasserstein-1) distance between the data distribution and the model distribution, instead of the Jensen–Shannon divergence of the original GAN. The distance is estimated by a critic . The critic is a network with a real-valued output and no sigmoid, trained to maximise while its weights are clipped to a box so that stays -Lipschitz. Introduced by Martin Arjovsky, Soumith Chintala and Léon Bottou, "Wasserstein Generative Adversarial Networks" (ICML 2017).
The minimax game of the Generative adversarial network (GAN) minimises when the discriminator is optimal. In a companion paper (Arjovsky and Bottou, ICLR 2017) the same authors showed what goes wrong. Natural images lie near a low-dimensional manifold, and so do the outputs of a generator fed a low-dimensional . Two such manifolds almost never overlap in a set of positive measure. Then a perfect discriminator exists, no matter how close the manifolds are, and the gradient reaching the generator vanishes. The ("non-saturating") generator loss avoids the vanishing gradient, but its updates are unstable. WGAN replaces the divergence with a distance that stays finite, continuous and informative when the supports are disjoint. It also reported two practical gains: a loss value that tracks sample quality, and no mode collapse in any of its experiments.
Why the choice of distance matters
- all joint distributions (couplings, "transport plans") whose marginals are and
- how much mass is moved from to ; the cost is mass × distance
- the cost of the cheapest plan that turns into : the Earth-Mover distance
The paper's Example 1 makes the difference concrete. Let , the distribution of and that of : two parallel vertical segments. Then , while , and the total variation is for every . Only is continuous in , and only tells gradient descent which way to move. Theorem 1 of the paper generalises this. If is continuous in , so is . If is locally Lipschitz (with a mild moment condition, which holds for feed-forward networks), is continuous everywhere and differentiable almost everywhere. Neither statement holds for JS or KL. Theorem 2 shows that induces a weaker topology: every sequence that converges in KL, JS or total variation also converges in , but not the other way round.
The dual: a critic instead of a coupling
- is 1-Lipschitz: (Kantorovich–Rubinstein duality)
- the critic: a network with a scalar output and no sigmoid; it scores, it does not classify
- a compact set of weights, in the paper. Every is then -Lipschitz for some that depends only on , so the maximum is
- a maximiser of the dual problem (Theorem 3 of the paper; it exists under the same assumptions as Theorem 1)
- the generator's loss: it pushes samples uphill on the critic, along
The infimum over couplings is intractable in high dimensions. The supremum over 1-Lipschitz functions can be approximated by gradient ascent on a network, and Theorem 3 says that back-propagating through the critic gives the true gradient of , as long as the critic is optimal. That changes how training is run. A GAN discriminator trained to optimality gives a useless gradient. A WGAN critic should be trained close to optimality, because only then does approximate up to the constant . The optimal critic is also linear-like: between a real point and the fake point it is coupled with, its slope is exactly 1 (see WGAN-GP, which builds on this property).
The algorithm
- critic iterations per generator iteration
- clipping parameter: every weight is clamped to [−0.01, 0.01] after each critic update
- learning rate for both networks, with RMSProp
- batch size
Setup in the paper (Algorithm 1 and Section 4)
- Defaults
- , , , ; used in all experiments
- Optimiser
- RMSProp. With Adam () training was sometimes unstable: when the loss blew up, the cosine between the Adam step and the gradient usually turned negative. Momentum seemed to hurt with this nonstationary critic loss
- Critic
- a DCGAN discriminator without the final sigmoid, so the output is an unbounded score
- Data
- LSUN bedrooms (64×64)
- Generators tried
- DCGAN; DCGAN without batch norm and with a constant number of filters; a 4-layer ReLU MLP with 512 hidden units
- Reported
- the critic's estimate of W falls as sample quality improves; WGAN still produced samples with the no-batch-norm generator, where the standard GAN failed to learn, and with the MLP, where the standard GAN mode-collapsed; no mode collapse was observed in any WGAN experiment
The authors call weight clipping "a clearly terrible way to enforce a Lipschitz constraint". A large makes the weights slow to reach their limit, so the critic is hard to train to optimality. A small makes gradients vanish in deep networks or networks without batch norm. They kept clipping for its simplicity and its results. WGAN-GP later traced the failures to the clipping itself and replaced it with a gradient penalty.
Each generator loss uses the optimal discriminator or critic for the current θ, which is the regime WGAN aims for by training the critic n_critic times per step.
Worked example: W, its dual, and a generator step
Two real points and two generated points on a line, each with mass . In one dimension the optimal coupling matches sorted points: and . So . Now evaluate some critics (all values checked in Node):
| critic f | Lipschitz constant | E_r f | E_g f | E_r f − E_g f |
|---|---|---|---|---|
| |x − 1.5| | 1 | 2 | 0.5 | 1.5 (= W, optimal) |
| x | 1 | 2 | 1.5 | 0.5 |
| 0.01·x (linear, weight clipped to 0.01) | 0.01 | 0.02 | 0.015 | 0.005 |
| 2·|x − 1.5| | 2 (not allowed) | 4 | 1 | 3.0 |
What each critic tells the generator
- 1The dual is tight: is 1-Lipschitz and reaches . Without the constraint, doubling doubles the score (last row), so the supremum would be . That is why the constraint is essential.
- 2Generator step with the optimal critic: the loss moves each fake point along . and , so with step the fakes become . Each moves toward its coupled real point, and drops to , by exactly .
- 3JS sees nothing: and are both disjoint from , so before and after the step.
- 4A clipped linear critic , , picks (real mean 2 > fake mean 1.5). It pushes both fakes right by to , and stays at : one point moves closer, the other moves away. A critic with too little capacity gives a biased gradient. This is the failure that WGAN-GP measures on deep clipped critics.
1import torch, torch.nn as nn23torch.manual_seed(0)4def mlp(i, o, h=128):5 return nn.Sequential(nn.Linear(i, h), nn.ReLU(), nn.Linear(h, h), nn.ReLU(), nn.Linear(h, o))67def real_batch(m): # 8 Gaussians on a circle of radius 28 k = torch.randint(0, 8, (m,))9 ang = k.float() * (2 * torch.pi / 8)10 return 2 * torch.stack([ang.cos(), ang.sin()], 1) + 0.05 * torch.randn(m, 2)1112G, f = mlp(2, 2), mlp(2, 1) # generator g_θ and critic f_w (no sigmoid)13opt_g = torch.optim.RMSprop(G.parameters(), lr=5e-5)14opt_f = torch.optim.RMSprop(f.parameters(), lr=5e-5)15c, n_critic, m = 0.01, 5, 64 # the paper's defaults1617for it in range(200):18 for _ in range(n_critic):19 x, z = real_batch(m), torch.randn(m, 2)20 loss_f = -(f(x).mean() - f(G(z).detach()).mean()) # ascend E_r f − E_g f21 opt_f.zero_grad(); loss_f.backward(); opt_f.step()22 with torch.no_grad():23 for p in f.parameters():24 p.clamp_(-c, c) # weight clipping keeps f K-Lipschitz25 z = torch.randn(m, 2)26 loss_g = -f(G(z)).mean() # generator: descend −E_g f27 opt_g.zero_grad(); loss_g.backward(); opt_g.step()2829print(f(x).shape, G(z).shape) # torch.Size([64, 1]) torch.Size([64, 2])30w_est = -loss_f.item() # ≈ K · W(P_r, P_θ) for an unknown K31print(f"critic estimate {w_est:.5f}")32print(max(p.abs().max().item() for p in f.parameters()) <= c) # TrueUse when
- + is continuous and almost everywhere differentiable in even when supports are disjoint, so the generator always gets a gradient
- +The critic can be trained to optimality; no careful balancing of D and G
- +The critic's estimate decreases as samples improve, so it can be used to monitor training and compare hyperparameters within a run
- +Robust to the generator architecture: it trained a generator without batch norm and an MLP generator, which the standard GAN could not; no mode collapse observed in the paper
Avoid when
- −Weight clipping is a crude Lipschitz constraint: c is hard to set, deep clipped critics get vanishing or exploding gradients and learn overly simple functions
- − critic steps per generator step make each iteration several times slower
- −Slow convergence with the small RMSProp learning rate; momentum-based optimisers were unstable with clipping
- −The loss is with an unknown , and only for a near-optimal critic
Check yourself
P₀ is uniform on the segment {0} × [0, 1] and P_θ on {θ} × [0, 1]. For θ ≠ 0, which quantity changes with θ?
Check yourself
Why does the WGAN critic have no sigmoid and why are its weights clipped?
Remember
- ✓ (Kantorovich–Rubinstein). A critic approximates the sup, and .
- ✓WGAN defaults: weights clipped to , critic steps per generator step, RMSProp with , batch 64.
- ✓For disjoint supports JS is stuck at and KL is infinite, while is continuous and differentiable almost everywhere in (Theorem 1).
- ✓Train the critic close to optimality: it then gives the right gradient and does not saturate, and its estimate falls as sample quality improves.
- ✓Weight clipping is crude: the estimate is for an unknown , and deep clipped critics suffer vanishing or exploding gradients. WGAN-GP replaced it.