# Wasserstein GAN (WGAN)

> WGAN (Arjovsky, Chintala and Bottou, 2017) trains the generator to minimise the Earth-Mover distance W(P_r, P_θ). A critic f_w estimates it through the Kantorovich–Rubinstein dual sup over 1-Lipschitz f of E_r f − E_θ f. The critic is kept Lipschitz by clipping its weights to [−0.01, 0.01].

Source: https://www.mlfoundry.in/learn/wasserstein-gan-wgan  
Section: Deep Learning › Generative Deep Learning Algorithms  
Level: advanced · 14 min read · updated 2026-10-12

**Wasserstein GAN (WGAN):** is a generative adversarial network whose generator $g_\theta$ minimises the **Earth-Mover (Wasserstein-1) distance** $W(P_r,P_\theta)$ between the data distribution and the model distribution, instead of the Jensen–Shannon divergence of the original GAN. The distance is estimated by a **critic** $f_w$. The critic is a network with a real-valued output and no sigmoid, trained to maximise $\mathbb E_{P_r}[f_w(x)]-\mathbb E_{z}[f_w(g_\theta(z))]$ while its weights are clipped to a box so that $f_w$ stays $K$-Lipschitz. Introduced by Martin Arjovsky, Soumith Chintala and Léon Bottou, "Wasserstein Generative Adversarial Networks" (ICML 2017).

The minimax game of the **Generative adversarial network (GAN)** minimises $2\,\mathrm{JS}(P_r,P_\theta)-2\log 2$ when the discriminator is optimal. In a companion paper (Arjovsky and Bottou, ICLR 2017) the same authors showed what goes wrong. Natural images lie near a low-dimensional manifold, and so do the outputs of a generator fed a low-dimensional $z$. Two such manifolds almost never overlap in a set of positive measure. Then a perfect discriminator exists, $\mathrm{JS}=\log 2$ no matter how close the manifolds are, and the gradient reaching the generator vanishes. The $-\log D$ ("non-saturating") generator loss avoids the vanishing gradient, but its updates are unstable. WGAN replaces the divergence with a distance that stays finite, continuous and informative when the supports are disjoint. It also reported two practical gains: a loss value that tracks sample quality, and no mode collapse in any of its experiments.

## Why the choice of distance matters

$$
W(P_r,P_g)=\inf_{\gamma\in\Pi(P_r,P_g)}\ \mathbb E_{(x,y)\sim\gamma}\big[\|x-y\|\big]
$$

- $\Pi(P_r,P_g)$: all joint distributions (couplings, "transport plans") whose marginals are $P_r$ and $P_g$
- $\gamma(x,y)$: how much mass is moved from $x$ to $y$; the cost is mass × distance
- $W$: the cost of the cheapest plan that turns $P_r$ into $P_g$: the Earth-Mover distance

The paper's Example 1 makes the difference concrete. Let $Z\sim U[0,1]$, $P_0$ the distribution of $(0,Z)$ and $P_\theta$ that of $(\theta,Z)$: two parallel vertical segments. Then $W(P_0,P_\theta)=|\theta|$, while $\mathrm{JS}(P_0,P_\theta)=\log 2$, $\mathrm{KL}(P_\theta\|P_0)=+\infty$ and the total variation is $1$ for every $\theta\neq 0$. Only $W$ is continuous in $\theta$, and only $W$ tells gradient descent which way to move. Theorem 1 of the paper generalises this. If $g_\theta$ is continuous in $\theta$, so is $W(P_r,P_\theta)$. If $g_\theta$ is locally Lipschitz (with a mild moment condition, which holds for feed-forward networks), $W$ is continuous everywhere and differentiable almost everywhere. Neither statement holds for JS or KL. Theorem 2 shows that $W$ induces a weaker topology: every sequence that converges in KL, JS or total variation also converges in $W$, but not the other way round.

## The dual: a critic instead of a coupling

$$
W(P_r,P_\theta)=\sup_{\|f\|_L\le 1}\ \mathbb E_{x\sim P_r}[f(x)]-\mathbb E_{x\sim P_\theta}[f(x)]\qquad\Longrightarrow\qquad \max_{w\in\mathcal W}\ \mathbb E_{x\sim P_r}[f_w(x)]-\mathbb E_{z\sim p(z)}[f_w(g_\theta(z))]
$$

- $\|f\|_L\le 1$: $f$ is 1-Lipschitz: $|f(x)-f(y)|\le\|x-y\|$ (Kantorovich–Rubinstein duality)
- $f_w$: the **critic**: a network with a scalar output and no sigmoid; it scores, it does not classify
- $\mathcal W$: a compact set of weights, $[-c,c]^l$ in the paper. Every $f_w$ is then $K$-Lipschitz for some $K$ that depends only on $\mathcal W$, so the maximum is $K\cdot W(P_r,P_\theta)$

$$
\nabla_\theta W(P_r,P_\theta)=-\,\mathbb E_{z\sim p(z)}\big[\nabla_\theta f(g_\theta(z))\big]
$$

- $f$: a maximiser of the dual problem (Theorem 3 of the paper; it exists under the same assumptions as Theorem 1)
- $-f(g_\theta(z))$: the generator's loss: it pushes samples uphill on the critic, along $\nabla_x f$

The infimum over couplings is intractable in high dimensions. The supremum over 1-Lipschitz functions can be approximated by gradient ascent on a network, and Theorem 3 says that back-propagating through the critic gives the true gradient of $W$, as long as the critic is optimal. That changes how training is run. A GAN discriminator trained to optimality gives a useless gradient. A WGAN critic should be trained close to optimality, because only then does $-\mathbb E[f_w(g_\theta(z))]$ approximate $W$ up to the constant $K$. The optimal critic is also linear-like: between a real point and the fake point it is coupled with, its slope is exactly 1 (see **WGAN-GP**, which builds on this property).

## The algorithm

$$
\begin{aligned}&\text{repeat } n_{\text{critic}}\text{ times:}\quad g_w=\nabla_w\Big[\tfrac1m\textstyle\sum_i f_w(x^{(i)})-\tfrac1m\sum_i f_w(g_\theta(z^{(i)}))\Big],\quad w\leftarrow w+\alpha\cdot\mathrm{RMSProp}(w,g_w),\quad w\leftarrow\mathrm{clip}(w,-c,c)\\&\text{then:}\quad g_\theta=-\nabla_\theta\tfrac1m\textstyle\sum_i f_w(g_\theta(z^{(i)})),\quad \theta\leftarrow\theta-\alpha\cdot\mathrm{RMSProp}(\theta,g_\theta)\end{aligned}
$$

- $n_{\text{critic}}=5$: critic iterations per generator iteration
- $c=0.01$: clipping parameter: every weight is clamped to [−0.01, 0.01] after each critic update
- $\alpha=5\cdot10^{-5}$: learning rate for both networks, with RMSProp
- $m=64$: batch size

**Setup in the paper (Algorithm 1 and Section 4)**

- **Defaults:** $\alpha=0.00005$, $c=0.01$, $m=64$, $n_{\text{critic}}=5$; used in all experiments
- **Optimiser:** RMSProp. With Adam ($\beta_1>0$) training was sometimes unstable: when the loss blew up, the cosine between the Adam step and the gradient usually turned negative. Momentum seemed to hurt with this nonstationary critic loss
- **Critic:** a DCGAN discriminator without the final sigmoid, so the output is an unbounded score
- **Data:** LSUN bedrooms (64×64)
- **Generators tried:** DCGAN; DCGAN without batch norm and with a constant number of filters; a 4-layer ReLU MLP with 512 hidden units
- **Reported:** the critic's estimate of W falls as sample quality improves; WGAN still produced samples with the no-batch-norm generator, where the standard GAN failed to learn, and with the MLP, where the standard GAN mode-collapsed; no mode collapse was observed in any WGAN experiment

The authors call weight clipping "a clearly terrible way to enforce a Lipschitz constraint". A large $c$ makes the weights slow to reach their limit, so the critic is hard to train to optimality. A small $c$ makes gradients vanish in deep networks or networks without batch norm. They kept clipping for its simplicity and its results. **WGAN-GP** later traced the failures to the clipping itself and replaced it with a gradient penalty.

> Figure: Example 1 of the paper in one dimension. Real data is N(0, σ²) and the generator is N(θ, σ²), a pure shift, so W₁ = |θ| exactly and the supports become disjoint as σ → 0. The top panel shows the two densities, the optimal GAN discriminator D*(x) = pᵣ/(pᵣ + p_θ), and the optimal 1-Lipschitz critic f*(x) = −sign(θ)·x. The bottom panel shows W₁ and JS as functions of θ. Start at θ = 2, σ = 0.1. JS = 0.693, which is log 2 to three digits. KL(P_θ‖Pᵣ) = 200. The optimal D* scores the fake mode at 1.4·10⁻⁸⁷, so the minimax gradient |∂(2JS)/∂θ| is below 10⁻⁹, the readout's floor. After 20 gradient steps with η = 0.05, the minimax generator is still at θ = 2.000. The −log D generator gets a gradient of 200 and overshoots: θ goes 2 → −8 → 32 → −128 and leaves the plot. The WGAN generator gets slope 1 everywhere and moves exactly 0.05 per step, reaching θ = 1.000. With overlapping supports (σ = 0.5) all three work: JS = 0.633, the minimax gradient is 0.274, and after 20 steps θ = 1.637 (minimax), 0.128 (−log D) and 1.000 (WGAN). Every loss uses the optimal D or critic for the current θ, which is the regime WGAN's n_critic loop aims for.

## Worked example: W, its dual, and a generator step

Two real points $\{0,\,4\}$ and two generated points $\{1,\,2\}$ on a line, each with mass $\tfrac12$. In one dimension the optimal coupling matches sorted points: $0\leftrightarrow1$ and $4\leftrightarrow2$. So $W=\tfrac12(|0-1|+|4-2|)=1.5$. Now evaluate some critics (all values checked in Node):

Critic candidates for real {0, 4} vs fake {1, 2}

| critic f | Lipschitz constant | E_r f | E_g f | E_r f − E_g f |
| --- | --- | --- | --- | --- |
| \|x − 1.5\| | 1 | 2 | 0.5 | 1.5 (= W, optimal) |
| x | 1 | 2 | 1.5 | 0.5 |
| 0.01·x (linear, weight clipped to 0.01) | 0.01 | 0.02 | 0.015 | 0.005 |
| 2·\|x − 1.5\| | 2 (not allowed) | 4 | 1 | 3.0 |

**What each critic tells the generator**

1. The dual is tight: $f(x)=|x-1.5|$ is 1-Lipschitz and reaches $1.5=W$. Without the constraint, doubling $f$ doubles the score (last row), so the supremum would be $+\infty$. That is why the constraint is essential.
2. Generator step with the optimal critic: the loss $-\tfrac12\sum f(\tilde x)$ moves each fake point along $f'(\tilde x)$. $f'(1)=-1$ and $f'(2)=+1$, so with step $\eta=0.1$ the fakes become $\{0.9,\,2.1\}$. Each moves toward its coupled real point, and $W$ drops to $\tfrac12(0.9+1.9)=1.4$, by exactly $\eta$.
3. JS sees nothing: $\{1,2\}$ and $\{0.9,2.1\}$ are both disjoint from $\{0,4\}$, so $\mathrm{JS}=\log 2$ before and after the step.
4. A clipped linear critic $f=wx$, $|w|\le0.01$, picks $w=0.01$ (real mean 2 > fake mean 1.5). It pushes both fakes right by $0.1\times0.01$ to $\{1.001,\,2.001\}$, and $W$ stays at $1.5$: one point moves closer, the other moves away. A critic with too little capacity gives a biased gradient. This is the failure that **WGAN-GP** measures on deep clipped critics.

```python
import torch, torch.nn as nn

torch.manual_seed(0)
def mlp(i, o, h=128):
    return nn.Sequential(nn.Linear(i, h), nn.ReLU(), nn.Linear(h, h), nn.ReLU(), nn.Linear(h, o))

def real_batch(m):                                        # 8 Gaussians on a circle of radius 2
    k = torch.randint(0, 8, (m,))
    ang = k.float() * (2 * torch.pi / 8)
    return 2 * torch.stack([ang.cos(), ang.sin()], 1) + 0.05 * torch.randn(m, 2)

G, f = mlp(2, 2), mlp(2, 1)                               # generator g_θ and critic f_w (no sigmoid)
opt_g = torch.optim.RMSprop(G.parameters(), lr=5e-5)
opt_f = torch.optim.RMSprop(f.parameters(), lr=5e-5)
c, n_critic, m = 0.01, 5, 64                              # the paper's defaults

for it in range(200):
    for _ in range(n_critic):
        x, z = real_batch(m), torch.randn(m, 2)
        loss_f = -(f(x).mean() - f(G(z).detach()).mean())  # ascend E_r f − E_g f
        opt_f.zero_grad(); loss_f.backward(); opt_f.step()
        with torch.no_grad():
            for p in f.parameters():
                p.clamp_(-c, c)                           # weight clipping keeps f K-Lipschitz
    z = torch.randn(m, 2)
    loss_g = -f(G(z)).mean()                              # generator: descend −E_g f
    opt_g.zero_grad(); loss_g.backward(); opt_g.step()

print(f(x).shape, G(z).shape)                    # torch.Size([64, 1]) torch.Size([64, 2])
w_est = -loss_f.item()                           # ≈ K · W(P_r, P_θ) for an unknown K
print(f"critic estimate {w_est:.5f}")
print(max(p.abs().max().item() for p in f.parameters()) <= c)   # True
```

WGAN with weight clipping on a 2-D toy (8 Gaussians), the paper's defaults

> **The loss is W only up to an unknown factor K:** With clipping, the critic's maximum is $K\cdot W$, and $K$ depends on $c$ and on the architecture. The estimate is comparable across iterations of one run, which is what the paper's curves show. It is not comparable across architectures or values of $c$, and it is not an absolute distance. It is also only meaningful when the critic is near its optimum. If $n_{\text{critic}}$ is too small, the "W curve" measures how undertrained the critic is.

> **Momentum and the critic:** The paper switched from Adam to RMSProp after seeing the Adam step turn against the gradient when training became unstable. With weight clipping this matters. Later work (**WGAN-GP**) made Adam work again with $\beta_1=0$ and a gradient penalty, so the advice is tied to clipping, not universal.

> **Train the critic to optimality, on purpose:** For a standard GAN a perfect discriminator saturates and gives the generator no gradient (the viz: $D^*\approx10^{-87}$ on the fakes). For a WGAN the optimal critic is exactly what supplies the right gradient ($\nabla_\theta W=-\mathbb E[\nabla_\theta f(g_\theta(z))]$), and its slope never saturates. So WGAN can, and should, run several critic steps per generator step. That removes the need to balance the two networks carefully.

> **Where the critic idea went next:** Clipping was replaced by a gradient penalty in **WGAN-GP** and later by spectral normalisation, which caps each layer's largest singular value at 1 (used by **BigGAN**). Optimal-transport ideas return in **Flow matching** (its optimal-transport conditional paths) and in **Rectified flow**, which straightens the coupling between noise and data.

**Strengths**

- $W$ is continuous and almost everywhere differentiable in $\theta$ even when supports are disjoint, so the generator always gets a gradient
- The critic can be trained to optimality; no careful balancing of D and G
- The critic's estimate decreases as samples improve, so it can be used to monitor training and compare hyperparameters within a run
- Robust to the generator architecture: it trained a generator without batch norm and an MLP generator, which the standard GAN could not; no mode collapse observed in the paper

**Limitations**

- Weight clipping is a crude Lipschitz constraint: c is hard to set, deep clipped critics get vanishing or exploding gradients and learn overly simple functions
- $n_{\text{critic}}=5$ critic steps per generator step make each iteration several times slower
- Slow convergence with the small RMSProp learning rate; momentum-based optimisers were unstable with clipping
- The loss is $K\cdot W$ with an unknown $K$, and only for a near-optimal critic

**Question:** P₀ is uniform on the segment {0} × [0, 1] and P_θ on {θ} × [0, 1]. For θ ≠ 0, which quantity changes with θ?

- A. JS(P₀, P_θ)
- B. KL(P_θ‖P₀)
- C. W(P₀, P_θ)
- D. Total variation distance

**Answer:** C. W(P₀, P_θ) — $W=|\theta|$: every unit of mass moves a distance $|\theta|$. JS is stuck at $\log 2$, KL is $+\infty$ and total variation is $1$ for every $\theta\ne0$, so none of them gives gradient descent a direction. This is Example 1 of the paper.

**Question:** Why does the WGAN critic have no sigmoid and why are its weights clipped?

- A. The sigmoid is too slow; clipping speeds up training
- B. It estimates sup over Lipschitz f of E_r f − E_g f, so its output is an unbounded score; clipping to a compact box keeps every f_w K-Lipschitz, otherwise the supremum is infinite
- C. Clipping makes the critic a probability
- D. Both are needed only for batch normalisation

**Answer:** B. It estimates sup over Lipschitz f of E_r f − E_g f, so its output is an unbounded score; clipping to a compact box keeps every f_w K-Lipschitz, otherwise the supremum is infinite — The critic is a function in the Kantorovich–Rubinstein dual, not a classifier. Without a Lipschitz constraint, scaling $f$ by any factor scales $\mathbb E_rf-\mathbb E_gf$ by it (the $2|x-1.5|$ row of the worked example). Clipping weights to $[-c,c]$ bounds the Lipschitz constant by some $K$, and the critic then estimates $K\cdot W$.

**Key points**

- $W(P_r,P_\theta)=\sup_{\|f\|_L\le1}\mathbb E_{P_r}f-\mathbb E_{P_\theta}f$ (Kantorovich–Rubinstein). A critic $f_w$ approximates the sup, and $\nabla_\theta W=-\mathbb E_z\nabla_\theta f(g_\theta(z))$.
- WGAN defaults: weights clipped to $[-0.01,0.01]$, $n_{\text{critic}}=5$ critic steps per generator step, RMSProp with $\alpha=5\cdot10^{-5}$, batch 64.
- For disjoint supports JS is stuck at $\log 2$ and KL is infinite, while $W$ is continuous and differentiable almost everywhere in $\theta$ (Theorem 1).
- Train the critic close to optimality: it then gives the right gradient and does not saturate, and its estimate falls as sample quality improves.
- Weight clipping is crude: the estimate is $K\cdot W$ for an unknown $K$, and deep clipped critics suffer vanishing or exploding gradients. **WGAN-GP** replaced it.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
