Spectral normalization

Divide each weight matrix by its largest singular value, estimated with one power-iteration step per training step, so every layer is 1-Lipschitz. It made GAN discriminators stable at almost no cost and bounds the whole network's Lipschitz constant.

13 min read1020 wordsadvanced0 viewsupdated 2026-10-11

Definition

Spectral normalization reparametrizes a layer's weight as Wˉ=W/σ(W)\bar W=W/\sigma(W), where σ(W)\sigma(W) is the spectral norm: the largest singular value, i.e. the most any unit vector can be stretched by WW. The normalized layer can stretch no input difference by more than a factor 1, so it is 1-Lipschitz. σ(W)\sigma(W) is not computed exactly but tracked by one step of power iteration per training step.

Most regularizers in this module act on the loss or on the activations. Spectral normalization acts on the geometry of each layer: it fixes how sensitive the layer's output is to its input. That is exactly what an adversarial critic needs. A discriminator that can change its output arbitrarily fast in input space gives the generator gradients that are huge in some places and useless in others, and training oscillates or collapses.

Miyato, Kataoka, Koyama and Yoshida (2018), Spectral Normalization for Generative Adversarial Networks (ICLR), proposed it for GAN discriminators. It builds on Yoshida and Miyato (2017), who penalized the spectral norm instead of dividing by it. As the Adversarial loss article explains, WGAN needs a 1-Lipschitz critic. Arjovsky et al. (2017) enforced that by clipping weights and Gulrajani et al. (2017) with a gradient penalty on random interpolates. Spectral normalization bounds the Lipschitz constant everywhere, not only on sampled points, and costs one matrix–vector product pair per layer. Paired with the hinge loss it became the default discriminator recipe: Miyato and Koyama's projection discriminator (2018), SAGAN (Zhang et al., 2019) and BigGAN (Brock et al., 2019), the last two applying it to the generator as well.

Formula

Wˉ=Wσ(W),σ(W)=max⁡∥h∥2=1∥Wh∥2=max⁡∥u∥=∥v∥=1u⊤Wv\bar W=\frac{W}{\sigma(W)},\qquad \sigma(W)=\max_{\|h\|_2=1}\|Wh\|_2=\max_{\|u\|=\|v\|=1}u^\top W v
WW
the raw weight, the parameter the optimizer updates (dout×dind_{out}\times d_{in})
Wˉ\bar W
the weight the layer actually uses in the forward pass
σ(W)\sigma(W)
largest singular value σ1\sigma_1; the operator 2-norm of WW
u, vu,\,v
the top left and right singular vectors, at which the maximum is attained

After normalization σ(Wˉ)=1\sigma(\bar W)=1 exactly: every direction is scaled by σi/σ1≤1\sigma_i/\sigma_1\le1, and the top direction by exactly 1. The shape of WW (its singular vectors and the ratios σi/σ1\sigma_i/\sigma_1) is untouched. Only its overall gain is fixed.

Lipschitz bound for a whole network

A function is LL-Lipschitz if ∥f(x)−f(x′)∥≤L∥x−x′∥\|f(x)-f(x')\|\le L\|x-x'\|. Lipschitz constants multiply under composition. A linear layer's constant is σ(W)\sigma(W), and ReLU, Leaky ReLU and tanh are 1-Lipschitz. So for a feed-forward network:

∥f∥Lip  ≤  ∏l=1Lσ(Wl)⋅∏l∥φl∥Lip  =  ∏l=1Lσ(Wl)⟹∥fSN∥Lip≤1\|f\|_{\mathrm{Lip}}\;\le\;\prod_{l=1}^{L}\sigma(W_l)\cdot\prod_{l}\|\varphi_l\|_{\mathrm{Lip}}\;=\;\prod_{l=1}^{L}\sigma(W_l)\qquad\Longrightarrow\qquad \|f_{\mathrm{SN}}\|_{\mathrm{Lip}}\le1
φl\varphi_l
the activation after layer ll (1-Lipschitz for ReLU, Leaky ReLU, tanh)
fSNf_{\mathrm{SN}}
the same network with every WlW_l replaced by Wˉl\bar W_l

Biases do not change the constant. A three-layer net with σ(Wl)=3.1, 2.4, 1.7\sigma(W_l)=3.1,\,2.4,\,1.7 has bound 3.1⋅2.4⋅1.7=12.653.1\cdot2.4\cdot1.7=12.65. After spectral normalization the bound is 1, whatever the weights. The bound is usually loose: the top singular directions of successive layers rarely line up, so the true gradient norm of a normalized network is often well below 1.

At a glance

Acts on
each weight matrix; conv kernels are reshaped to cout×(cinkhkw)c_{out}\times(c_{in}k_hk_w)
Guarantees
σ(Wˉ)≈1\sigma(\bar W)\approx1 per layer, so the network is (about) 1-Lipschitz with 1-Lipschitz activations
Cost
one W⊤uW^\top u and one WvWv per layer per step; two stored vectors u,vu,v
Hyperparameters
none in practice (n_power_iterations=1)
Main use
GAN discriminators (with hinge loss), also Lipschitz/robust models, invertible ResNets, uncertainty (SNGP)
Not
a normalization of activations like batch norm; it does not depend on the batch

Power iteration with a persistent u

An exact SVD of every weight matrix at every step would be expensive. Power iteration finds the top singular pair by alternately applying W⊤W^\top and WW: the component along u1u_1 grows by a factor σ12\sigma_1^2 per round trip and the others by σi2\sigma_i^2, so the error shrinks like (σ2/σ1)2(\sigma_2/\sigma_1)^2 per iteration.

v←W⊤u∥W⊤u∥2,u←Wv∥Wv∥2,σ^=u⊤Wvv\leftarrow\frac{W^\top u}{\|W^\top u\|_2},\qquad u\leftarrow\frac{Wv}{\|Wv\|_2},\qquad \hat\sigma=u^\top W v
uu
a stored vector (buffer), carried from one training step to the next
σ^\hat\sigma
the estimate; σ^≤σ\hat\sigma\le\sigma always, since u,vu,v are unit vectors

Miyato et al.'s key observation is that one iteration per training step is enough, provided uu is kept between steps. The optimizer changes WW only slightly per step, so last step's uu is already almost the top singular vector of the new WW. Each step does one more iteration on a slowly moving target, and the estimate stays converged throughout training. Because σ^\hat\sigma can only under-estimate, σ(W/σ^)=σ/σ^\sigma(W/\hat\sigma)=\sigma/\hat\sigma is slightly above 1 whenever the estimate lags.

step 0: unit circle → W x and W x / σ̂W: σ₁ 2.000, σ₂ 0.800W/σ̂: σ₁ 1.0136u vs true u₁: 4.2°true σ(W_t) and estimate σ̂_t = uᵀW v0123σ 2.0000 · σ̂ 1.9732 · error -1.34%σ(W_t / σ̂_t): the Lipschitz constant actually used1.01.21.41.6steps 5–60: mean |error| 0.01% · worst σ 1.000━ σ̂ ┅ true σ ━ u ━ W x
iterations/step
fig.A 2×2 weight matrix drifts over 60 steps (rotating by 0.03 rad per step, σ₁ between 1.4 and 2.6, σ₂/σ₁ = 0.4), and one power iteration runs per step. Left: the unit circle (dashed), its image under W (orange) and under W/σ̂ (violet). At step 0, from a random u, σ̂ = 1.973 against σ = 2.000 (1.3% low). With persistent u the mean error over steps 5 to 60 is 0.01% and σ(W/σ̂) never exceeds 1.000. Switch to a fresh random u every step: one iteration is no longer enough, the mean error is 17.2% and σ(W/σ̂) reaches 2.5. Raise the drift to 0.3 rad per step and the persistent estimate lags (0.94% mean error); 3 iterations per step remove the lag.

Worked example: three power iterations

Take W=(2101)W=\begin{pmatrix}2&1\\0&1\end{pmatrix}. Then W⊤W=(4222)W^\top W=\begin{pmatrix}4&2\\2&2\end{pmatrix} has eigenvalues 3±53\pm\sqrt5, so σ1=3+5=2.2882\sigma_1=\sqrt{3+\sqrt5}=2.2882 and σ2=0.8740\sigma_2=0.8740. Start from u=(0,1)u=(0,1). Values checked in Node:

iterationvuσ̂ = uᵀWvσ(W / σ̂)
1(0, 1)(0.7071, 0.7071)1.41421.6180
2(0.7071, 0.7071)(0.9487, 0.3162)2.23611.0233
3(0.8321, 0.5547)(0.9701, 0.2425)2.28711.0005
4(0.8480, 0.5300)(0.9728, 0.2316)2.28821.0000
Power iteration on W = [[2, 1], [0, 1]] from u = (0, 1); true σ = 2.2882

The error ratio (σ2/σ1)2=0.146(\sigma_2/\sigma_1)^2=0.146 means each iteration removes about 85% of the remaining error. From a poor start the first estimate is 38% low. During training, though, uu is never a poor start: after the first few steps every step begins where row 4 ends.

The gradient through σ

The normalization is part of the computation graph, so WW is trained through Wˉ=W/σ(W)\bar W=W/\sigma(W). With ∂σ/∂W=uv⊤\partial\sigma/\partial W=uv^\top and G=∂L/∂WˉG=\partial L/\partial\bar W:

∂L∂W=1σ(G−⟨G,Wˉ⟩ uv⊤),⟨G,Wˉ⟩=tr⁡(G⊤Wˉ)\frac{\partial L}{\partial W}=\frac{1}{\sigma}\Big(G-\langle G,\bar W\rangle\,u v^\top\Big),\qquad \langle G,\bar W\rangle=\operatorname{tr}(G^\top\bar W)
GG
the gradient the layer would get without normalization
uv⊤uv^\top
the rank-one direction that changes σ\sigma

The second term removes the part of the update that would simply scale WW up along its top singular direction. Miyato et al. read this as an adaptive penalty: the layer cannot gain sensitivity by growing one direction, so it is pushed to use several. In PyTorch, uu and vv are computed under no_grad and treated as constants, and the rest of this gradient comes from ordinary autograd through σ^=u⊤Wv\hat\sigma=u^\top Wv.

python
1import torch, torch.nn as nn, torch.nn.functional as F
2from torch.nn.utils.parametrizations import spectral_norm
3
4# An SN discriminator for 32x32 RGB images (as in SN-GAN): wrap every conv / linear layer.
5D = nn.Sequential(
6 spectral_norm(nn.Conv2d(3, 64, 4, stride=2, padding=1)), nn.LeakyReLU(0.1),
7 spectral_norm(nn.Conv2d(64, 128, 4, stride=2, padding=1)), nn.LeakyReLU(0.1),
8 nn.Flatten(),
9 spectral_norm(nn.Linear(128 * 8 * 8, 1)),
10)
11
12lin = spectral_norm(nn.Linear(20, 40)) # 15 power iterations at registration, then
13lin.train() # 1 per forward pass in training mode (n_power_iterations=1)
14W_raw = lin.parametrizations.weight.original # what the optimizer updates
15u = lin.parametrizations.weight[0]._u # the persistent vector (a buffer, saved in state_dict)
16print(torch.linalg.matrix_norm(lin.weight, ord=2)) # ~1: lin.weight is W_raw / sigma_hat
17# In eval() mode u and v are frozen, so the weight is deterministic at inference.
18
19# The same thing by hand
20@torch.no_grad()
21def power_step(W, u):
22 v = F.normalize(W.T @ u, dim=0)
23 u = F.normalize(W @ v, dim=0)
24 return u, v
25
26W = torch.randn(40, 20, requires_grad=True)
27u = F.normalize(torch.randn(40), dim=0)
28u, v = power_step(W, u) # keep u for the next training step
29sigma = u @ W @ v # differentiable in W; u, v are constants
30W_sn = W / sigma
Spectral normalization in PyTorch: the built-in parametrization, and one step by hand

Use when

  • +Global Lipschitz bound per layer, for every input, not only sampled points (unlike the gradient penalty)
  • +Cheap: two matrix–vector products per layer per step, no double backprop
  • +No hyperparameter to tune; stable GAN training with hinge or non-saturating loss
  • +Keeps singular vectors and ratios; only the gain is fixed

Avoid when

  • −The estimate σ^≤σ\hat\sigma\le\sigma lags when weights move fast (high learning rate), so σ(Wˉ)\sigma(\bar W) can exceed 1
  • −Conv layers: the reshaped-kernel norm under-estimates the operator norm, so the network bound is not exact
  • −The product bound ∏σl\prod\sigma_l is loose; a heavily constrained network can be too smooth and lose capacity
  • −Extra buffers and state to save; behaviour differs between train() (updates u) and eval() (frozen)

Check yourself

Spectral normalization uses only one power iteration per training step. Why is the estimate still accurate?

Check yourself

A 4-layer MLP with ReLU has spectral norms σ(W) = 2, 3, 0.5, 4. What is the best guaranteed Lipschitz bound before and after spectral normalization?

Remember

  • ✓Wˉ=W/σ(W)\bar W=W/\sigma(W) with σ\sigma the largest singular value: each layer becomes 1-Lipschitz and the network's bound ∏lσ(Wl)\prod_l\sigma(W_l) becomes 1.
  • ✓σ\sigma is tracked by power iteration, v∝W⊤uv\propto W^\top u, u∝Wvu\propto Wv, σ^=u⊤Wv\hat\sigma=u^\top Wv, one iteration per step with uu kept between steps.
  • ✓Introduced by Miyato et al. (2018) for GAN discriminators: a cheap, everywhere-valid alternative to WGAN's weight clipping and gradient penalty.
  • ✓Gradient: 1σ(G−⟨G,Wˉ⟩uv⊤)\frac1\sigma(G-\langle G,\bar W\rangle uv^\top), which removes the update along the top singular direction.
  • ✓In PyTorch: torch.nn.utils.parametrizations.spectral_norm; conv kernels are reshaped, which under-estimates the true operator norm.
← back to Deep Learning roadmap