Definition
Spectral normalization reparametrizes a layer's weight as , where is the spectral norm: the largest singular value, i.e. the most any unit vector can be stretched by . The normalized layer can stretch no input difference by more than a factor 1, so it is 1-Lipschitz. is not computed exactly but tracked by one step of power iteration per training step.
Most regularizers in this module act on the loss or on the activations. Spectral normalization acts on the geometry of each layer: it fixes how sensitive the layer's output is to its input. That is exactly what an adversarial critic needs. A discriminator that can change its output arbitrarily fast in input space gives the generator gradients that are huge in some places and useless in others, and training oscillates or collapses.
Miyato, Kataoka, Koyama and Yoshida (2018), Spectral Normalization for Generative Adversarial Networks (ICLR), proposed it for GAN discriminators. It builds on Yoshida and Miyato (2017), who penalized the spectral norm instead of dividing by it. As the Adversarial loss article explains, WGAN needs a 1-Lipschitz critic. Arjovsky et al. (2017) enforced that by clipping weights and Gulrajani et al. (2017) with a gradient penalty on random interpolates. Spectral normalization bounds the Lipschitz constant everywhere, not only on sampled points, and costs one matrix–vector product pair per layer. Paired with the hinge loss it became the default discriminator recipe: Miyato and Koyama's projection discriminator (2018), SAGAN (Zhang et al., 2019) and BigGAN (Brock et al., 2019), the last two applying it to the generator as well.
Formula
- the raw weight, the parameter the optimizer updates ()
- the weight the layer actually uses in the forward pass
- largest singular value ; the operator 2-norm of
- the top left and right singular vectors, at which the maximum is attained
After normalization exactly: every direction is scaled by , and the top direction by exactly 1. The shape of (its singular vectors and the ratios ) is untouched. Only its overall gain is fixed.
Lipschitz bound for a whole network
A function is -Lipschitz if . Lipschitz constants multiply under composition. A linear layer's constant is , and ReLU, Leaky ReLU and tanh are 1-Lipschitz. So for a feed-forward network:
- the activation after layer (1-Lipschitz for ReLU, Leaky ReLU, tanh)
- the same network with every replaced by
Biases do not change the constant. A three-layer net with has bound . After spectral normalization the bound is 1, whatever the weights. The bound is usually loose: the top singular directions of successive layers rarely line up, so the true gradient norm of a normalized network is often well below 1.
At a glance
- Acts on
- each weight matrix; conv kernels are reshaped to
- Guarantees
- per layer, so the network is (about) 1-Lipschitz with 1-Lipschitz activations
- Cost
- one and one per layer per step; two stored vectors
- Hyperparameters
- none in practice (
n_power_iterations=1) - Main use
- GAN discriminators (with hinge loss), also Lipschitz/robust models, invertible ResNets, uncertainty (SNGP)
- Not
- a normalization of activations like batch norm; it does not depend on the batch
Power iteration with a persistent u
An exact SVD of every weight matrix at every step would be expensive. Power iteration finds the top singular pair by alternately applying and : the component along grows by a factor per round trip and the others by , so the error shrinks like per iteration.
- a stored vector (buffer), carried from one training step to the next
- the estimate; always, since are unit vectors
Miyato et al.'s key observation is that one iteration per training step is enough, provided is kept between steps. The optimizer changes only slightly per step, so last step's is already almost the top singular vector of the new . Each step does one more iteration on a slowly moving target, and the estimate stays converged throughout training. Because can only under-estimate, is slightly above 1 whenever the estimate lags.
Worked example: three power iterations
Take . Then has eigenvalues , so and . Start from . Values checked in Node:
| iteration | v | u | σ̂ = uᵀWv | σ(W / σ̂) |
|---|---|---|---|---|
| 1 | (0, 1) | (0.7071, 0.7071) | 1.4142 | 1.6180 |
| 2 | (0.7071, 0.7071) | (0.9487, 0.3162) | 2.2361 | 1.0233 |
| 3 | (0.8321, 0.5547) | (0.9701, 0.2425) | 2.2871 | 1.0005 |
| 4 | (0.8480, 0.5300) | (0.9728, 0.2316) | 2.2882 | 1.0000 |
The error ratio means each iteration removes about 85% of the remaining error. From a poor start the first estimate is 38% low. During training, though, is never a poor start: after the first few steps every step begins where row 4 ends.
The gradient through σ
The normalization is part of the computation graph, so is trained through . With and :
- the gradient the layer would get without normalization
- the rank-one direction that changes
The second term removes the part of the update that would simply scale up along its top singular direction. Miyato et al. read this as an adaptive penalty: the layer cannot gain sensitivity by growing one direction, so it is pushed to use several. In PyTorch, and are computed under no_grad and treated as constants, and the rest of this gradient comes from ordinary autograd through .
1import torch, torch.nn as nn, torch.nn.functional as F2from torch.nn.utils.parametrizations import spectral_norm34# An SN discriminator for 32x32 RGB images (as in SN-GAN): wrap every conv / linear layer.5D = nn.Sequential(6 spectral_norm(nn.Conv2d(3, 64, 4, stride=2, padding=1)), nn.LeakyReLU(0.1),7 spectral_norm(nn.Conv2d(64, 128, 4, stride=2, padding=1)), nn.LeakyReLU(0.1),8 nn.Flatten(),9 spectral_norm(nn.Linear(128 * 8 * 8, 1)),10)1112lin = spectral_norm(nn.Linear(20, 40)) # 15 power iterations at registration, then13lin.train() # 1 per forward pass in training mode (n_power_iterations=1)14W_raw = lin.parametrizations.weight.original # what the optimizer updates15u = lin.parametrizations.weight[0]._u # the persistent vector (a buffer, saved in state_dict)16print(torch.linalg.matrix_norm(lin.weight, ord=2)) # ~1: lin.weight is W_raw / sigma_hat17# In eval() mode u and v are frozen, so the weight is deterministic at inference.1819# The same thing by hand20@torch.no_grad()21def power_step(W, u):22 v = F.normalize(W.T @ u, dim=0)23 u = F.normalize(W @ v, dim=0)24 return u, v2526W = torch.randn(40, 20, requires_grad=True)27u = F.normalize(torch.randn(40), dim=0)28u, v = power_step(W, u) # keep u for the next training step29sigma = u @ W @ v # differentiable in W; u, v are constants30W_sn = W / sigmaUse when
- +Global Lipschitz bound per layer, for every input, not only sampled points (unlike the gradient penalty)
- +Cheap: two matrix–vector products per layer per step, no double backprop
- +No hyperparameter to tune; stable GAN training with hinge or non-saturating loss
- +Keeps singular vectors and ratios; only the gain is fixed
Avoid when
- −The estimate lags when weights move fast (high learning rate), so can exceed 1
- −Conv layers: the reshaped-kernel norm under-estimates the operator norm, so the network bound is not exact
- −The product bound is loose; a heavily constrained network can be too smooth and lose capacity
- −Extra buffers and state to save; behaviour differs between train() (updates u) and eval() (frozen)
Check yourself
Spectral normalization uses only one power iteration per training step. Why is the estimate still accurate?
Check yourself
A 4-layer MLP with ReLU has spectral norms σ(W) = 2, 3, 0.5, 4. What is the best guaranteed Lipschitz bound before and after spectral normalization?
Remember
- ✓ with the largest singular value: each layer becomes 1-Lipschitz and the network's bound becomes 1.
- ✓ is tracked by power iteration, , , , one iteration per step with kept between steps.
- ✓Introduced by Miyato et al. (2018) for GAN discriminators: a cheap, everywhere-valid alternative to WGAN's weight clipping and gradient penalty.
- ✓Gradient: , which removes the update along the top singular direction.
- ✓In PyTorch: torch.nn.utils.parametrizations.spectral_norm; conv kernels are reshaped, which under-estimates the true operator norm.