# Spectral normalization

> Divide each weight matrix by its largest singular value, estimated with one power-iteration step per training step, so every layer is 1-Lipschitz. It made GAN discriminators stable at almost no cost and bounds the whole network's Lipschitz constant.

Source: https://www.mlfoundry.in/learn/spectral-normalization  
Section: Deep Learning › Regularization & Normalization  
Level: advanced · 13 min read · updated 2026-10-11

**Spectral normalization:** reparametrizes a layer's weight as $\bar W=W/\sigma(W)$, where $\sigma(W)$ is the **spectral norm**: the largest singular value, i.e. the most any unit vector can be stretched by $W$. The normalized layer can stretch no input difference by more than a factor 1, so it is **1-Lipschitz**. $\sigma(W)$ is not computed exactly but tracked by one step of power iteration per training step.

Most regularizers in this module act on the loss or on the activations. Spectral normalization acts on the **geometry of each layer**: it fixes how sensitive the layer's output is to its input. That is exactly what an adversarial critic needs. A discriminator that can change its output arbitrarily fast in input space gives the generator gradients that are huge in some places and useless in others, and training oscillates or collapses.

**Miyato, Kataoka, Koyama and Yoshida (2018)**, *Spectral Normalization for Generative Adversarial Networks* (ICLR), proposed it for GAN discriminators. It builds on Yoshida and Miyato (2017), who **penalized** the spectral norm instead of dividing by it. As the **Adversarial loss** article explains, WGAN needs a 1-Lipschitz critic. Arjovsky et al. (2017) enforced that by clipping weights and Gulrajani et al. (2017) with a gradient penalty on random interpolates. Spectral normalization bounds the Lipschitz constant **everywhere**, not only on sampled points, and costs one matrix–vector product pair per layer. Paired with the hinge loss it became the default discriminator recipe: Miyato and Koyama's projection discriminator (2018), SAGAN (Zhang et al., 2019) and BigGAN (Brock et al., 2019), the last two applying it to the generator as well.

## Formula

$$
\bar W=\frac{W}{\sigma(W)},\qquad \sigma(W)=\max_{\|h\|_2=1}\|Wh\|_2=\max_{\|u\|=\|v\|=1}u^\top W v
$$

- $W$: the raw weight, the parameter the optimizer updates ($d_{out}\times d_{in}$)
- $\bar W$: the weight the layer actually uses in the forward pass
- $\sigma(W)$: largest singular value $\sigma_1$; the operator 2-norm of $W$
- $u,\,v$: the top left and right singular vectors, at which the maximum is attained

After normalization $\sigma(\bar W)=1$ exactly: every direction is scaled by $\sigma_i/\sigma_1\le1$, and the top direction by exactly 1. The **shape** of $W$ (its singular vectors and the ratios $\sigma_i/\sigma_1$) is untouched. Only its overall gain is fixed.

## Lipschitz bound for a whole network

A function is $L$-Lipschitz if $\|f(x)-f(x')\|\le L\|x-x'\|$. Lipschitz constants multiply under composition. A linear layer's constant is $\sigma(W)$, and ReLU, Leaky ReLU and tanh are 1-Lipschitz. So for a feed-forward network:

$$
\|f\|_{\mathrm{Lip}}\;\le\;\prod_{l=1}^{L}\sigma(W_l)\cdot\prod_{l}\|\varphi_l\|_{\mathrm{Lip}}\;=\;\prod_{l=1}^{L}\sigma(W_l)\qquad\Longrightarrow\qquad \|f_{\mathrm{SN}}\|_{\mathrm{Lip}}\le1
$$

- $\varphi_l$: the activation after layer $l$ (1-Lipschitz for ReLU, Leaky ReLU, tanh)
- $f_{\mathrm{SN}}$: the same network with every $W_l$ replaced by $\bar W_l$

Biases do not change the constant. A three-layer net with $\sigma(W_l)=3.1,\,2.4,\,1.7$ has bound $3.1\cdot2.4\cdot1.7=12.65$. After spectral normalization the bound is 1, whatever the weights. The bound is usually loose: the top singular directions of successive layers rarely line up, so the true gradient norm of a normalized network is often well below 1.

**At a glance**

- **Acts on:** each weight matrix; conv kernels are reshaped to $c_{out}\times(c_{in}k_hk_w)$
- **Guarantees:** $\sigma(\bar W)\approx1$ per layer, so the network is (about) 1-Lipschitz with 1-Lipschitz activations
- **Cost:** one $W^\top u$ and one $Wv$ per layer per step; two stored vectors $u,v$
- **Hyperparameters:** none in practice (`n_power_iterations=1`)
- **Main use:** GAN discriminators (with hinge loss), also Lipschitz/robust models, invertible ResNets, uncertainty (SNGP)
- **Not:** a normalization of activations like batch norm; it does not depend on the batch

## Power iteration with a persistent u

An exact SVD of every weight matrix at every step would be expensive. Power iteration finds the top singular pair by alternately applying $W^\top$ and $W$: the component along $u_1$ grows by a factor $\sigma_1^2$ per round trip and the others by $\sigma_i^2$, so the error shrinks like $(\sigma_2/\sigma_1)^2$ per iteration.

$$
v\leftarrow\frac{W^\top u}{\|W^\top u\|_2},\qquad u\leftarrow\frac{Wv}{\|Wv\|_2},\qquad \hat\sigma=u^\top W v
$$

- $u$: a stored vector (buffer), carried from one training step to the next
- $\hat\sigma$: the estimate; $\hat\sigma\le\sigma$ always, since $u,v$ are unit vectors

Miyato et al.'s key observation is that **one iteration per training step is enough**, provided $u$ is kept between steps. The optimizer changes $W$ only slightly per step, so last step's $u$ is already almost the top singular vector of the new $W$. Each step does one more iteration on a slowly moving target, and the estimate stays converged throughout training. Because $\hat\sigma$ can only under-estimate, $\sigma(W/\hat\sigma)=\sigma/\hat\sigma$ is slightly **above** 1 whenever the estimate lags.

> Figure: A 2×2 weight matrix drifts over 60 steps (rotating by 0.03 rad per step, σ₁ between 1.4 and 2.6, σ₂/σ₁ = 0.4), and one power iteration runs per step. Left: the unit circle (dashed), its image under W (orange) and under W/σ̂ (violet). At step 0, from a random u, σ̂ = 1.973 against σ = 2.000 (1.3% low). With persistent u the mean error over steps 5 to 60 is 0.01% and σ(W/σ̂) never exceeds 1.000. Switch to a fresh random u every step: one iteration is no longer enough, the mean error is 17.2% and σ(W/σ̂) reaches 2.5. Raise the drift to 0.3 rad per step and the persistent estimate lags (0.94% mean error); 3 iterations per step remove the lag.

## Worked example: three power iterations

Take $W=\begin{pmatrix}2&1\\0&1\end{pmatrix}$. Then $W^\top W=\begin{pmatrix}4&2\\2&2\end{pmatrix}$ has eigenvalues $3\pm\sqrt5$, so $\sigma_1=\sqrt{3+\sqrt5}=2.2882$ and $\sigma_2=0.8740$. Start from $u=(0,1)$. Values checked in Node:

Power iteration on W = [[2, 1], [0, 1]] from u = (0, 1); true σ = 2.2882

| iteration | v | u | σ̂ = uᵀWv | σ(W / σ̂) |
| --- | --- | --- | --- | --- |
| 1 | (0, 1) | (0.7071, 0.7071) | 1.4142 | 1.6180 |
| 2 | (0.7071, 0.7071) | (0.9487, 0.3162) | 2.2361 | 1.0233 |
| 3 | (0.8321, 0.5547) | (0.9701, 0.2425) | 2.2871 | 1.0005 |
| 4 | (0.8480, 0.5300) | (0.9728, 0.2316) | 2.2882 | 1.0000 |

The error ratio $(\sigma_2/\sigma_1)^2=0.146$ means each iteration removes about 85% of the remaining error. From a poor start the first estimate is 38% low. During training, though, $u$ is never a poor start: after the first few steps every step begins where row 4 ends.

## The gradient through σ

The normalization is part of the computation graph, so $W$ is trained through $\bar W=W/\sigma(W)$. With $\partial\sigma/\partial W=uv^\top$ and $G=\partial L/\partial\bar W$:

$$
\frac{\partial L}{\partial W}=\frac{1}{\sigma}\Big(G-\langle G,\bar W\rangle\,u v^\top\Big),\qquad \langle G,\bar W\rangle=\operatorname{tr}(G^\top\bar W)
$$

- $G$: the gradient the layer would get without normalization
- $uv^\top$: the rank-one direction that changes $\sigma$

The second term removes the part of the update that would simply scale $W$ up along its top singular direction. Miyato et al. read this as an adaptive penalty: the layer cannot gain sensitivity by growing one direction, so it is pushed to use several. In PyTorch, $u$ and $v$ are computed under `no_grad` and treated as constants, and the rest of this gradient comes from ordinary autograd through $\hat\sigma=u^\top Wv$.

```python
import torch, torch.nn as nn, torch.nn.functional as F
from torch.nn.utils.parametrizations import spectral_norm

# An SN discriminator for 32x32 RGB images (as in SN-GAN): wrap every conv / linear layer.
D = nn.Sequential(
    spectral_norm(nn.Conv2d(3, 64, 4, stride=2, padding=1)),   nn.LeakyReLU(0.1),
    spectral_norm(nn.Conv2d(64, 128, 4, stride=2, padding=1)), nn.LeakyReLU(0.1),
    nn.Flatten(),
    spectral_norm(nn.Linear(128 * 8 * 8, 1)),
)

lin = spectral_norm(nn.Linear(20, 40))       # 15 power iterations at registration, then
lin.train()                                  # 1 per forward pass in training mode (n_power_iterations=1)
W_raw = lin.parametrizations.weight.original  # what the optimizer updates
u = lin.parametrizations.weight[0]._u        # the persistent vector (a buffer, saved in state_dict)
print(torch.linalg.matrix_norm(lin.weight, ord=2))   # ~1: lin.weight is W_raw / sigma_hat
# In eval() mode u and v are frozen, so the weight is deterministic at inference.

# The same thing by hand
@torch.no_grad()
def power_step(W, u):
    v = F.normalize(W.T @ u, dim=0)
    u = F.normalize(W @ v, dim=0)
    return u, v

W = torch.randn(40, 20, requires_grad=True)
u = F.normalize(torch.randn(40), dim=0)
u, v = power_step(W, u)          # keep u for the next training step
sigma = u @ W @ v                # differentiable in W; u, v are constants
W_sn = W / sigma
```

Spectral normalization in PyTorch: the built-in parametrization, and one step by hand

> **Convolutions: the reshaped kernel is not the operator:** For a conv layer, PyTorch (like the paper) reshapes the kernel to a $c_{out}\times(c_{in}k_hk_w)$ matrix and normalizes that. The convolution as a linear map on images has a different, usually **larger** norm, because each input pixel is reused by $k_hk_w$ overlapping windows. A single-channel 3×3 kernel of ones has reshaped norm $\sqrt9=3$, but as a convolution with circular padding it scales a constant image by 9. In Node, a random Gaussian 3×3 kernel with 2 input and 2 output channels gives 4.91 for the reshaped matrix and 9.27 for the convolution on a 16×16 grid. An SN-GAN discriminator is therefore not exactly 1-Lipschitz. If the bound matters (certified robustness, invertible ResNets), compute the true norm: via the FFT (Sedghi, Gupta and Long, 2019) or by power iteration on the convolution itself, using `conv2d` and `conv_transpose2d` in place of $W$ and $W^\top$.

> **Why persistent u works:** Power iteration converges geometrically; the weights change slowly. Spreading the iterations over training steps amortizes the cost to one iteration each, and the estimate never has to start from scratch. In the figure a fresh random $u$ every step gives estimates up to 2.5 times too small. Keeping $u$ gives 0.01% error at the same cost.

> **Gain and capacity:** Fixing $\sigma(\bar W_l)=1$ removes each layer's freedom to set its own scale. For GAN discriminators that is the point. For classifiers it can cost capacity, and the network has to make up the scale elsewhere (a learned output temperature, or a Lipschitz constant $c\neq1$ via $c\,W/\sigma$). Invertible ResNets (Behrmann et al., 2019) use $c<1$ so that each residual branch is a contraction and the block $x+g(x)$ can be inverted by fixed-point iteration.

**Strengths**

- Global Lipschitz bound per layer, for every input, not only sampled points (unlike the gradient penalty)
- Cheap: two matrix–vector products per layer per step, no double backprop
- No hyperparameter to tune; stable GAN training with hinge or non-saturating loss
- Keeps singular vectors and ratios; only the gain is fixed

**Limitations**

- The estimate $\hat\sigma\le\sigma$ lags when weights move fast (high learning rate), so $\sigma(\bar W)$ can exceed 1
- Conv layers: the reshaped-kernel norm under-estimates the operator norm, so the network bound is not exact
- The product bound $\prod\sigma_l$ is loose; a heavily constrained network can be too smooth and lose capacity
- Extra buffers and state to save; behaviour differs between train() (updates u) and eval() (frozen)

**Question:** Spectral normalization uses only one power iteration per training step. Why is the estimate still accurate?

- A. The vector u is kept between steps, and W changes little per step, so each step continues an almost-converged iteration
- B. One iteration of power iteration is always exact for any matrix
- C. The optimizer adjusts W so that its top singular vector equals the initial u
- D. The gradient of σ corrects the estimate during backprop

**Answer:** A. The vector u is kept between steps, and W changes little per step, so each step continues an almost-converged iteration — A single iteration from a random vector can be far off (38% low in the worked example). But $u$ is a persistent buffer: after a few steps it is essentially the top singular vector, and since the optimizer only nudges $W$, one more iteration per step keeps it there. In the figure, a fresh $u$ each step gives 17% mean error; a persistent one gives 0.01%.

**Question:** A 4-layer MLP with ReLU has spectral norms σ(W) = 2, 3, 0.5, 4. What is the best guaranteed Lipschitz bound before and after spectral normalization?

- A. 12 before, 1 after
- B. 9.5 before, 4 after
- C. 4 before, 1 after
- D. 12 before, 0.25 after

**Answer:** A. 12 before, 1 after — Lipschitz constants multiply through composition and ReLU is 1-Lipschitz: $2\cdot3\cdot0.5\cdot4=12$. After normalization every $\sigma(\bar W_l)=1$, so the product is 1. The true constant can be smaller than either bound.

**Key points**

- $\bar W=W/\sigma(W)$ with $\sigma$ the largest singular value: each layer becomes 1-Lipschitz and the network's bound $\prod_l\sigma(W_l)$ becomes 1.
- $\sigma$ is tracked by power iteration, $v\propto W^\top u$, $u\propto Wv$, $\hat\sigma=u^\top Wv$, one iteration per step with $u$ kept between steps.
- Introduced by Miyato et al. (2018) for GAN discriminators: a cheap, everywhere-valid alternative to WGAN's weight clipping and gradient penalty.
- Gradient: $\frac1\sigma(G-\langle G,\bar W\rangle uv^\top)$, which removes the update along the top singular direction.
- In PyTorch: torch.nn.utils.parametrizations.spectral_norm; conv kernels are reshaped, which under-estimates the true operator norm.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
