# L2 regularization

> Add (λ/2)‖w‖² to the loss. Near a minimum it scales each Hessian eigen-coordinate of w* by hᵢ/(hᵢ+λ): stiff directions survive, flat ones shrink away. In front of BatchNorm it only changes the effective learning rate.

Source: https://www.mlfoundry.in/learn/l2-regularization  
Section: Deep Learning › Regularization & Normalization  
Level: core · 11 min read · updated 2026-10-11

**L2 regularization:** adds half the squared Euclidean norm of the weights to the training loss, $J(w)=L(w)+\tfrac{\lambda}{2}\|w\|_2^2$. Its gradient $\lambda w$ pulls every weight toward zero in proportion to its size, so weights shrink smoothly but are never set exactly to zero.

L2 is the default explicit regularizer of deep learning, usually under the name **weight decay**. The ML branch's **L1 & L2 regularization** article derives ridge regression in closed form; here the question is what the penalty does to a network trained by gradient descent: which directions in weight space it actually shrinks, how it relates to the `weight_decay` argument of an optimizer, and why it behaves strangely in front of normalization layers.

For linear models the penalty is Tikhonov regularization (1943) and ridge regression (Hoerl and Kennard, 1970). In neural networks it appeared as "weight decay" in the late 1980s (Hanson and Pratt, 1988), and Krogh and Hertz (1991, "A Simple Weight Decay Can Improve Generalization") showed that it suppresses irrelevant components of the weight vector and part of the effect of target noise. The eigendirection analysis below follows Goodfellow, Bengio and Courville's *Deep Learning* (2016, §7.1.1).

## Objective and gradient

$$
J(w)=L(w)+\frac{\lambda}{2}\|w\|_2^2,\qquad \nabla J=\nabla L(w)+\lambda w,\qquad w\leftarrow w-\eta(\nabla L+\lambda w)=(1-\eta\lambda)\,w-\eta\nabla L
$$

- $\lambda$: L2 strength (`weight_decay` in PyTorch optimizers); the $\tfrac12$ makes the gradient exactly $\lambda w$
- $\lambda w$: the penalty's pull: proportional to the weight, so it fades as the weight approaches 0
- $(1-\eta\lambda)$: for plain SGD the penalty is a multiplicative shrink of every weight per step: weight decay

**Bayesian view.** With a Gaussian prior $w\sim\mathcal N(0,\sigma_p^2 I)$, $-\log p(w)=\|w\|^2/(2\sigma_p^2)+\text{const}$. Minimizing the summed negative log-likelihood plus $\tfrac{\lambda}{2}\|w\|^2$ is therefore the **MAP** estimate with $\lambda=1/\sigma_p^2$: a strong penalty is a narrow prior. If your loss is a **mean** over $n$ examples, the equivalent is $\lambda=1/(n\sigma_p^2)$, which is one reason good $\lambda$ values look tiny in practice.

**At a glance**

- **Penalty:** $\tfrac{\lambda}{2}\|w\|_2^2$; gradient $\lambda w$
- **Prior:** Gaussian $\mathcal N(0,1/\lambda)$ per weight (MAP), for a summed loss
- **Effect near a minimum:** eigen-coordinate $i$ of $w^*$ scaled by $h_i/(h_i+\lambda)$
- **Sparsity:** none: weights shrink geometrically but do not reach 0
- **PyTorch:** `torch.optim.SGD(..., weight_decay=λ)` adds $\lambda w$ to the gradient, exactly L2. Adam's `weight_decay` is also L2 (not decoupled); `AdamW` is decoupled
- **Typical λ:** $10^{-4}$ (ResNet on ImageNet, SGD) to $5\times10^{-4}$ (AlexNet, many CIFAR recipes)

## Which directions get shrunk

Expand the loss to second order around its minimum $w^*$: $L(w)\approx L(w^*)+\tfrac12(w-w^*)^\top H(w-w^*)$. Setting $\nabla J=H(w-w^*)+\lambda w=0$ gives the regularized minimum. Write $H=Q\Lambda Q^\top$ with eigenvectors $q_i$ and curvatures $h_i\ge0$:

$$
\tilde w=(H+\lambda I)^{-1}Hw^*=\sum_i\frac{h_i}{h_i+\lambda}\,(q_i^\top w^*)\,q_i
$$

- $h_i$: curvature of the loss along $q_i$: how much the training loss rises if you move along that direction
- $h_i/(h_i+\lambda)$: shrink factor. $\approx1$ if $h_i\gg\lambda$ (the data care), $\approx h_i/\lambda\to0$ if $h_i\ll\lambda$ (the data barely notice)
- $\sum_i h_i/(h_i+\lambda)$: the effective number of parameters, between 0 and the dimension

L2 does not shrink all weights equally. It removes the components of $w^*$ that the training loss is nearly indifferent to: precisely the directions that noise in a finite training set can push around freely. In an over-parameterized network most Hessian eigenvalues are close to zero, so a small $\lambda$ already fixes those directions near 0 while leaving the few stiff directions that carry the fit almost untouched.

> Figure: A quadratic loss with curvatures h₁ = 2 along q₁ and h₂ = 0.1 along q₂ (tilted 30°), unregularized minimum w* = (1.6, 1.4), whose eigen-coordinates are (2.086, 0.412). With α = 0.3 the L2 minimum w̃ (green) sits where a loss contour (violet) touches the circle ‖w‖ = ‖w̃‖: the steep coordinate keeps 0.870 of its value (→ 1.814) while the flat one keeps 0.250 (→ 0.103). ‖w‖ falls from 2.126 to 1.817 and the training loss rises by only 0.079. Gradient descent on the penalized loss (η = 0.4, grey path from (−1.6, −0.6)) lands on the closed-form w̃. Drag w* anywhere; slide α: at α = 1 the factors are 0.667 and 0.091, at α = 2 they are 0.500 and 0.048.

## Worked example

The figure's problem, $h=(2,\,0.1)$, $w^*=(1.6,\,1.4)$: shrink factors, effective parameters and the regularized minimum (computed in Node from the figure's lib)

| α | h₁/(h₁+α) | h₂/(h₂+α) | effective params | w̃ | ‖w̃‖ |
| --- | --- | --- | --- | --- | --- |
| 0 | 1.000 | 1.000 | 2.000 | (1.600, 1.400) | 2.126 |
| 0.1 | 0.952 | 0.500 | 1.452 | (1.617, 1.172) | 1.997 |
| 0.3 | 0.870 | 0.250 | 1.120 | (1.519, 0.996) | 1.817 |
| 1 | 0.667 | 0.091 | 0.758 | (1.185, 0.728) | 1.391 |
| 2 | 0.500 | 0.048 | 0.548 | (0.893, 0.538) | 1.043 |

Look at the row $\alpha=0.1$: the **first raw weight grows**, from 1.600 to 1.617. L2 shrinks the norm and every eigen-coordinate, but not necessarily each individual weight. Removing most of the flat-direction component rotates $w$ toward $q_1$, and $q_1$ happens to have a larger first entry. "L2 makes every weight smaller" is only true when the Hessian is diagonal.

## In the loss or in the optimizer?

There are two places to put L2. You can add `0.5 * lam * sum(p.pow(2).sum() ...)` to the loss, or pass `weight_decay=lam` to the optimizer. For `torch.optim.SGD` the two are identical, with or without momentum: PyTorch adds $\lambda w$ to the gradient before the momentum buffer, exactly what autograd would do. The optimizer version is cheaper (no extra graph) and can be set per **parameter group**, so you can exempt biases and normalization parameters.

For adaptive optimizers the two are still identical to each other (Adam's `weight_decay` is also L2 added to the gradient), but both differ from true **weight decay**. Adam divides the penalty gradient $\lambda w$ by $\sqrt{\hat v}$, so weights with large gradients are barely regularized. Decoupling the decay from the gradient is the point of **AdamW**; the **Weight decay** article covers the difference.

## L2 in front of BatchNorm: a learning-rate knob

If a weight vector $w$ feeds a BatchNorm or LayerNorm, the network's output is unchanged when $w$ is rescaled: $L(cw)=L(w)$ for $c>0$. Two consequences follow by differentiating that identity: $w^\top\nabla L(w)=0$ (the gradient is orthogonal to $w$) and $\nabla L(cw)=\nabla L(w)/c$ (it shrinks as $w$ grows). The penalty $\tfrac{\lambda}{2}\|w\|^2$ therefore cannot limit what the layer computes, because every scale gives the same function.

What it does change is the optimization. The direction $w/\|w\|$ moves by about $\eta\|\nabla L\|/\|w\|\propto\eta/\|w\|^2$ per step, so the **effective learning rate** is $\eta/\|w\|^2$. Without the penalty, orthogonal gradient steps only lengthen $w$ and training slows down by itself; with it, $\|w\|$ settles at an equilibrium and the effective learning rate stays up (van Laarhoven, 2017; Zhang, Wang, Xu and Grosse, 2019). The **Weight decay** figure simulates exactly this.

```python
import torch, torch.nn as nn

# Same one-step result both ways: w = 2, grad 0.5, lr 0.1, lam 0.1
for how in ("loss", "optimizer"):
    w = nn.Parameter(torch.tensor([2.0]))
    opt = torch.optim.SGD([w], lr=0.1, momentum=0.9,
                          weight_decay=0.1 if how == "optimizer" else 0.0)
    loss = 0.5 * w.sum()                        # dL/dw = 0.5
    if how == "loss":
        loss = loss + 0.5 * 0.1 * w.pow(2).sum()  # (lam/2)||w||^2
    loss.backward()
    opt.step()
    print(how, round(w.item(), 4))              # both 1.93 = 2 - 0.1 * (0.5 + 0.1 * 2)

# In practice: decay matrices and conv kernels, not biases or norm parameters
model = nn.Sequential(nn.Conv2d(3, 64, 3, padding=1), nn.BatchNorm2d(64), nn.ReLU(),
                      nn.Flatten(), nn.Linear(64 * 32 * 32, 10))
decay = [p for p in model.parameters() if p.ndim >= 2]
no_decay = [p for p in model.parameters() if p.ndim < 2]
opt = torch.optim.SGD([{"params": decay, "weight_decay": 5e-4},
                       {"params": no_decay, "weight_decay": 0.0}],
                      lr=0.1, momentum=0.9, nesterov=True)
```

L2 in the loss versus weight_decay in SGD (identical), and the usual parameter groups

> **Counting the penalty twice, or on the wrong things:** Adding $\tfrac{\lambda}{2}\|w\|^2$ to the loss **and** passing `weight_decay` doubles the strength. Applying it to every parameter by default also decays biases and BatchNorm/LayerNorm gains, which pulls normalized outputs toward 0 and usually hurts. Use parameter groups and decay only weight matrices and kernels.

> **Shrinking where the data are silent:** The factor $h_i/(h_i+\lambda)$ says L2 acts like a soft low-rank projection: it keeps the directions the loss is stiff in and erases those it is flat in. That is also what early stopping does approximately, since gradient descent moves slowest along low-curvature directions; Goodfellow et al. show the two are equivalent for a quadratic loss with $\lambda\approx1/(\eta t)$.

> **Where you meet it:** Nearly every CNN recipe trained with SGD: AlexNet used weight decay $5\times10^{-4}$, ResNet on ImageNet $10^{-4}$. In transformers trained with AdamW it is replaced by decoupled weight decay, usually $0.01$ to $0.1$.

**Strengths**

- Smooth and differentiable: works with any gradient method, no special update needed
- Removes exactly the low-curvature directions that overfitting exploits
- Clear Bayesian meaning (Gaussian prior) and a closed form for quadratic losses
- For SGD it is a one-line optimizer argument with per-group control

**Limitations**

- No sparsity: every weight stays non-zero
- Inside Adam it is rescaled by $1/\sqrt{\hat v}$ and loses its meaning; use AdamW
- For weights before normalization layers it does not regularize the function, only the effective learning rate
- λ interacts with the learning rate, the batch size and the schedule, so values do not transfer between recipes

**Question:** Near a minimum the Hessian has eigenvalues 4 and 0.04. With L2 strength λ = 0.04, by what factors are the two eigen-coordinates of w* scaled?

- A. 0.99 and 0.5
- B. Both by 0.96
- C. 0.5 and 0.99
- D. Both by 1/(1 + 0.04)

**Answer:** A. 0.99 and 0.5 — The factor is $h/(h+\lambda)$: $4/4.04=0.990$ for the stiff direction and $0.04/0.08=0.5$ for the flat one. L2 halves the component the loss barely constrains and leaves the stiff one almost untouched.

**Question:** A conv kernel w feeds a BatchNorm layer. You double λ on w. What changes?

- A. The function the layer can represent becomes simpler
- B. ‖w‖ settles lower, so the effective learning rate η/‖w‖² goes up
- C. Nothing at all
- D. The gradients with respect to w are no longer orthogonal to w

**Answer:** B. ‖w‖ settles lower, so the effective learning rate η/‖w‖² goes up — BatchNorm makes the loss invariant to the scale of $w$, so the penalty cannot restrict the function. It only lowers the equilibrium norm, and because the direction moves by about $\eta/\|w\|^2$ per step, that raises the effective learning rate. The orthogonality $w^\top\nabla L=0$ is a property of scale invariance and does not depend on $\lambda$.

**Key points**

- L2 adds $\tfrac{\lambda}{2}\|w\|^2$; its gradient $\lambda w$ shrinks weights in proportion to their size and never zeroes them.
- It is MAP estimation with a Gaussian prior, $\lambda=1/\sigma_p^2$ for a summed loss.
- Near a minimum it scales each Hessian eigen-coordinate by $h_i/(h_i+\lambda)$: flat directions vanish, stiff ones stay.
- `SGD(weight_decay=λ)` is exactly L2 in the loss (even with momentum); for Adam neither is true weight decay, so use AdamW.
- Before BatchNorm/LayerNorm the loss is scale-invariant, and L2 acts only through the effective learning rate $\eta/\|w\|^2$.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
