L2 regularization

Add (λ/2)‖w‖² to the loss. Near a minimum it scales each Hessian eigen-coordinate of w* by hᵢ/(hᵢ+λ): stiff directions survive, flat ones shrink away. In front of BatchNorm it only changes the effective learning rate.

11 min read884 wordscore0 viewsupdated 2026-10-11

Definition

L2 regularization adds half the squared Euclidean norm of the weights to the training loss, J(w)=L(w)+λ2∥w∥22J(w)=L(w)+\tfrac{\lambda}{2}\|w\|_2^2. Its gradient λw\lambda w pulls every weight toward zero in proportion to its size, so weights shrink smoothly but are never set exactly to zero.

L2 is the default explicit regularizer of deep learning, usually under the name weight decay. The ML branch's L1 & L2 regularization article derives ridge regression in closed form; here the question is what the penalty does to a network trained by gradient descent: which directions in weight space it actually shrinks, how it relates to the weight_decay argument of an optimizer, and why it behaves strangely in front of normalization layers.

For linear models the penalty is Tikhonov regularization (1943) and ridge regression (Hoerl and Kennard, 1970). In neural networks it appeared as "weight decay" in the late 1980s (Hanson and Pratt, 1988), and Krogh and Hertz (1991, "A Simple Weight Decay Can Improve Generalization") showed that it suppresses irrelevant components of the weight vector and part of the effect of target noise. The eigendirection analysis below follows Goodfellow, Bengio and Courville's Deep Learning (2016, §7.1.1).

Objective and gradient

J(w)=L(w)+λ2∥w∥22,∇J=∇L(w)+λw,w←w−η(∇L+λw)=(1−ηλ) w−η∇LJ(w)=L(w)+\frac{\lambda}{2}\|w\|_2^2,\qquad \nabla J=\nabla L(w)+\lambda w,\qquad w\leftarrow w-\eta(\nabla L+\lambda w)=(1-\eta\lambda)\,w-\eta\nabla L
λ\lambda
L2 strength (weight_decay in PyTorch optimizers); the 12\tfrac12 makes the gradient exactly λw\lambda w
λw\lambda w
the penalty's pull: proportional to the weight, so it fades as the weight approaches 0
(1−ηλ)(1-\eta\lambda)
for plain SGD the penalty is a multiplicative shrink of every weight per step: weight decay

Bayesian view. With a Gaussian prior w∼N(0,σp2I)w\sim\mathcal N(0,\sigma_p^2 I), −log⁡p(w)=∥w∥2/(2σp2)+const-\log p(w)=\|w\|^2/(2\sigma_p^2)+\text{const}. Minimizing the summed negative log-likelihood plus λ2∥w∥2\tfrac{\lambda}{2}\|w\|^2 is therefore the MAP estimate with λ=1/σp2\lambda=1/\sigma_p^2: a strong penalty is a narrow prior. If your loss is a mean over nn examples, the equivalent is λ=1/(nσp2)\lambda=1/(n\sigma_p^2), which is one reason good λ\lambda values look tiny in practice.

At a glance

Penalty
λ2∥w∥22\tfrac{\lambda}{2}\|w\|_2^2; gradient λw\lambda w
Prior
Gaussian N(0,1/λ)\mathcal N(0,1/\lambda) per weight (MAP), for a summed loss
Effect near a minimum
eigen-coordinate ii of w∗w^* scaled by hi/(hi+λ)h_i/(h_i+\lambda)
Sparsity
none: weights shrink geometrically but do not reach 0
PyTorch
torch.optim.SGD(..., weight_decay=λ) adds λw\lambda w to the gradient, exactly L2. Adam's weight_decay is also L2 (not decoupled); AdamW is decoupled
Typical λ
10−410^{-4} (ResNet on ImageNet, SGD) to 5×10−45\times10^{-4} (AlexNet, many CIFAR recipes)

Which directions get shrunk

Expand the loss to second order around its minimum w∗w^*: L(w)≈L(w∗)+12(w−w∗)⊤H(w−w∗)L(w)\approx L(w^*)+\tfrac12(w-w^*)^\top H(w-w^*). Setting ∇J=H(w−w∗)+λw=0\nabla J=H(w-w^*)+\lambda w=0 gives the regularized minimum. Write H=QΛQ⊤H=Q\Lambda Q^\top with eigenvectors qiq_i and curvatures hi≥0h_i\ge0:

w~=(H+λI)−1Hw∗=∑ihihi+λ (qi⊤w∗) qi\tilde w=(H+\lambda I)^{-1}Hw^*=\sum_i\frac{h_i}{h_i+\lambda}\,(q_i^\top w^*)\,q_i
hih_i
curvature of the loss along qiq_i: how much the training loss rises if you move along that direction
hi/(hi+λ)h_i/(h_i+\lambda)
shrink factor. ≈1\approx1 if hi≫λh_i\gg\lambda (the data care), ≈hi/λ→0\approx h_i/\lambda\to0 if hi≪λh_i\ll\lambda (the data barely notice)
∑ihi/(hi+λ)\sum_i h_i/(h_i+\lambda)
the effective number of parameters, between 0 and the dimension

L2 does not shrink all weights equally. It removes the components of w∗w^* that the training loss is nearly indifferent to: precisely the directions that noise in a finite training set can push around freely. In an over-parameterized network most Hessian eigenvalues are close to zero, so a small λ\lambda already fixes those directions near 0 while leaving the few stiff directions that carry the fit almost untouched.

q₁ h=2q₂ h=0.1drag w* · w̃ = L2 minimumcoordinates along the Hessian eigenvectors▭ w* · ■ w̃ = w* × h/(h + α)q₁ (steep)×0.8702.09 → 1.81q₂ (flat)×0.2500.41 → 0.10‖w*‖ 2.126 → ‖w̃‖ 1.817 · L(w̃) = 0.0788
GD on L + (α/2)‖w‖², η = 0.4, 150 steps → (1.519, 0.996) · w̃ = (1.519, 0.996)
fig.A quadratic loss with curvatures h₁ = 2 along q₁ and h₂ = 0.1 along q₂ (tilted 30°), unregularized minimum w* = (1.6, 1.4), whose eigen-coordinates are (2.086, 0.412). With α = 0.3 the L2 minimum w̃ (green) sits where a loss contour (violet) touches the circle ‖w‖ = ‖w̃‖: the steep coordinate keeps 0.870 of its value (→ 1.814) while the flat one keeps 0.250 (→ 0.103). ‖w‖ falls from 2.126 to 1.817 and the training loss rises by only 0.079. Gradient descent on the penalized loss (η = 0.4, grey path from (−1.6, −0.6)) lands on the closed-form w̃. Drag w* anywhere; slide α: at α = 1 the factors are 0.667 and 0.091, at α = 2 they are 0.500 and 0.048.

Worked example

αh₁/(h₁+α)h₂/(h₂+α)effective paramsw̃‖w̃‖
01.0001.0002.000(1.600, 1.400)2.126
0.10.9520.5001.452(1.617, 1.172)1.997
0.30.8700.2501.120(1.519, 0.996)1.817
10.6670.0910.758(1.185, 0.728)1.391
20.5000.0480.548(0.893, 0.538)1.043
The figure's problem, h=(2, 0.1)h=(2,\,0.1), w∗=(1.6, 1.4)w^*=(1.6,\,1.4): shrink factors, effective parameters and the regularized minimum (computed in Node from the figure's lib)

Look at the row α=0.1\alpha=0.1: the first raw weight grows, from 1.600 to 1.617. L2 shrinks the norm and every eigen-coordinate, but not necessarily each individual weight. Removing most of the flat-direction component rotates ww toward q1q_1, and q1q_1 happens to have a larger first entry. "L2 makes every weight smaller" is only true when the Hessian is diagonal.

In the loss or in the optimizer?

There are two places to put L2. You can add 0.5 * lam * sum(p.pow(2).sum() ...) to the loss, or pass weight_decay=lam to the optimizer. For torch.optim.SGD the two are identical, with or without momentum: PyTorch adds λw\lambda w to the gradient before the momentum buffer, exactly what autograd would do. The optimizer version is cheaper (no extra graph) and can be set per parameter group, so you can exempt biases and normalization parameters.

For adaptive optimizers the two are still identical to each other (Adam's weight_decay is also L2 added to the gradient), but both differ from true weight decay. Adam divides the penalty gradient λw\lambda w by v^\sqrt{\hat v}, so weights with large gradients are barely regularized. Decoupling the decay from the gradient is the point of AdamW; the Weight decay article covers the difference.

L2 in front of BatchNorm: a learning-rate knob

If a weight vector ww feeds a BatchNorm or LayerNorm, the network's output is unchanged when ww is rescaled: L(cw)=L(w)L(cw)=L(w) for c>0c>0. Two consequences follow by differentiating that identity: w⊤∇L(w)=0w^\top\nabla L(w)=0 (the gradient is orthogonal to ww) and ∇L(cw)=∇L(w)/c\nabla L(cw)=\nabla L(w)/c (it shrinks as ww grows). The penalty λ2∥w∥2\tfrac{\lambda}{2}\|w\|^2 therefore cannot limit what the layer computes, because every scale gives the same function.

What it does change is the optimization. The direction w/∥w∥w/\|w\| moves by about η∥∇L∥/∥w∥∝η/∥w∥2\eta\|\nabla L\|/\|w\|\propto\eta/\|w\|^2 per step, so the effective learning rate is η/∥w∥2\eta/\|w\|^2. Without the penalty, orthogonal gradient steps only lengthen ww and training slows down by itself; with it, ∥w∥\|w\| settles at an equilibrium and the effective learning rate stays up (van Laarhoven, 2017; Zhang, Wang, Xu and Grosse, 2019). The Weight decay figure simulates exactly this.

python
1import torch, torch.nn as nn
2
3# Same one-step result both ways: w = 2, grad 0.5, lr 0.1, lam 0.1
4for how in ("loss", "optimizer"):
5 w = nn.Parameter(torch.tensor([2.0]))
6 opt = torch.optim.SGD([w], lr=0.1, momentum=0.9,
7 weight_decay=0.1 if how == "optimizer" else 0.0)
8 loss = 0.5 * w.sum() # dL/dw = 0.5
9 if how == "loss":
10 loss = loss + 0.5 * 0.1 * w.pow(2).sum() # (lam/2)||w||^2
11 loss.backward()
12 opt.step()
13 print(how, round(w.item(), 4)) # both 1.93 = 2 - 0.1 * (0.5 + 0.1 * 2)
14
15# In practice: decay matrices and conv kernels, not biases or norm parameters
16model = nn.Sequential(nn.Conv2d(3, 64, 3, padding=1), nn.BatchNorm2d(64), nn.ReLU(),
17 nn.Flatten(), nn.Linear(64 * 32 * 32, 10))
18decay = [p for p in model.parameters() if p.ndim >= 2]
19no_decay = [p for p in model.parameters() if p.ndim < 2]
20opt = torch.optim.SGD([{"params": decay, "weight_decay": 5e-4},
21 {"params": no_decay, "weight_decay": 0.0}],
22 lr=0.1, momentum=0.9, nesterov=True)
L2 in the loss versus weight_decay in SGD (identical), and the usual parameter groups

Use when

  • +Smooth and differentiable: works with any gradient method, no special update needed
  • +Removes exactly the low-curvature directions that overfitting exploits
  • +Clear Bayesian meaning (Gaussian prior) and a closed form for quadratic losses
  • +For SGD it is a one-line optimizer argument with per-group control

Avoid when

  • −No sparsity: every weight stays non-zero
  • −Inside Adam it is rescaled by 1/v^1/\sqrt{\hat v} and loses its meaning; use AdamW
  • −For weights before normalization layers it does not regularize the function, only the effective learning rate
  • −λ interacts with the learning rate, the batch size and the schedule, so values do not transfer between recipes

Check yourself

Near a minimum the Hessian has eigenvalues 4 and 0.04. With L2 strength λ = 0.04, by what factors are the two eigen-coordinates of w* scaled?

Check yourself

A conv kernel w feeds a BatchNorm layer. You double λ on w. What changes?

Remember

  • ✓L2 adds λ2∥w∥2\tfrac{\lambda}{2}\|w\|^2; its gradient λw\lambda w shrinks weights in proportion to their size and never zeroes them.
  • ✓It is MAP estimation with a Gaussian prior, λ=1/σp2\lambda=1/\sigma_p^2 for a summed loss.
  • ✓Near a minimum it scales each Hessian eigen-coordinate by hi/(hi+λ)h_i/(h_i+\lambda): flat directions vanish, stiff ones stay.
  • ✓SGD(weight_decay=λ) is exactly L2 in the loss (even with momentum); for Adam neither is true weight decay, so use AdamW.
  • ✓Before BatchNorm/LayerNorm the loss is scale-invariant, and L2 acts only through the effective learning rate η/∥w∥2\eta/\|w\|^2.
← back to Deep Learning roadmap