Definition
L2 regularization adds half the squared Euclidean norm of the weights to the training loss, . Its gradient pulls every weight toward zero in proportion to its size, so weights shrink smoothly but are never set exactly to zero.
L2 is the default explicit regularizer of deep learning, usually under the name weight decay. The ML branch's L1 & L2 regularization article derives ridge regression in closed form; here the question is what the penalty does to a network trained by gradient descent: which directions in weight space it actually shrinks, how it relates to the weight_decay argument of an optimizer, and why it behaves strangely in front of normalization layers.
For linear models the penalty is Tikhonov regularization (1943) and ridge regression (Hoerl and Kennard, 1970). In neural networks it appeared as "weight decay" in the late 1980s (Hanson and Pratt, 1988), and Krogh and Hertz (1991, "A Simple Weight Decay Can Improve Generalization") showed that it suppresses irrelevant components of the weight vector and part of the effect of target noise. The eigendirection analysis below follows Goodfellow, Bengio and Courville's Deep Learning (2016, §7.1.1).
Objective and gradient
- L2 strength (
weight_decayin PyTorch optimizers); the makes the gradient exactly - the penalty's pull: proportional to the weight, so it fades as the weight approaches 0
- for plain SGD the penalty is a multiplicative shrink of every weight per step: weight decay
Bayesian view. With a Gaussian prior , . Minimizing the summed negative log-likelihood plus is therefore the MAP estimate with : a strong penalty is a narrow prior. If your loss is a mean over examples, the equivalent is , which is one reason good values look tiny in practice.
At a glance
- Penalty
- ; gradient
- Prior
- Gaussian per weight (MAP), for a summed loss
- Effect near a minimum
- eigen-coordinate of scaled by
- Sparsity
- none: weights shrink geometrically but do not reach 0
- PyTorch
torch.optim.SGD(..., weight_decay=λ)adds to the gradient, exactly L2. Adam'sweight_decayis also L2 (not decoupled);AdamWis decoupled- Typical λ
- (ResNet on ImageNet, SGD) to (AlexNet, many CIFAR recipes)
Which directions get shrunk
Expand the loss to second order around its minimum : . Setting gives the regularized minimum. Write with eigenvectors and curvatures :
- curvature of the loss along : how much the training loss rises if you move along that direction
- shrink factor. if (the data care), if (the data barely notice)
- the effective number of parameters, between 0 and the dimension
L2 does not shrink all weights equally. It removes the components of that the training loss is nearly indifferent to: precisely the directions that noise in a finite training set can push around freely. In an over-parameterized network most Hessian eigenvalues are close to zero, so a small already fixes those directions near 0 while leaving the few stiff directions that carry the fit almost untouched.
Worked example
| α | h₁/(h₁+α) | h₂/(h₂+α) | effective params | w̃ | ‖w̃‖ |
|---|---|---|---|---|---|
| 0 | 1.000 | 1.000 | 2.000 | (1.600, 1.400) | 2.126 |
| 0.1 | 0.952 | 0.500 | 1.452 | (1.617, 1.172) | 1.997 |
| 0.3 | 0.870 | 0.250 | 1.120 | (1.519, 0.996) | 1.817 |
| 1 | 0.667 | 0.091 | 0.758 | (1.185, 0.728) | 1.391 |
| 2 | 0.500 | 0.048 | 0.548 | (0.893, 0.538) | 1.043 |
Look at the row : the first raw weight grows, from 1.600 to 1.617. L2 shrinks the norm and every eigen-coordinate, but not necessarily each individual weight. Removing most of the flat-direction component rotates toward , and happens to have a larger first entry. "L2 makes every weight smaller" is only true when the Hessian is diagonal.
In the loss or in the optimizer?
There are two places to put L2. You can add 0.5 * lam * sum(p.pow(2).sum() ...) to the loss, or pass weight_decay=lam to the optimizer. For torch.optim.SGD the two are identical, with or without momentum: PyTorch adds to the gradient before the momentum buffer, exactly what autograd would do. The optimizer version is cheaper (no extra graph) and can be set per parameter group, so you can exempt biases and normalization parameters.
For adaptive optimizers the two are still identical to each other (Adam's weight_decay is also L2 added to the gradient), but both differ from true weight decay. Adam divides the penalty gradient by , so weights with large gradients are barely regularized. Decoupling the decay from the gradient is the point of AdamW; the Weight decay article covers the difference.
L2 in front of BatchNorm: a learning-rate knob
If a weight vector feeds a BatchNorm or LayerNorm, the network's output is unchanged when is rescaled: for . Two consequences follow by differentiating that identity: (the gradient is orthogonal to ) and (it shrinks as grows). The penalty therefore cannot limit what the layer computes, because every scale gives the same function.
What it does change is the optimization. The direction moves by about per step, so the effective learning rate is . Without the penalty, orthogonal gradient steps only lengthen and training slows down by itself; with it, settles at an equilibrium and the effective learning rate stays up (van Laarhoven, 2017; Zhang, Wang, Xu and Grosse, 2019). The Weight decay figure simulates exactly this.
1import torch, torch.nn as nn23# Same one-step result both ways: w = 2, grad 0.5, lr 0.1, lam 0.14for how in ("loss", "optimizer"):5 w = nn.Parameter(torch.tensor([2.0]))6 opt = torch.optim.SGD([w], lr=0.1, momentum=0.9,7 weight_decay=0.1 if how == "optimizer" else 0.0)8 loss = 0.5 * w.sum() # dL/dw = 0.59 if how == "loss":10 loss = loss + 0.5 * 0.1 * w.pow(2).sum() # (lam/2)||w||^211 loss.backward()12 opt.step()13 print(how, round(w.item(), 4)) # both 1.93 = 2 - 0.1 * (0.5 + 0.1 * 2)1415# In practice: decay matrices and conv kernels, not biases or norm parameters16model = nn.Sequential(nn.Conv2d(3, 64, 3, padding=1), nn.BatchNorm2d(64), nn.ReLU(),17 nn.Flatten(), nn.Linear(64 * 32 * 32, 10))18decay = [p for p in model.parameters() if p.ndim >= 2]19no_decay = [p for p in model.parameters() if p.ndim < 2]20opt = torch.optim.SGD([{"params": decay, "weight_decay": 5e-4},21 {"params": no_decay, "weight_decay": 0.0}],22 lr=0.1, momentum=0.9, nesterov=True)Use when
- +Smooth and differentiable: works with any gradient method, no special update needed
- +Removes exactly the low-curvature directions that overfitting exploits
- +Clear Bayesian meaning (Gaussian prior) and a closed form for quadratic losses
- +For SGD it is a one-line optimizer argument with per-group control
Avoid when
- −No sparsity: every weight stays non-zero
- −Inside Adam it is rescaled by and loses its meaning; use AdamW
- −For weights before normalization layers it does not regularize the function, only the effective learning rate
- −λ interacts with the learning rate, the batch size and the schedule, so values do not transfer between recipes
Check yourself
Near a minimum the Hessian has eigenvalues 4 and 0.04. With L2 strength λ = 0.04, by what factors are the two eigen-coordinates of w* scaled?
Check yourself
A conv kernel w feeds a BatchNorm layer. You double λ on w. What changes?
Remember
- ✓L2 adds ; its gradient shrinks weights in proportion to their size and never zeroes them.
- ✓It is MAP estimation with a Gaussian prior, for a summed loss.
- ✓Near a minimum it scales each Hessian eigen-coordinate by : flat directions vanish, stiff ones stay.
- ✓
SGD(weight_decay=λ)is exactly L2 in the loss (even with momentum); for Adam neither is true weight decay, so use AdamW. - ✓Before BatchNorm/LayerNorm the loss is scale-invariant, and L2 acts only through the effective learning rate .