L1 regularization

Add λ‖w‖₁ to the loss: every weight feels a constant pull of size λ toward zero, so weak weights end at exactly 0. Exact zeros need a proximal (soft-threshold) step; plain SGD on the subgradient only hovers near zero.

11 min read964 wordscore0 viewsupdated 2026-10-11

Definition

L1 regularization adds the sum of absolute weights to the training loss, J(w)=L(w)+λ∥w∥1=L(w)+λ∑j∣wj∣J(w)=L(w)+\lambda\|w\|_1=L(w)+\lambda\sum_j|w_j|. Its gradient is a constant-size push λ sign⁡(wj)\lambda\,\operatorname{sign}(w_j) toward zero, independent of how large the weight is, so weights whose data gradient is weaker than λ\lambda are driven to exactly zero.

The ML branch's L1 & L2 regularization article covers the linear-model story: the lasso, its diamond-shaped constraint region and coordinate descent. This article is about what happens when you put the same penalty on a neural network trained with stochastic gradient methods. Two things change. The optimizer no longer solves the problem exactly, so whether you get real zeros depends on how the penalty enters the update. And the zeros become a tool for pruning: removing weights, channels or whole units after training.

The penalty was popularized for regression by Tibshirani's lasso (1996). Williams (1995, "Bayesian regularization and pruning using a Laplace prior") had already used it in neural networks to prune weights. For large-scale stochastic training the key results are about the update rule: the proximal-gradient / iterative soft-thresholding scheme (Daubechies, Defrise and De Mol, 2004), truncated gradient for online learning (Langford, Li and Zhang, 2009), and the cumulative-penalty SGD of Tsuruoka, Tsujii and Ananiadou (2009). All three exist because naive SGD on ∣w∣|w| does not produce sparse weights.

Objective and (sub)gradient

J(w)=L(w)+λ∑j∣wj∣,∂wjJ=∂L∂wj+λ sj,sj∈{{+1}wj>0{−1}wj<0[−1,1]wj=0J(w)=L(w)+\lambda\sum_j|w_j|,\qquad \partial_{w_j}J=\frac{\partial L}{\partial w_j}+\lambda\,s_j,\quad s_j\in\begin{cases}\{+1\}&w_j>0\\ \{-1\}&w_j<0\\ [-1,1]&w_j=0\end{cases}
L(w)L(w)
the data loss, for example cross-entropy averaged over the mini-batch
λ\lambda
the L1 strength; same units as the loss per unit of weight
sjs_j
a subgradient of ∣wj∣|w_j|. At wj=0w_j=0 any value in [−1,1][-1,1] is valid; autograd uses sign⁡(0)=0\operatorname{sign}(0)=0

Optimality makes the sparsity explicit. A weight can rest at wj=0w_j=0 if some sj∈[−1,1]s_j\in[-1,1] cancels the data gradient, i.e. if ∣∂L/∂wj∣≤λ|\partial L/\partial w_j|\le\lambda there. Any weight whose loss gradient at zero is weaker than λ\lambda stays exactly off. L2 has no such dead zone: its pull λwj\lambda w_j vanishes as wj→0w_j\to0, so it never wins against a nonzero data gradient.

For gradient methods the clean way to handle the kink is the proximal step: take the ordinary gradient step on LL, then solve the penalty exactly. For ηλ∣w∣\eta\lambda|w| the solution is soft-thresholding:

wj←Sηλ ⁣(wj−η ∂L∂wj),Sτ(z)=sign⁡(z) max⁡(∣z∣−τ, 0)=arg⁡min⁡u  12(u−z)2+τ∣u∣w_j\leftarrow S_{\eta\lambda}\!\left(w_j-\eta\,\frac{\partial L}{\partial w_j}\right),\qquad S_\tau(z)=\operatorname{sign}(z)\,\max(|z|-\tau,\,0)=\arg\min_u\;\tfrac12(u-z)^2+\tau|u|
η\eta
learning rate
SτS_\tau
soft-threshold: shrink the magnitude by τ=ηλ\tau=\eta\lambda and clip at 0. PyTorch: F.softshrink(z, lambd=τ)

At a glance

Penalty
λ∥w∥1\lambda\|w\|_1; gradient λsign⁡(w)\lambda\operatorname{sign}(w), constant size
Effect
subtracts a fixed amount per step from every magnitude; weights with ∣∂L/∂w∣≤λ|\partial L/\partial w|\le\lambda at 0 stay at 0
Bayesian view
MAP estimate under a Laplace prior p(wj)∝e−∣wj∣/bp(w_j)\propto e^{-|w_j|/b}, with λ=1/b\lambda=1/b for a summed negative log-likelihood
Exact zeros
only with a proximal / truncation step; autograd on w.abs().sum() leaves weights chattering at size ≤ηλ\le\eta\lambda
Main uses in DL
sparse first layers / feature selection, sparse activations, pruning (L1 on weights or on BatchNorm scales), sparse autoencoders
Cost
one abs and sum forward, one sign backward; the prox is one elementwise op after the step
16 weights at step 800 · ┃ true weight · ▭ no penalty · ● exactly 0-2-101212345678910111213141516half-MSE vs step00.511.520400800valtraindashed: no penaltyexact zeros 7 / 16 · |w| < 0.01: 7 · ‖w‖₁ = 4.533train 0.223 (no penalty 0.080) · val 0.512 (no penalty 1.480)
|η = 0.05 · threshold ηλ = 0.0050
fig.A 16-input linear layer fit by full-batch gradient descent (η = 0.05, 800 steps) to 20 noisy examples whose true weight vector has only 4 non-zero entries (black ticks; blue bars are those 4, orange bars the 12 irrelevant ones, dashed outlines the unpenalized fit). Without a penalty the layer overfits: train 0.080, validation 1.480. With λ = 0.1 and the proximal step, 7 of the irrelevant weights are exactly 0 (green dots), all 4 real ones survive, and validation drops to 0.512. Switch to the subgradient (autograd) version: the losses are the same to two decimals (val 0.514) but the exact-zero count is 0, while 7 weights sit below 0.01. Switch to mini-batches of 5: even the proximal step keeps only 2 exact zeros, because gradient noise kicks weights back out of the dead zone. At λ = 0.3 the proximal fit has 12 zeros but has started to underfit (val 0.723).

Why plain SGD rarely gives exact zeros

Writing loss = loss + lam * w.abs().sum() and calling backward() gives the subgradient update w←w−η(∂L/∂w+λsign⁡w)w\leftarrow w-\eta(\partial L/\partial w+\lambda\operatorname{sign}w). When ∣w∣<ηλ|w|<\eta\lambda that step jumps over zero instead of stopping at it: a weight at 0.020.02 with threshold 0.050.05 lands at −0.03-0.03, then at +0.02+0.02, and so on forever. The update only produces an exact zero if the arithmetic happens to hit 0.00.0, which essentially never happens in floating point.

Mini-batches add a second effect. Each batch gives a noisy ∂L/∂wj\partial L/\partial w_j, and every time its magnitude exceeds λ\lambda the weight is pushed out of zero, even with a proximal step. The zeros are there in the exact solution of the full-batch problem, but SGD only visits the neighbourhood of it. In practice people therefore either use a proximal or truncated update with a decaying learning rate, or train with L1 and then threshold: set every weight below some small ε\varepsilon to zero and fine-tune.

stepsubgradient: w − ηλ·sign(w)proximal: S₀.₀₅(w)
00.120.12
10.070.07
20.020.02
3−0.030
40.020
5−0.030
60.020
An irrelevant weight (data gradient 0) starting at 0.12, with η = 0.1 and λ = 0.5, so the threshold is ηλ = 0.05 (computed in Node with the figure's update functions)

Both rules subtract the same 0.05 per step while the weight is large. They differ only at the end: soft-thresholding clips the last step at zero, the subgradient overshoots and then oscillates between 0.020.02 and −0.03-0.03 with amplitude set by ηλ\eta\lambda. A decaying learning rate shrinks that oscillation, which is why subgradient L1 gives weights that are small but not zero.

The Laplace prior and what L1 is not

Minimizing a summed negative log-likelihood plus λ∥w∥1\lambda\|w\|_1 is MAP estimation with independent Laplace priors p(wj)=12be−∣wj∣/bp(w_j)=\tfrac{1}{2b}e^{-|w_j|/b}, λ=1/b\lambda=1/b. The Laplace density has a sharp peak at 0 and heavier tails than a Gaussian, so it tolerates a few large weights while pulling the rest to the peak. The sparsity is a property of the mode: the posterior mean under a Laplace prior is not sparse. Bayesian methods that want truly sparse posteriors use spike-and-slab or horseshoe priors instead.

In deep networks L1 on weights is less common than L2. With many redundant weights, the loss alone rarely makes any single weight irrelevant, and unstructured zeros scattered through a dense matrix do not speed up a GPU. Its main uses are where the zeros line up with something you can remove.

Where it's used

Pruning

Han et al. (2015, "Learning both weights and connections") trained with a weight penalty, removed small-magnitude weights and retrained. They found L1 gave better accuracy immediately after pruning, L2 better after retraining. torch.nn.utils.prune.l1_unstructured does the magnitude cut.

Channel pruning

Network slimming (Liu et al., 2017) puts L1 on the BatchNorm scale γ\gamma of every channel; channels whose γ\gamma is driven to (near) zero are deleted, giving a genuinely smaller network.

Structured sparsity

Group lasso λ∑g∥wg∥2\lambda\sum_g\|w_g\|_2 zeroes whole filters, rows or units at once (Wen et al., 2016). It is L1 applied to the group norms.

Sparse codes

An L1 penalty on activations, not weights, gives sparse autoencoders and sparse dictionary features (see Sparse autoencoder).

Worked example: one update, three rules

η = 0.1, λ = 0.5, threshold ηλ = 0.05

  1. 1Strong weight w=0.30w=0.30 with data gradient g=0.5g=0.5. SGD without penalty: 0.30−0.05=0.250.30-0.05=0.25. Subgradient: 0.25−0.05=0.200.25-0.05=0.20. Proximal: S0.05(0.25)=0.20S_{0.05}(0.25)=0.20. Away from zero the two L1 rules are identical.
  2. 2Weak weight w=0.03w=0.03, g=0.1g=0.1. SGD: 0.020.02. Subgradient: 0.02−0.05=−0.030.02-0.05=-0.03, the weight changed sign. Proximal: ∣0.02∣≤0.05|0.02|\le0.05, so S0.05(0.02)=0S_{0.05}(0.02)=0 exactly.
  3. 3Negative weight w=−0.4w=-0.4, g=−0.2g=-0.2. SGD: −0.38-0.38. Both L1 rules: −0.38+0.05=−0.33-0.38+0.05=-0.33. The pull is always toward zero with the same size ηλ\eta\lambda, whatever the sign.
python
1import torch, torch.nn as nn, torch.nn.functional as F
2import torch.nn.utils.prune as prune
3
4model = nn.Sequential(nn.Linear(784, 256), nn.ReLU(), nn.Linear(256, 10))
5weights = [p for p in model.parameters() if p.ndim >= 2] # leave biases alone
6lam, lr = 1e-4, 0.1
7opt = torch.optim.SGD(model.parameters(), lr=lr)
8
9def train_step(x, y, proximal=True):
10 loss = F.cross_entropy(model(x), y)
11 if not proximal:
12 # (a) subgradient: small weights chatter around 0, almost never exactly 0
13 loss = loss + lam * sum(w.abs().sum() for w in weights)
14 opt.zero_grad()
15 loss.backward()
16 opt.step()
17 if proximal:
18 # (b) proximal / soft-threshold: weights with |w| <= lr*lam become exactly 0
19 with torch.no_grad():
20 for w in weights:
21 w.copy_(F.softshrink(w, lambd=lr * lam))
22 return loss.item()
23
24# Sparsity report
25zeros = sum((w == 0).sum().item() for w in weights)
26total = sum(w.numel() for w in weights)
27print(f"exact zeros: {zeros / total:.1%}")
28
29# (c) magnitude pruning: zero the 50% smallest |w| of the first layer, keep a mask
30prune.l1_unstructured(model[0], name="weight", amount=0.5)
31prune.remove(model[0], "weight") # make it permanent
L1 three ways: autograd penalty, proximal step after the optimizer, and pruning

Use when

  • +Produces exact zeros (with a proximal step): built-in feature selection and pruning
  • +Robust to a few large weights: the pull does not grow with ∣w∣|w|
  • +Applied to groups (filters, channels, BN scales) it removes whole structures and gives real speed-ups
  • +Cheap: one elementwise op per step

Avoid when

  • −Non-differentiable at 0: autograd only gives a subgradient and no exact zeros
  • −Mini-batch noise and momentum keep knocking weights out of zero
  • −Unstructured zeros in dense layers save little time on GPUs
  • −Among correlated features it picks one somewhat arbitrarily; usually worse than L2 for plain accuracy in deep nets

Check yourself

You add lam * w.abs().sum() to the loss and train with SGD (lr 0.1, lam 0.5). An irrelevant weight is at 0.02. What happens over the next steps?

Check yourself

At the optimum of L(w) + λ‖w‖₁, when is w_j = 0?

Remember

  • ✓L1 adds λ∥w∥1\lambda\|w\|_1; its pull λsign⁡(w)\lambda\operatorname{sign}(w) has constant size, so weak weights reach exactly zero.
  • ✓Zero is optimal for wjw_j whenever ∣∂L/∂wj∣≤λ|\partial L/\partial w_j|\le\lambda there.
  • ✓Use a proximal step, w←Sηλ(w−ηg)w\leftarrow S_{\eta\lambda}(w-\eta g) (F.softshrink), to get exact zeros; autograd on abs() only chatters around 0.
  • ✓Mini-batch noise and adaptive optimizers blur the zeros; threshold and fine-tune at the end.
  • ✓L1 is MAP with a Laplace prior. In deep learning it is mainly a pruning tool, best applied to groups such as channels or BN scales.
← back to Deep Learning roadmap