Definition
L1 regularization adds the sum of absolute weights to the training loss, . Its gradient is a constant-size push toward zero, independent of how large the weight is, so weights whose data gradient is weaker than are driven to exactly zero.
The ML branch's L1 & L2 regularization article covers the linear-model story: the lasso, its diamond-shaped constraint region and coordinate descent. This article is about what happens when you put the same penalty on a neural network trained with stochastic gradient methods. Two things change. The optimizer no longer solves the problem exactly, so whether you get real zeros depends on how the penalty enters the update. And the zeros become a tool for pruning: removing weights, channels or whole units after training.
The penalty was popularized for regression by Tibshirani's lasso (1996). Williams (1995, "Bayesian regularization and pruning using a Laplace prior") had already used it in neural networks to prune weights. For large-scale stochastic training the key results are about the update rule: the proximal-gradient / iterative soft-thresholding scheme (Daubechies, Defrise and De Mol, 2004), truncated gradient for online learning (Langford, Li and Zhang, 2009), and the cumulative-penalty SGD of Tsuruoka, Tsujii and Ananiadou (2009). All three exist because naive SGD on does not produce sparse weights.
Objective and (sub)gradient
- the data loss, for example cross-entropy averaged over the mini-batch
- the L1 strength; same units as the loss per unit of weight
- a subgradient of . At any value in is valid; autograd uses
Optimality makes the sparsity explicit. A weight can rest at if some cancels the data gradient, i.e. if there. Any weight whose loss gradient at zero is weaker than stays exactly off. L2 has no such dead zone: its pull vanishes as , so it never wins against a nonzero data gradient.
For gradient methods the clean way to handle the kink is the proximal step: take the ordinary gradient step on , then solve the penalty exactly. For the solution is soft-thresholding:
- learning rate
- soft-threshold: shrink the magnitude by and clip at 0. PyTorch:
F.softshrink(z, lambd=τ)
At a glance
- Penalty
- ; gradient , constant size
- Effect
- subtracts a fixed amount per step from every magnitude; weights with at 0 stay at 0
- Bayesian view
- MAP estimate under a Laplace prior , with for a summed negative log-likelihood
- Exact zeros
- only with a proximal / truncation step; autograd on
w.abs().sum()leaves weights chattering at size - Main uses in DL
- sparse first layers / feature selection, sparse activations, pruning (L1 on weights or on BatchNorm scales), sparse autoencoders
- Cost
- one abs and sum forward, one sign backward; the prox is one elementwise op after the step
Why plain SGD rarely gives exact zeros
Writing loss = loss + lam * w.abs().sum() and calling backward() gives the subgradient update . When that step jumps over zero instead of stopping at it: a weight at with threshold lands at , then at , and so on forever. The update only produces an exact zero if the arithmetic happens to hit , which essentially never happens in floating point.
Mini-batches add a second effect. Each batch gives a noisy , and every time its magnitude exceeds the weight is pushed out of zero, even with a proximal step. The zeros are there in the exact solution of the full-batch problem, but SGD only visits the neighbourhood of it. In practice people therefore either use a proximal or truncated update with a decaying learning rate, or train with L1 and then threshold: set every weight below some small to zero and fine-tune.
| step | subgradient: w − ηλ·sign(w) | proximal: S₀.₀₅(w) |
|---|---|---|
| 0 | 0.12 | 0.12 |
| 1 | 0.07 | 0.07 |
| 2 | 0.02 | 0.02 |
| 3 | −0.03 | 0 |
| 4 | 0.02 | 0 |
| 5 | −0.03 | 0 |
| 6 | 0.02 | 0 |
Both rules subtract the same 0.05 per step while the weight is large. They differ only at the end: soft-thresholding clips the last step at zero, the subgradient overshoots and then oscillates between and with amplitude set by . A decaying learning rate shrinks that oscillation, which is why subgradient L1 gives weights that are small but not zero.
The Laplace prior and what L1 is not
Minimizing a summed negative log-likelihood plus is MAP estimation with independent Laplace priors , . The Laplace density has a sharp peak at 0 and heavier tails than a Gaussian, so it tolerates a few large weights while pulling the rest to the peak. The sparsity is a property of the mode: the posterior mean under a Laplace prior is not sparse. Bayesian methods that want truly sparse posteriors use spike-and-slab or horseshoe priors instead.
In deep networks L1 on weights is less common than L2. With many redundant weights, the loss alone rarely makes any single weight irrelevant, and unstructured zeros scattered through a dense matrix do not speed up a GPU. Its main uses are where the zeros line up with something you can remove.
Where it's used
Pruning
Han et al. (2015, "Learning both weights and connections") trained with a weight penalty, removed small-magnitude weights and retrained. They found L1 gave better accuracy immediately after pruning, L2 better after retraining. torch.nn.utils.prune.l1_unstructured does the magnitude cut.
Channel pruning
Network slimming (Liu et al., 2017) puts L1 on the BatchNorm scale of every channel; channels whose is driven to (near) zero are deleted, giving a genuinely smaller network.
Structured sparsity
Group lasso zeroes whole filters, rows or units at once (Wen et al., 2016). It is L1 applied to the group norms.
Sparse codes
An L1 penalty on activations, not weights, gives sparse autoencoders and sparse dictionary features (see Sparse autoencoder).
Worked example: one update, three rules
η = 0.1, λ = 0.5, threshold ηλ = 0.05
- 1Strong weight with data gradient . SGD without penalty: . Subgradient: . Proximal: . Away from zero the two L1 rules are identical.
- 2Weak weight , . SGD: . Subgradient: , the weight changed sign. Proximal: , so exactly.
- 3Negative weight , . SGD: . Both L1 rules: . The pull is always toward zero with the same size , whatever the sign.
1import torch, torch.nn as nn, torch.nn.functional as F2import torch.nn.utils.prune as prune34model = nn.Sequential(nn.Linear(784, 256), nn.ReLU(), nn.Linear(256, 10))5weights = [p for p in model.parameters() if p.ndim >= 2] # leave biases alone6lam, lr = 1e-4, 0.17opt = torch.optim.SGD(model.parameters(), lr=lr)89def train_step(x, y, proximal=True):10 loss = F.cross_entropy(model(x), y)11 if not proximal:12 # (a) subgradient: small weights chatter around 0, almost never exactly 013 loss = loss + lam * sum(w.abs().sum() for w in weights)14 opt.zero_grad()15 loss.backward()16 opt.step()17 if proximal:18 # (b) proximal / soft-threshold: weights with |w| <= lr*lam become exactly 019 with torch.no_grad():20 for w in weights:21 w.copy_(F.softshrink(w, lambd=lr * lam))22 return loss.item()2324# Sparsity report25zeros = sum((w == 0).sum().item() for w in weights)26total = sum(w.numel() for w in weights)27print(f"exact zeros: {zeros / total:.1%}")2829# (c) magnitude pruning: zero the 50% smallest |w| of the first layer, keep a mask30prune.l1_unstructured(model[0], name="weight", amount=0.5)31prune.remove(model[0], "weight") # make it permanentUse when
- +Produces exact zeros (with a proximal step): built-in feature selection and pruning
- +Robust to a few large weights: the pull does not grow with
- +Applied to groups (filters, channels, BN scales) it removes whole structures and gives real speed-ups
- +Cheap: one elementwise op per step
Avoid when
- −Non-differentiable at 0: autograd only gives a subgradient and no exact zeros
- −Mini-batch noise and momentum keep knocking weights out of zero
- −Unstructured zeros in dense layers save little time on GPUs
- −Among correlated features it picks one somewhat arbitrarily; usually worse than L2 for plain accuracy in deep nets
Check yourself
You add lam * w.abs().sum() to the loss and train with SGD (lr 0.1, lam 0.5). An irrelevant weight is at 0.02. What happens over the next steps?
Check yourself
At the optimum of L(w) + λ‖w‖₁, when is w_j = 0?
Remember
- ✓L1 adds ; its pull has constant size, so weak weights reach exactly zero.
- ✓Zero is optimal for whenever there.
- ✓Use a proximal step, (
F.softshrink), to get exact zeros; autograd onabs()only chatters around 0. - ✓Mini-batch noise and adaptive optimizers blur the zeros; threshold and fine-tune at the end.
- ✓L1 is MAP with a Laplace prior. In deep learning it is mainly a pruning tool, best applied to groups such as channels or BN scales.