# L1 regularization

> Add λ‖w‖₁ to the loss: every weight feels a constant pull of size λ toward zero, so weak weights end at exactly 0. Exact zeros need a proximal (soft-threshold) step; plain SGD on the subgradient only hovers near zero.

Source: https://www.mlfoundry.in/learn/l1-regularization  
Section: Deep Learning › Regularization & Normalization  
Level: core · 11 min read · updated 2026-10-11

**L1 regularization:** adds the sum of absolute weights to the training loss, $J(w)=L(w)+\lambda\|w\|_1=L(w)+\lambda\sum_j|w_j|$. Its gradient is a constant-size push $\lambda\,\operatorname{sign}(w_j)$ toward zero, independent of how large the weight is, so weights whose data gradient is weaker than $\lambda$ are driven to **exactly** zero.

The ML branch's **L1 & L2 regularization** article covers the linear-model story: the lasso, its diamond-shaped constraint region and coordinate descent. This article is about what happens when you put the same penalty on a neural network trained with stochastic gradient methods. Two things change. The optimizer no longer solves the problem exactly, so whether you get real zeros depends on **how** the penalty enters the update. And the zeros become a tool for **pruning**: removing weights, channels or whole units after training.

The penalty was popularized for regression by Tibshirani's lasso (1996). Williams (1995, "Bayesian regularization and pruning using a Laplace prior") had already used it in neural networks to prune weights. For large-scale stochastic training the key results are about the update rule: the proximal-gradient / iterative soft-thresholding scheme (Daubechies, Defrise and De Mol, 2004), truncated gradient for online learning (Langford, Li and Zhang, 2009), and the cumulative-penalty SGD of Tsuruoka, Tsujii and Ananiadou (2009). All three exist because naive SGD on $|w|$ does not produce sparse weights.

## Objective and (sub)gradient

$$
J(w)=L(w)+\lambda\sum_j|w_j|,\qquad \partial_{w_j}J=\frac{\partial L}{\partial w_j}+\lambda\,s_j,\quad s_j\in\begin{cases}\{+1\}&w_j>0\\ \{-1\}&w_j<0\\ [-1,1]&w_j=0\end{cases}
$$

- $L(w)$: the data loss, for example cross-entropy averaged over the mini-batch
- $\lambda$: the L1 strength; same units as the loss per unit of weight
- $s_j$: a subgradient of $|w_j|$. At $w_j=0$ any value in $[-1,1]$ is valid; autograd uses $\operatorname{sign}(0)=0$

Optimality makes the sparsity explicit. A weight can rest at $w_j=0$ if some $s_j\in[-1,1]$ cancels the data gradient, i.e. if $|\partial L/\partial w_j|\le\lambda$ there. Any weight whose loss gradient at zero is weaker than $\lambda$ stays exactly off. L2 has no such dead zone: its pull $\lambda w_j$ vanishes as $w_j\to0$, so it never wins against a nonzero data gradient.

For gradient methods the clean way to handle the kink is the **proximal step**: take the ordinary gradient step on $L$, then solve the penalty exactly. For $\eta\lambda|w|$ the solution is **soft-thresholding**:

$$
w_j\leftarrow S_{\eta\lambda}\!\left(w_j-\eta\,\frac{\partial L}{\partial w_j}\right),\qquad S_\tau(z)=\operatorname{sign}(z)\,\max(|z|-\tau,\,0)=\arg\min_u\;\tfrac12(u-z)^2+\tau|u|
$$

- $\eta$: learning rate
- $S_\tau$: soft-threshold: shrink the magnitude by $\tau=\eta\lambda$ and clip at 0. PyTorch: `F.softshrink(z, lambd=τ)`

**At a glance**

- **Penalty:** $\lambda\|w\|_1$; gradient $\lambda\operatorname{sign}(w)$, constant size
- **Effect:** subtracts a fixed amount per step from every magnitude; weights with $|\partial L/\partial w|\le\lambda$ at 0 stay at 0
- **Bayesian view:** MAP estimate under a Laplace prior $p(w_j)\propto e^{-|w_j|/b}$, with $\lambda=1/b$ for a summed negative log-likelihood
- **Exact zeros:** only with a proximal / truncation step; autograd on `w.abs().sum()` leaves weights chattering at size $\le\eta\lambda$
- **Main uses in DL:** sparse first layers / feature selection, sparse activations, pruning (L1 on weights or on BatchNorm scales), sparse autoencoders
- **Cost:** one abs and sum forward, one sign backward; the prox is one elementwise op after the step

> Figure: A 16-input linear layer fit by full-batch gradient descent (η = 0.05, 800 steps) to 20 noisy examples whose true weight vector has only 4 non-zero entries (black ticks; blue bars are those 4, orange bars the 12 irrelevant ones, dashed outlines the unpenalized fit). Without a penalty the layer overfits: train 0.080, validation 1.480. With λ = 0.1 and the proximal step, 7 of the irrelevant weights are exactly 0 (green dots), all 4 real ones survive, and validation drops to 0.512. Switch to the subgradient (autograd) version: the losses are the same to two decimals (val 0.514) but the exact-zero count is 0, while 7 weights sit below 0.01. Switch to mini-batches of 5: even the proximal step keeps only 2 exact zeros, because gradient noise kicks weights back out of the dead zone. At λ = 0.3 the proximal fit has 12 zeros but has started to underfit (val 0.723).

## Why plain SGD rarely gives exact zeros

Writing `loss = loss + lam * w.abs().sum()` and calling `backward()` gives the subgradient update $w\leftarrow w-\eta(\partial L/\partial w+\lambda\operatorname{sign}w)$. When $|w|<\eta\lambda$ that step **jumps over** zero instead of stopping at it: a weight at $0.02$ with threshold $0.05$ lands at $-0.03$, then at $+0.02$, and so on forever. The update only produces an exact zero if the arithmetic happens to hit $0.0$, which essentially never happens in floating point.

Mini-batches add a second effect. Each batch gives a noisy $\partial L/\partial w_j$, and every time its magnitude exceeds $\lambda$ the weight is pushed out of zero, even with a proximal step. The zeros are there in the exact solution of the full-batch problem, but SGD only visits the neighbourhood of it. In practice people therefore either use a proximal or truncated update with a decaying learning rate, or train with L1 and then **threshold**: set every weight below some small $\varepsilon$ to zero and fine-tune.

An irrelevant weight (data gradient 0) starting at 0.12, with η = 0.1 and λ = 0.5, so the threshold is ηλ = 0.05 (computed in Node with the figure's update functions)

| step | subgradient: w − ηλ·sign(w) | proximal: S₀.₀₅(w) |
| --- | --- | --- |
| 0 | 0.12 | 0.12 |
| 1 | 0.07 | 0.07 |
| 2 | 0.02 | 0.02 |
| 3 | −0.03 | **0** |
| 4 | 0.02 | 0 |
| 5 | −0.03 | 0 |
| 6 | 0.02 | 0 |

Both rules subtract the same 0.05 per step while the weight is large. They differ only at the end: soft-thresholding clips the last step at zero, the subgradient overshoots and then oscillates between $0.02$ and $-0.03$ with amplitude set by $\eta\lambda$. A decaying learning rate shrinks that oscillation, which is why subgradient L1 gives weights that are small but not zero.

## The Laplace prior and what L1 is not

Minimizing a summed negative log-likelihood plus $\lambda\|w\|_1$ is MAP estimation with independent Laplace priors $p(w_j)=\tfrac{1}{2b}e^{-|w_j|/b}$, $\lambda=1/b$. The Laplace density has a sharp peak at 0 and heavier tails than a Gaussian, so it tolerates a few large weights while pulling the rest to the peak. The sparsity is a property of the **mode**: the posterior mean under a Laplace prior is not sparse. Bayesian methods that want truly sparse posteriors use spike-and-slab or horseshoe priors instead.

In deep networks L1 on weights is less common than L2. With many redundant weights, the loss alone rarely makes any single weight irrelevant, and unstructured zeros scattered through a dense matrix do not speed up a GPU. Its main uses are where the zeros line up with something you can remove.

- **Pruning:** Han et al. (2015, "Learning both weights and connections") trained with a weight penalty, removed small-magnitude weights and retrained. They found L1 gave better accuracy immediately after pruning, L2 better after retraining. `torch.nn.utils.prune.l1_unstructured` does the magnitude cut.
- **Channel pruning:** Network slimming (Liu et al., 2017) puts L1 on the BatchNorm scale $\gamma$ of every channel; channels whose $\gamma$ is driven to (near) zero are deleted, giving a genuinely smaller network.
- **Structured sparsity:** Group lasso $\lambda\sum_g\|w_g\|_2$ zeroes whole filters, rows or units at once (Wen et al., 2016). It is L1 applied to the group norms.
- **Sparse codes:** An L1 penalty on activations, not weights, gives sparse autoencoders and sparse dictionary features (see **Sparse autoencoder**).

## Worked example: one update, three rules

**η = 0.1, λ = 0.5, threshold ηλ = 0.05**

1. Strong weight $w=0.30$ with data gradient $g=0.5$. SGD without penalty: $0.30-0.05=0.25$. Subgradient: $0.25-0.05=0.20$. Proximal: $S_{0.05}(0.25)=0.20$. Away from zero the two L1 rules are identical.
2. Weak weight $w=0.03$, $g=0.1$. SGD: $0.02$. Subgradient: $0.02-0.05=-0.03$, the weight **changed sign**. Proximal: $|0.02|\le0.05$, so $S_{0.05}(0.02)=0$ exactly.
3. Negative weight $w=-0.4$, $g=-0.2$. SGD: $-0.38$. Both L1 rules: $-0.38+0.05=-0.33$. The pull is always toward zero with the same size $\eta\lambda$, whatever the sign.

```python
import torch, torch.nn as nn, torch.nn.functional as F
import torch.nn.utils.prune as prune

model = nn.Sequential(nn.Linear(784, 256), nn.ReLU(), nn.Linear(256, 10))
weights = [p for p in model.parameters() if p.ndim >= 2]      # leave biases alone
lam, lr = 1e-4, 0.1
opt = torch.optim.SGD(model.parameters(), lr=lr)

def train_step(x, y, proximal=True):
    loss = F.cross_entropy(model(x), y)
    if not proximal:
        # (a) subgradient: small weights chatter around 0, almost never exactly 0
        loss = loss + lam * sum(w.abs().sum() for w in weights)
    opt.zero_grad()
    loss.backward()
    opt.step()
    if proximal:
        # (b) proximal / soft-threshold: weights with |w| <= lr*lam become exactly 0
        with torch.no_grad():
            for w in weights:
                w.copy_(F.softshrink(w, lambd=lr * lam))
    return loss.item()

# Sparsity report
zeros = sum((w == 0).sum().item() for w in weights)
total = sum(w.numel() for w in weights)
print(f"exact zeros: {zeros / total:.1%}")

# (c) magnitude pruning: zero the 50% smallest |w| of the first layer, keep a mask
prune.l1_unstructured(model[0], name="weight", amount=0.5)
prune.remove(model[0], "weight")                              # make it permanent
```

L1 three ways: autograd penalty, proximal step after the optimizer, and pruning

> **Proximal steps and adaptive optimizers:** Soft-thresholding by $\eta\lambda$ is the exact prox only for plain SGD. With Adam the effective step of each weight is $\eta/\sqrt{\hat v}$, so a single global threshold over- or under-shrinks; with momentum the velocity keeps pushing weights out of zero. Either use a per-parameter threshold $\eta\lambda/(\sqrt{\hat v}+\epsilon)$, or accept approximate sparsity and finish with a magnitude cut plus fine-tuning.

> **Constant pull versus proportional pull:** L1 removes the same amount $\eta\lambda$ per step from every weight; L2 removes the same **fraction** $\eta\lambda$. A small weight under L1 loses a large share of its value each step and is gone quickly; under L2 it decays geometrically and never reaches zero. That one difference is the whole sparsity story.

> **Scale of λ:** The penalty is summed over all weights while the loss is averaged over examples, so useful $\lambda$ values for a whole network are small, often $10^{-6}$ to $10^{-4}$. Too large a $\lambda$ kills entire layers: every weight's data gradient is below the threshold and the network outputs a constant.

**Strengths**

- Produces exact zeros (with a proximal step): built-in feature selection and pruning
- Robust to a few large weights: the pull does not grow with $|w|$
- Applied to groups (filters, channels, BN scales) it removes whole structures and gives real speed-ups
- Cheap: one elementwise op per step

**Limitations**

- Non-differentiable at 0: autograd only gives a subgradient and no exact zeros
- Mini-batch noise and momentum keep knocking weights out of zero
- Unstructured zeros in dense layers save little time on GPUs
- Among correlated features it picks one somewhat arbitrarily; usually worse than L2 for plain accuracy in deep nets

**Question:** You add lam * w.abs().sum() to the loss and train with SGD (lr 0.1, lam 0.5). An irrelevant weight is at 0.02. What happens over the next steps?

- A. It becomes exactly 0 on the next step
- B. It oscillates between about −0.03 and 0.02
- C. It decays geometrically toward 0 without reaching it
- D. It stays at 0.02 because its data gradient is 0

**Answer:** B. It oscillates between about −0.03 and 0.02 — The subgradient step subtracts $\eta\lambda\operatorname{sign}(w)=0.05$, so $0.02\to-0.03$, then $+0.05$ brings it back to $0.02$. Only a proximal step $S_{0.05}(0.02)=0$ stops it at exactly zero. Geometric decay is what L2 does.

**Question:** At the optimum of L(w) + λ‖w‖₁, when is w_j = 0?

- A. When |∂L/∂w_j| ≤ λ at w_j = 0
- B. When ∂L/∂w_j = 0 exactly
- C. When λ is larger than |w_j| of the unregularized solution
- D. Never: L1 only shrinks weights

**Answer:** A. When |∂L/∂w_j| ≤ λ at w_j = 0 — At $w_j=0$ the subgradient of $|w_j|$ is the whole interval $[-1,1]$, so $0\in\partial L/\partial w_j+\lambda[-1,1]$ exactly when $|\partial L/\partial w_j|\le\lambda$. That dead zone is why L1 gives exact zeros and L2 does not.

**Key points**

- L1 adds $\lambda\|w\|_1$; its pull $\lambda\operatorname{sign}(w)$ has constant size, so weak weights reach exactly zero.
- Zero is optimal for $w_j$ whenever $|\partial L/\partial w_j|\le\lambda$ there.
- Use a proximal step, $w\leftarrow S_{\eta\lambda}(w-\eta g)$ (`F.softshrink`), to get exact zeros; autograd on `abs()` only chatters around 0.
- Mini-batch noise and adaptive optimizers blur the zeros; threshold and fine-tune at the end.
- L1 is MAP with a Laplace prior. In deep learning it is mainly a pruning tool, best applied to groups such as channels or BN scales.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
