# L1 & L2 regularization

> Penalizing weight size trades a little bias for less variance; L1 also pushes weights to exactly zero.

Source: https://www.mlfoundry.in/learn/l1-l2-regularization  
Section: ML Algorithms › Evaluation & Generalization  
Level: core · 17 min read · updated 2026-10-07

**Regularization:** adds a penalty on the size of a model's weights to its training loss: minimize $\text{loss}(w)+\lambda\,R(w)$. The **L2** penalty $R(w)=\tfrac12\lVert w\rVert_2^2=\tfrac12\sum_j w_j^2$ shrinks every weight smoothly toward zero. The **L1** penalty $R(w)=\lVert w\rVert_1=\sum_j |w_j|$ shrinks them too, and sets the least useful ones to **exactly** zero. One knob, $\lambda\ge 0$, sets how much training fit you give up in exchange for smaller weights.

- **Task:** Not a model of its own: a modification of the training objective of any weight-based model (linear and logistic regression, SVMs, neural networks)
- **Input → output:** a loss $\mathcal L(w)$, a penalty type and a strength $\lambda$ → a weight vector with smaller norm (and, for L1, some zeros)
- **Learns:** the same parameters as the base model, just pulled toward 0
- **Training cost:** L2: almost free (adds $\lambda w$ to the gradient, or $\lambda I$ to the normal equations). L1: the objective has a kink at 0, so it needs coordinate descent or proximal steps instead of plain gradient descent
- **Prediction cost:** unchanged; with L1, features whose weight is 0 need not be computed at all
- **Hyperparameters:** strength λ (sklearn: `alpha`, or `C` = inverse strength); penalty type `l1` / `l2` / elastic-net mix
- **Assumes:** features on comparable scales (standardize first); the intercept is not penalized; the true signal is spread out (L2) or carried by a few features (L1)

## The problem

Least squares and maximum likelihood pick the weights that fit the **training** set best. When there are many features compared with rows, or the features are correlated, there are many weight vectors that fit almost equally well, and the fitting procedure picks one that bends to the noise in this particular sample. Its weights are large and unstable: draw another training set from the same source and they change a lot. That instability is **variance**, and it shows up as a big gap between training and test error.

The overfitting lab below is a concrete case: 20 training rows, 10 correlated features, and only 3 of them actually matter. Ordinary least squares reaches a training MSE of **0.223** but a test MSE of **2.807**, more than twelve times worse. The extra features let it fit noise. We could delete features by hand, but which ones? Regularization makes the model pay for every unit of weight, so a weight has to earn its size by reducing the loss.

## The idea

Picture every weight attached to zero by a spring. The data pulls the weights toward their least-squares values and the springs pull back. With **L2** the pull back is proportional to the weight, like an ordinary spring. Big weights feel a big pull and small ones a gentle one, so every weight shrinks by the same *fraction* and none reaches zero. With **L1** the pull is a constant $\lambda$ whatever the weight's size, like friction. A weight whose data-pull is weaker than $\lambda$ is dragged all the way to zero and stays there. That is why L1 does **feature selection** and L2 only shrinks.

Either way the model becomes a little **biased**, because the weights are systematically too small. In return the weights move much less from one training sample to the next, so the **variance** drops. When the variance saved is larger than the squared bias added, test error goes down. Most of the work is finding the $\lambda$ where that happens.

**Vocabulary**

- **Penalty / regularizer R(w):** the term added to the loss; $\tfrac12\lVert w\rVert_2^2$ (L2, ridge, weight decay) or $\lVert w\rVert_1$ (L1, lasso)
- **λ (regularization strength):** the exchange rate between fit and weight size. $\lambda=0$ is the unregularized model; $\lambda\to\infty$ forces $w=0$
- **Shrinkage:** moving estimates toward 0 (or another fixed value). It always adds bias and reduces variance
- **Sparsity:** having many weights exactly 0. L1 produces it, L2 does not
- **Soft-thresholding:** $S_\lambda(z)=\operatorname{sign}(z)\max(|z|-\lambda,0)$: subtract $\lambda$ from the magnitude and clip at 0. This is what L1 does to each weight
- **Subgradient:** the set of slopes of a kinked function at the kink. For $|w|$ at $w=0$ it is the whole interval $[-1,1]$
- **Weight decay:** the L2 penalty seen from gradient descent: each step multiplies $w$ by $(1-\eta\lambda)$ before the loss gradient is applied
- **Regularization path:** the fitted weights as a function of $\lambda$, from all-zero (large $\lambda$) to the unregularized fit ($\lambda\to0$)
- **λ_max:** for L1, the smallest $\lambda$ at which every weight is 0. The path starts there
- **Effective degrees of freedom:** how many parameters the fit really uses. For L2 it is $\sum_k s_k/(s_k+\lambda)$, between 0 and $p$; for L1 it is roughly the number of non-zero weights

## The math

Write the model as $\hat y=b+x^\top w$ with $p$ standardized features (mean 0, variance 1). The penalized objective is the average loss plus $\lambda$ times the penalty. For squared error:

$$
J(w)=\frac{1}{2n}\sum_{i=1}^{n}\bigl(y_i-b-x_i^\top w\bigr)^2+\lambda\,R(w),\qquad R_2(w)=\tfrac12\lVert w\rVert_2^2,\quad R_1(w)=\lVert w\rVert_1
$$

- $n,\;p$: number of training rows and of features
- $x_i,\;y_i$: feature vector and target of row $i$
- $w,\;b$: weights and intercept. $b$ is **not** penalized; with centred features its optimum is $\bar y$
- $\lambda\ge0$: regularization strength
- $\lVert w\rVert_2^2=\sum_j w_j^2$: squared Euclidean length (L2)
- $\lVert w\rVert_1=\sum_j|w_j|$: sum of absolute values (L1)

The same recipe works for any loss $\mathcal L$: log-loss for logistic regression, hinge loss for SVMs, cross-entropy for a neural network. Only the first term changes.

## Deriving the two shrinkage rules

The difference between L1 and L2 is clearest when the features are **orthonormal**: centred, $\tfrac1n x_j^\top x_j=1$ and $x_j^\top x_k=0$ for $j\ne k$. Designed experiments have this property, and so do principal-component scores after scaling. The unregularized solution is then simply $\hat w_j=\tfrac1n x_j^\top(y-\bar y)$, and expanding the square gives

$$
J(w)=\text{const}+\sum_{j=1}^{p}\Bigl[\tfrac12\,(w_j-\hat w_j)^2+\lambda\,r(w_j)\Bigr]
$$

- $\hat w_j$: the ordinary least-squares weight of feature j
- $r(w_j)$: $\tfrac12 w_j^2$ for L2, $|w_j|$ for L1

The objective splits into $p$ separate one-dimensional problems, so we can solve each weight on its own.

**Solving one coordinate**

1. **L2.** $\tfrac12(w-\hat w)^2+\tfrac\lambda2 w^2$ is a smooth parabola. Set the derivative to zero: $w-\hat w+\lambda w=0$, so $w^\star=\dfrac{\hat w}{1+\lambda}$. Every weight is divided by the same factor, and it reaches 0 only when $\lambda\to\infty$.
2. **L1, case $w>0$.** The derivative is $w-\hat w+\lambda=0$, so $w=\hat w-\lambda$. This is consistent only if $\hat w>\lambda$.
3. **L1, case $w<0$.** By symmetry $w=\hat w+\lambda$, consistent only if $\hat w<-\lambda$.
4. **L1, case $w=0$.** $|w|$ has no derivative at 0; its subgradient there is the interval $[-1,1]$. Zero is optimal when $0\in -\hat w+\lambda[-1,1]$, which means $|\hat w|\le\lambda$. Every OLS weight in the band $[-\lambda,\lambda]$ is set to exactly 0.
5. **Together:** $w^\star=S_\lambda(\hat w)=\operatorname{sign}(\hat w)\max(|\hat w|-\lambda,0)$, the **soft-thresholding** operator.

$$
w^\star_{\text{L2}}=\frac{\hat w}{1+\lambda}\qquad\qquad w^\star_{\text{L1}}=\operatorname{sign}(\hat w)\,\max\bigl(|\hat w|-\lambda,\;0\bigr)
$$

- $w^\star$: the regularized weight (orthonormal features)
- $1/(1+\lambda)$: L2: proportional shrinkage. The same fraction is removed from every weight
- $|\hat w|-\lambda$: L1: constant shrinkage, clipped at 0. The same amount is removed from every weight

For general correlated features there is no per-weight formula. L2 still has a closed form, $w=(\tfrac1n X^\top X+\lambda I)^{-1}\tfrac1n X^\top(y-\bar y)$, and the added $\lambda I$ makes the matrix invertible even when $p>n$ (see **Ridge regression**). L1 is solved by applying soft-thresholding to one coordinate at a time, which is coordinate descent (see **Lasso regression**).

## Why the kink makes zeros: the gradient view

For gradient-trained models such as networks and logistic regression, it helps to look at what each penalty does to a single update with step size $\eta$:

$$
\text{L2: } w\leftarrow(1-\eta\lambda)\,w-\eta\nabla\mathcal L(w)\qquad\qquad\text{L1: } w\leftarrow S_{\eta\lambda}\bigl(w-\eta\nabla\mathcal L(w)\bigr)
$$

- $\eta$: learning rate
- $(1-\eta\lambda)$: L2 multiplies each weight by a factor just below 1 on every step, hence the name weight decay
- $S_{\eta\lambda}$: L1's **proximal step**: take the ordinary gradient step, then soft-threshold by $\eta\lambda$ (the ISTA algorithm)

The L2 pull, $\lambda w$, fades as the weight gets small, so it never quite reaches zero. The L1 pull, $\lambda\operatorname{sign}(w)$, stays the same size all the way down. Once a weight is at 0 it stays there while $|\partial\mathcal L/\partial w_j|\le\lambda$. Gradient descent on $\lambda|w|$ without the proximal step is not enough: the weights oscillate around zero and never land on it. Use a solver that handles the kink.

> **The geometric picture and the Bayesian picture:** **Geometry.** Penalizing is equivalent to constraining: for every $\lambda$ there is a budget $t$ such that minimizing the loss subject to $R(w)\le t$ gives the same answer (a Lagrange multiplier argument). The L2 budget is a sphere and the L1 budget is a diamond with corners on the axes. The loss contours are ellipses, and they usually first touch the diamond at a corner, where some coordinates are 0. **Bayes.** With Gaussian noise of variance $\sigma^2$, the minimizer of $J$ is the MAP estimate under a prior on $w$. A Gaussian prior $\mathcal N(0,\tau^2)$ gives L2 with $\lambda=\sigma^2/(n\tau^2)$. A Laplace prior with scale $\beta$, which has a sharp peak at 0, gives L1 with $\lambda=\sigma^2/(n\beta)$. A strong prior belief that weights are small is the same thing as a large $\lambda$.

## What λ does

**Reading the knob**

- **λ = 0:** no penalty: the ordinary least-squares or maximum-likelihood fit, with the most variance
- **Small λ:** removes the wild, noise-driven weight directions first. L2 shrinks most along directions where the data has little spread (small eigenvalues of XᵀX/n)
- **Moderate λ:** usually the sweet spot: test error is lowest, and L1 has dropped the irrelevant features
- **λ ≥ λ_max (L1):** every weight is 0 and the model predicts $\bar y$. $\lambda_{\max}=\max_j\bigl|\tfrac1n x_j^\top(y-\bar y)\bigr|$
- **λ → ∞ (L2):** weights approach 0 but are never exactly 0
- **Effective size:** L2 with eigenvalues $s_k$ of $\tfrac1n X^\top X$ keeps $\mathrm{df}(\lambda)=\sum_k \tfrac{s_k}{s_k+\lambda}$ effective parameters. In the lab below this falls from 10 to **8.44** at $\lambda=0.1$ and **4.22** at $\lambda=1$

## Worked example: a crop trial, by hand

An agronomist runs every combination of three two-level treatments: fertilizer $x_1$, irrigation $x_2$ and seed variety $x_3$, each coded $-1$ (low/standard) or $+1$ (high/new). That makes $2^3=8$ plots, and $y$ is the yield in tonnes per hectare. Coding a full factorial design this way makes the columns **orthonormal**, so the formulas above apply exactly and every step can be done by hand.

Eight plots of a 2³ factorial trial. Mean yield ȳ = 10. Highlighted: the plot with all three treatments high, the input we predict at the end.

| Plot | x₁ fertilizer | x₂ irrigation | x₃ variety | y (t/ha) | y − ȳ |
| --- | --- | --- | --- | --- | --- |
| 1 | −1 | −1 | −1 | 7.0 | −3.0 |
| 2 | −1 | −1 | +1 | 6.0 | −4.0 |
| 3 | −1 | +1 | −1 | 6.5 | −3.5 |
| 4 | −1 | +1 | +1 | 8.5 | −1.5 |
| 5 | +1 | −1 | −1 | 11.0 | +1.0 |
| 6 | +1 | −1 | +1 | 12.0 | +2.0 |
| 7 | +1 | +1 | −1 | 14.5 | +4.5 |
| 8 | +1 | +1 | +1 | 14.5 | +4.5 |

**Fit OLS, L2 and L1 at λ = 0.5**

1. **Intercept.** $b=\bar y=80/8=10$. It is never penalized: shrinking it would only pull every prediction toward 0 t/ha.
2. **Check orthonormality.** Each column has four $-1$s and four $+1$s, so its mean is 0 and $\tfrac18 x_j^\top x_j=1$. Any two columns agree on 4 rows and disagree on 4, so $x_j^\top x_k=0$.
3. **OLS weights** $\hat w_j=\tfrac18\sum_i x_{ij}(y_i-\bar y)$. $\hat w_1=\tfrac18(3+4+3.5+1.5+1+2+4.5+4.5)=\tfrac{24}{8}=3$. $\hat w_2=\tfrac18(3+4-3.5-1.5-1-2+4.5+4.5)=\tfrac88=1$. $\hat w_3=\tfrac18(3-4+3.5-1.5-1+2-4.5+4.5)=\tfrac28=0.25$. Training MSE $=0.5625$.
4. **L2 at $\lambda=0.5$.** Divide by $1+\lambda=1.5$: $w=(3/1.5,\;1/1.5,\;0.25/1.5)=(2,\;0.667,\;0.167)$. All three weights lose a third of their size.
5. **L1 at $\lambda=0.5$.** Subtract 0.5 from each magnitude and clip: $(3-0.5,\;1-0.5,\;\max(0.25-0.5,0))=(2.5,\;0.5,\;0)$. Variety ($x_3$) drops out of the model, and fertilizer keeps 83% of its effect.
6. **Training fit gets worse, as expected.** Training MSE rises from 0.5625 to 1.6806 (L2) and 1.1250 (L1). The full objective still improves. Under L2, $J=\tfrac12(1.6806)+0.5\cdot\tfrac12(2^2+0.667^2+0.167^2)=1.9583$, against 2.7969 for the OLS weights. Under L1, $J=\tfrac12(1.125)+0.5\cdot3=2.0625$, against 2.4063.
7. **Predict a new plot with every treatment high**, $x=(1,1,1)$: OLS $10+3+1+0.25=14.25$; L2 $10+2+0.667+0.167=12.83$; L1 $10+2.5+0.5+0=13.00$ t/ha.

Per-feature results. L1 removes the same amount λ from every weight, and the weight hits zero once λ ≥ |ŵ|. L2 removes the same fraction, so no weight ever reaches zero.

| Feature | OLS ŵ | L2, λ = 0.5 | L1, λ = 0.5 | L1 weight is 0 once | L2 weight at λ = 3 |
| --- | --- | --- | --- | --- | --- |
| x₁ fertilizer | 3.00 | 2.000 | 2.50 | λ ≥ 3 | 0.750 |
| x₂ irrigation | 1.00 | 0.667 | 0.50 | λ ≥ 1 | 0.250 |
| x₃ variety | 0.25 | 0.167 | **0** | λ ≥ 0.25 | 0.063 |

Is the shrunken model *better*? Training error alone cannot tell you, because the regularized fit is worse on the training set by construction. The real question is how it does on new data, and with 8 plots and 3 features there is little variance to remove. The lab tab in the figure has more features than the data can support, and there the answer is a clear yes.

## Watching it happen

> Figure: Two tabs, with one λ slider and an L2/L1 toggle. Worked example: the left panel maps each OLS weight ŵ to its penalized value. L2 is the straight line ŵ/(1+λ); L1 is soft-thresholding with a dead zone |ŵ| ≤ λ, shaded. The three trial weights (3, 1, 0.25) slide along these curves. At λ = 0.5 they read (2, 0.67, 0.17) for L2 and (2.5, 0.5, 0) for L1. The right panel shows the paths: L2 curves fall off hyperbolically, while L1 lines are straight and hit zero at λ = 3, 1 and 0.25. Overfitting lab: 20 rows, 10 correlated features, true weights 2, −1.5 and 1 on x1, x4 and x7. Bars are the fitted weights, dashed boxes are OLS and white ticks are the truth. Below are train and test MSE along λ. OLS has test MSE 2.807. The best L2 fit is at λ = 0.178 (test 1.944, all 10 weights non-zero); the best L1 fit is at λ = 0.133 (test 1.348, 7 non-zero). At the default λ = 0.316, L1 keeps 4 weights.

The overfitting lab, on standardized features. Test MSE is measured on 100 fresh rows from the same distribution (noise variance 1, so 1.0 is the floor). Highlighted: the best λ on the grid for each penalty.

| λ | L2 train | L2 test | L1 train | L1 test | L1 non-zero |
| --- | --- | --- | --- | --- | --- |
| 0 (OLS) | 0.223 | 2.807 | 0.223 | 2.807 | 10 |
| 0.001 | 0.223 | 2.789 | 0.223 | 2.783 | 10 |
| 0.100 | 0.354 | 2.000 | 0.425 | 1.457 | 7 |
| 0.133 | 0.419 | 1.950 | 0.566 | **1.348** | 7 |
| 0.178 | 0.509 | **1.944** | 0.686 | 1.369 | 6 |
| 0.316 | 0.782 | 2.115 | 0.972 | 1.608 | 4 |
| 1.000 | 1.759 | 3.222 | 4.117 | 5.033 | 3 |
| 3.162 | 3.062 | 4.606 | 5.246 | 6.212 | 0 |

Training error only goes **up** as $\lambda$ grows. Test error first falls, then rises, which is the U-shape of the bias–variance trade-off. Lasso wins here because the truth is sparse. Along its path the first three features to enter are exactly the true ones: $x_7$ at $\lambda\approx1.18$ ($=\lambda_{\max}$), then $x_1$ and $x_4$ at $\lambda\approx1.13$. The next one, $x_2$, does not appear until $\lambda\approx0.36$. Note also that at $\lambda=1$ lasso has the right three features but test MSE 5.03. Their weights $(0.20,-0.18,0.14)$ are shrunk far below the true $(2,-1.5,1)$, so correct selection alone does not give accurate estimates.

Where the improvement comes from. 1000 fresh training sets of 20 rows from the same distribution, predictions at the 100 test inputs. Expected test MSE = noise 1 + bias² + variance.

| λ | L2 bias² | L2 variance | L2 expected MSE | L1 bias² | L1 variance | L1 expected MSE |
| --- | --- | --- | --- | --- | --- | --- |
| 0.001 | 0.001 | 1.289 | 2.290 | 0.001 | 1.285 | 2.285 |
| 0.1 | 0.166 | 0.726 | **1.892** | 0.079 | 0.511 | **1.590** |
| 0.316 | 0.669 | 0.593 | 2.262 | 0.497 | 0.451 | 1.948 |
| 1.0 | 1.836 | 0.489 | 3.325 | 2.808 | 0.506 | 4.314 |

This is the claim in numbers. Going from $\lambda=0.001$ to $0.1$, L2 adds 0.165 of bias² and removes 0.563 of variance, for a net gain of 0.40. L1 adds 0.078 and removes 0.774. Past the optimum the bias term grows faster than the variance shrinks.

## The algorithm, step by step

**Fitting a regularized model**

1. **Split** the data into training and test sets (or set up cross-validation folds) *before* anything else.
2. **Standardize** each feature with the *training* mean and standard deviation. The penalty treats all weights alike, so they must be on the same scale. Leave the intercept out of the penalty.
3. **Build a λ grid**, log-spaced, from $\lambda_{\max}$ (L1) or a large value (L2) down to about $10^{-3}\lambda_{\max}$.
4. **Fit along the path**, from large to small $\lambda$, warm-starting each fit from the previous one. L2 uses the closed form, or gradient descent with weight decay. L1 uses coordinate descent: repeatedly set $w_j\leftarrow S_\lambda\bigl(\tfrac1n x_j^\top r_{(j)}\bigr)$, where $r_{(j)}$ is the residual leaving out feature $j$.
5. **Choose λ** with the lowest cross-validated error, or the largest $\lambda$ within one standard error of it (the "1-SE rule", which gives a simpler model).
6. **Refit** on all training data at that λ, then evaluate once on the test set.

**Complexity**

- **L2, closed form:** $O(np^2+p^3)$ once. After one SVD of $X$, $O(p^2)$ per extra $\lambda$, so the whole path is nearly free
- **L1, coordinate descent:** $O(np)$ per sweep over all coordinates, with a few to a few hundred sweeps per $\lambda$. Warm starts make the full path cost about as much as a few single fits
- **Gradient training:** L2 adds O(p) per step (weight decay); L1 needs a proximal or soft-threshold step, also O(p)
- **Prediction:** $O(p)$, or $O(\#\text{non-zero})$ after L1

## Variants and practical tuning

Penalties and how libraries spell them. Note that scikit-learn uses different scalings in different classes.

| Penalty | R(w) | Effect | Where you meet it |
| --- | --- | --- | --- |
| L2 (ridge, Tikhonov, weight decay) | ½‖w‖₂² | shrinks all weights proportionally; stabilizes correlated features | `Ridge(alpha)` minimizes ‖y−Xw‖² + α‖w‖², so α = nλ; `LogisticRegression(C)` with C = 1/(nλ); `weight_decay` in PyTorch optimizers |
| L1 (lasso) | ‖w‖₁ | constant shrinkage, exact zeros, feature selection | `Lasso(alpha)` uses the (1/2n) scaling, so α = λ; `LogisticRegression(penalty='l1', solver='liblinear' or 'saga')` |
| Elastic net | ρ‖w‖₁ + (1−ρ)½‖w‖₂² | sparse, but keeps groups of correlated features together | `ElasticNet(alpha, l1_ratio=ρ)`. See **Elastic Net regression** |
| Group lasso | Σ_g ‖w_g‖₂ | drops whole groups (e.g. all dummies of one categorical) | specialised packages (group-lasso, glmnet extensions) |
| L0 (best subset) | #{j : w_j ≠ 0} | hard selection with no shrinkage; combinatorial and non-convex | stepwise selection, mixed-integer solvers |

**Tuning checklist**

- **Scale first:** Put `StandardScaler` inside a `Pipeline` so it is refit on each CV fold
- **Search on a log grid:** λ matters on a multiplicative scale: try 10⁻³ to 10¹ (times λ_max for L1)
- **Use CV, not the test set:** `RidgeCV`, `LassoCV`, `LogisticRegressionCV`, `ElasticNetCV` compute the path and cross-validate efficiently
- **Prefer simpler on ties:** the 1-SE rule picks the largest λ whose CV error is within one standard error of the minimum
- **Debias L1 if you need sizes:** refit OLS on the features L1 selected (the relaxed lasso) if you need accurate effect sizes, not just a selected set
- **Neural nets:** L2 is `weight_decay`; with Adam use AdamW, which decouples the decay from the adaptive step

## In code

```python
import numpy as np

def standardize(X):
    mu, sd = X.mean(0), X.std(0)
    return (X - mu) / sd, mu, sd

def ridge(X, y, lam):
    """argmin (1/2n)||y - b - Xw||^2 + (lam/2)||w||^2 on standardized X."""
    Z, mu, sd = standardize(X)
    n, p = Z.shape
    w = np.linalg.solve(Z.T @ Z / n + lam * np.eye(p), Z.T @ (y - y.mean()) / n)
    return w, y.mean(), mu, sd

def soft(z, t):
    return np.sign(z) * np.maximum(np.abs(z) - t, 0.0)

def lasso(X, y, lam, tol=1e-10, max_sweeps=100_000):
    """argmin (1/2n)||y - b - Xw||^2 + lam*||w||_1 by cyclic coordinate descent."""
    Z, mu, sd = standardize(X)
    n, p = Z.shape
    w = np.zeros(p)
    r = y - y.mean()                       # residual for w = 0
    for _ in range(max_sweeps):
        biggest = 0.0
        for j in range(p):
            old = w[j]
            rho = Z[:, j] @ r / n + old    # z_j^T (partial residual) / n   (z_j^T z_j / n = 1)
            w[j] = soft(rho, lam)
            if w[j] != old:
                r -= Z[:, j] * (w[j] - old)
                biggest = max(biggest, abs(w[j] - old))
        if biggest < tol:
            break
    return w, y.mean(), mu, sd

def predict(model, X):
    w, b, mu, sd = model
    return b + ((X - mu) / sd) @ w

# Worked example: 2^3 factorial experiment, features coded -1/+1
X = np.array([[a, b, c] for a in (-1, 1) for b in (-1, 1) for c in (-1, 1)], float)
y = np.array([7, 6, 6.5, 8.5, 11, 12, 14.5, 14.5])
x_new = np.array([[1, 1, 1]])

for name, model in [("OLS  ", ridge(X, y, 0.0)), ("ridge", ridge(X, y, 0.5)), ("lasso", lasso(X, y, 0.5))]:
    mse = np.mean((y - predict(model, X)) ** 2)
    print(name, "w =", np.round(model[0], 3), " train MSE = %.4f" % mse, " predict(1,1,1) = %.2f" % predict(model, x_new)[0])
```

From scratch: L2 by its closed form, L1 by coordinate descent with soft-thresholding, on the crop trial. Prints (3, 1, 0.25) / (2, 0.667, 0.167) / (2.5, 0.5, 0), train MSE 0.5625 / 1.6806 / 1.1250 and predictions 14.25 / 12.83 / 13.00.

```python
import numpy as np
from sklearn.linear_model import Lasso, LassoCV, LogisticRegression, Ridge
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

# --- 1. The worked example. sklearn's Ridge minimizes ||y - Xw||^2 + alpha*||w||^2,
#        so our lambda = 0.5 is alpha = n * lambda = 4; Lasso already uses (1/2n) and alpha = lambda.
X = np.array([[a, b, c] for a in (-1, 1) for b in (-1, 1) for c in (-1, 1)], float)
y = np.array([7, 6, 6.5, 8.5, 11, 12, 14.5, 14.5])
n, lam = len(y), 0.5
ridge = Ridge(alpha=n * lam).fit(X, y)
lasso = Lasso(alpha=lam).fit(X, y)
print("ridge", ridge.intercept_, ridge.coef_.round(3), ridge.predict([[1, 1, 1]]).round(2))   # 10.0 [2. 0.667 0.167] [12.83]
print("lasso", lasso.intercept_, lasso.coef_.round(3), lasso.predict([[1, 1, 1]]).round(2))   # 10.0 [2.5 0.5 0. ] [13.]

# --- 2. The overfitting lab: 20 training rows, 10 correlated features, only x1, x4, x7 matter.
rs = np.random.RandomState(2)
p = 10
C = 0.5 ** np.abs(np.subtract.outer(np.arange(p), np.arange(p)))      # corr(x_i, x_j) = 0.5^|i-j|
L = np.linalg.cholesky(C)
w_true = np.zeros(p); w_true[[0, 3, 6]] = [2.0, -1.5, 1.0]
def draw(m):
    Xs = np.round(rs.randn(m, p) @ L.T, 2)
    return Xs, np.round(Xs @ w_true + rs.randn(m), 2)
Xtr, ytr = draw(20)
Xte, yte = draw(100)

mse = lambda model: np.mean((model.predict(Xte) - yte) ** 2)
grid = 10 ** (np.arange(-24, 5) / 8)                                   # lambda from 0.001 to 3.16
ridges = [make_pipeline(StandardScaler(), Ridge(alpha=20 * l)).fit(Xtr, ytr) for l in grid]
lassos = [make_pipeline(StandardScaler(), Lasso(alpha=l, tol=1e-12, max_iter=100_000)).fit(Xtr, ytr) for l in grid]
ols = make_pipeline(StandardScaler(), Ridge(alpha=0)).fit(Xtr, ytr)
print("OLS   test MSE %.3f" % mse(ols))                                    # 2.807
for name, models in [("ridge", ridges), ("lasso", lassos)]:
    k = int(np.argmin([mse(m) for m in models]))
    w = models[k][-1].coef_
    print(f"{name} best lambda {grid[k]:.3f}  test MSE {mse(models[k]):.3f}  non-zero {np.sum(w != 0)}/10  w = {w.round(2)}")
# ridge best lambda 0.178  test MSE 1.944  non-zero 10/10
# lasso best lambda 0.133  test MSE 1.348  non-zero 7/10

# --- 3. In practice: choose lambda by cross-validation, never on the test set.
cv = make_pipeline(StandardScaler(), LassoCV(alphas=grid, cv=5, max_iter=100_000)).fit(Xtr, ytr)
print("LassoCV lambda %.3f  test MSE %.3f" % (cv[-1].alpha_, mse(cv)))  # 0.024, 2.289

# --- 4. The same penalties for classification: C = 1/(n*lambda) is the INVERSE strength.
yc = (Xtr @ w_true > 0).astype(int)
for pen in ["l2", "l1"]:
    clf = make_pipeline(StandardScaler(), LogisticRegression(penalty=pen, C=0.5, solver="liblinear")).fit(Xtr, yc)
    print(pen, "logistic non-zero weights:", np.sum(clf[-1].coef_ != 0))   # l2: 10, l1: 5
```

scikit-learn: the crop trial (note α = nλ for Ridge), the overfitting lab regenerated from its seed (OLS 2.807, best ridge 1.944 at λ = 0.178, best lasso 1.348 at λ = 0.133 with 7/10 non-zero), LassoCV, and L1 vs L2 logistic regression.

> **Cross-validation on tiny data is noisy:** With only 20 rows, 5-fold `LassoCV` picks $\lambda=0.024$, which has test MSE 2.289. That beats OLS (2.807) but is far from the 1.348 you get by choosing $\lambda$ on the test set, which you are not allowed to do. Each fold trains on 16 rows, so the CV curve is jagged. Use repeated K-fold, the 1-SE rule, or more data before trusting the exact minimum.

**Strengths**

- You have many features relative to rows, or correlated features, and the unregularized fit has a large train/test gap.
- L1: you believe only a few features matter and want an automatically selected, cheaper, readable model.
- L2: many features each contribute a little, or you need a stable and unique solution (it also fixes singular XᵀX).
- You train neural nets or logistic regression: weight decay is a cheap, almost universal default.

**Limitations**

- Features are on different scales and you cannot standardize them. The penalty then depends on the units.
- You need unbiased effect sizes or valid p-values: shrinkage biases every weight toward 0 (refit or use post-selection inference).
- L1 with strongly correlated features: it picks one of the group more or less arbitrarily and the choice changes between samples. Use elastic net.
- You have plenty of data and few features: the variance is already small, so regularization buys little.

- **Genomics and high-dimensional biology:** Thousands of gene-expression features and a few hundred patients. L1 or elastic-net logistic regression picks a short list of marker genes.
- **Text classification:** L2- or L1-regularized logistic regression on bag-of-words with a vocabulary of 10⁵+. L1 gives compact models that fit on small devices.
- **Deep learning:** Weight decay (L2) is standard in training CNNs and transformers, typically via AdamW.
- **Econometrics and forecasting:** Lasso chooses a few predictors from hundreds of lagged indicators for macro nowcasting and demand forecasting.

> **Forgetting to standardize:** The penalty is on $w_j$, but its effect depends on the feature's units. Measure distance in metres instead of kilometres and the weight must be 1000 times smaller to produce the same prediction, so it is barely penalized, while a feature in large units is crushed. Always standardize (fit the scaler on the training fold only), and never penalize the intercept, or your predictions are pulled toward 0.

> **α, λ and C are not the same knob:** scikit-learn's `Ridge` uses an un-averaged loss, so its `alpha` equals nλ in this article's notation, while `Lasso`'s `alpha` equals λ. `LogisticRegression` and `SVC` use `C`, which is the **inverse** strength: larger C means *less* regularization. Copying a value from one class to another without converting it silently changes the model.

> **Where to go next:** This page covers the shared ideas. **Ridge regression** derives the closed form, its SVD view and the collinearity fix. **Lasso regression** covers coordinate descent and the diamond-corner geometry in depth. **Elastic Net regression** mixes the two penalties. **Bias–variance trade-off** explains the U-shaped test curve in general.

**Question:** Orthonormal features. A weight has OLS value ŵ = 0.8, and λ = 1. What do L2 and L1 give?

- A. L2: 0.4, L1: 0
- B. L2: 0, L1: 0.4
- C. L2: 0.8, L1: 0
- D. L2: 0.4, L1: −0.2

**Answer:** A. L2: 0.4, L1: 0 — L2: $\hat w/(1+\lambda)=0.8/2=0.4$. L1: $|\hat w|=0.8\le\lambda=1$, so soft-thresholding clips it to exactly 0 (not to $0.8-1=-0.2$).

**Question:** Why can L1 produce weights that are exactly zero while L2 cannot?

- A. L1 is applied after training and L2 during training
- B. At w = 0 the L1 penalty has a kink whose subgradient covers [−λ, λ], so any loss slope smaller than λ is balanced there; L2's slope λw vanishes at 0
- C. L2 only works for linear regression
- D. L1 uses a larger λ by convention

**Answer:** B. At w = 0 the L1 penalty has a kink whose subgradient covers [−λ, λ], so any loss slope smaller than λ is balanced there; L2's slope λw vanishes at 0 — Zero is optimal under L1 whenever $|\partial\mathcal L/\partial w_j|\le\lambda$ at $w_j=0$. Under L2 the penalty's slope at 0 is $\lambda\cdot0=0$, so any non-zero loss slope moves the weight off zero.

**Question:** In the overfitting lab, raising λ from 0.001 to 0.1 for L2 changes bias² from 0.001 to 0.166 and variance from 1.289 to 0.726. What happens to the expected test MSE (noise = 1)?

- A. Rises by 0.165
- B. Falls by about 0.40
- C. Falls by 0.563
- D. Unchanged, since bias and variance cancel

**Answer:** B. Falls by about 0.40 — $1+0.001+1.289=2.290$ becomes $1+0.166+0.726=1.892$, a drop of 0.398. The variance saved (0.563) is larger than the bias added (0.165).

**Question:** Which prior on the weights makes the MAP estimate equal to L1-regularized least squares?

- A. Gaussian N(0, τ²)
- B. Uniform on [−1, 1]
- C. Laplace (double exponential) centred at 0
- D. A flat improper prior

**Answer:** C. Laplace (double exponential) centred at 0 — $-\log p(w)\propto\sum_j|w_j|/\beta$ for a Laplace prior, which is the L1 penalty. A Gaussian prior gives $\sum_j w_j^2/(2\tau^2)$, the L2 penalty. A flat prior adds nothing, which is plain least squares.

**Key points**

- Regularization minimizes loss $+\;\lambda R(w)$. L2 uses $R=\tfrac12\lVert w\rVert_2^2$ and L1 uses $R=\lVert w\rVert_1$. The intercept is not penalized and features must be standardized.
- With orthonormal features, L2 divides each weight by $1+\lambda$ (proportional shrinkage) and L1 soft-thresholds it, $\operatorname{sign}(\hat w)\max(|\hat w|-\lambda,0)$, so every weight with $|\hat w|\le\lambda$ becomes exactly 0.
- Training error always rises with λ and test error is U-shaped: bias² grows while variance falls. In the lab, OLS test MSE 2.807 fell to 1.944 (L2) and 1.348 (L1, 7 of 10 weights kept).
- L2 is weight decay, a Gaussian prior and a sphere constraint. L1 is a proximal soft-threshold step, a Laplace prior and a diamond constraint whose corners produce zeros.
- Choose λ on a log grid by cross-validation inside a pipeline. Watch library conventions: Ridge α = nλ, Lasso α = λ, and C = inverse strength.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
