L1 & L2 regularization

Penalizing weight size trades a little bias for less variance; L1 also pushes weights to exactly zero.

17 min read1868 wordscore11 viewsupdated 2026-10-07

Definition

Regularization adds a penalty on the size of a model's weights to its training loss: minimize loss(w)+λ R(w)\text{loss}(w)+\lambda\,R(w). The L2 penalty R(w)=12∥w∥22=12∑jwj2R(w)=\tfrac12\lVert w\rVert_2^2=\tfrac12\sum_j w_j^2 shrinks every weight smoothly toward zero. The L1 penalty R(w)=∥w∥1=∑j∣wj∣R(w)=\lVert w\rVert_1=\sum_j |w_j| shrinks them too, and sets the least useful ones to exactly zero. One knob, λ≥0\lambda\ge 0, sets how much training fit you give up in exchange for smaller weights.

At a glance

Task
Not a model of its own: a modification of the training objective of any weight-based model (linear and logistic regression, SVMs, neural networks)
Input → output
a loss L(w)\mathcal L(w), a penalty type and a strength λ\lambda → a weight vector with smaller norm (and, for L1, some zeros)
Learns
the same parameters as the base model, just pulled toward 0
Training cost
L2: almost free (adds λw\lambda w to the gradient, or λI\lambda I to the normal equations). L1: the objective has a kink at 0, so it needs coordinate descent or proximal steps instead of plain gradient descent
Prediction cost
unchanged; with L1, features whose weight is 0 need not be computed at all
Hyperparameters
strength λ (sklearn: alpha, or C = inverse strength); penalty type l1 / l2 / elastic-net mix
Assumes
features on comparable scales (standardize first); the intercept is not penalized; the true signal is spread out (L2) or carried by a few features (L1)

The problem

Least squares and maximum likelihood pick the weights that fit the training set best. When there are many features compared with rows, or the features are correlated, there are many weight vectors that fit almost equally well, and the fitting procedure picks one that bends to the noise in this particular sample. Its weights are large and unstable: draw another training set from the same source and they change a lot. That instability is variance, and it shows up as a big gap between training and test error.

The overfitting lab below is a concrete case: 20 training rows, 10 correlated features, and only 3 of them actually matter. Ordinary least squares reaches a training MSE of 0.223 but a test MSE of 2.807, more than twelve times worse. The extra features let it fit noise. We could delete features by hand, but which ones? Regularization makes the model pay for every unit of weight, so a weight has to earn its size by reducing the loss.

The idea

Picture every weight attached to zero by a spring. The data pulls the weights toward their least-squares values and the springs pull back. With L2 the pull back is proportional to the weight, like an ordinary spring. Big weights feel a big pull and small ones a gentle one, so every weight shrinks by the same fraction and none reaches zero. With L1 the pull is a constant λ\lambda whatever the weight's size, like friction. A weight whose data-pull is weaker than λ\lambda is dragged all the way to zero and stays there. That is why L1 does feature selection and L2 only shrinks.

Either way the model becomes a little biased, because the weights are systematically too small. In return the weights move much less from one training sample to the next, so the variance drops. When the variance saved is larger than the squared bias added, test error goes down. Most of the work is finding the λ\lambda where that happens.

Vocabulary

Penalty / regularizer R(w)
the term added to the loss; 12∥w∥22\tfrac12\lVert w\rVert_2^2 (L2, ridge, weight decay) or ∥w∥1\lVert w\rVert_1 (L1, lasso)
λ (regularization strength)
the exchange rate between fit and weight size. λ=0\lambda=0 is the unregularized model; λ→∞\lambda\to\infty forces w=0w=0
Shrinkage
moving estimates toward 0 (or another fixed value). It always adds bias and reduces variance
Sparsity
having many weights exactly 0. L1 produces it, L2 does not
Soft-thresholding
Sλ(z)=sign⁡(z)max⁡(∣z∣−λ,0)S_\lambda(z)=\operatorname{sign}(z)\max(|z|-\lambda,0): subtract λ\lambda from the magnitude and clip at 0. This is what L1 does to each weight
Subgradient
the set of slopes of a kinked function at the kink. For ∣w∣|w| at w=0w=0 it is the whole interval [−1,1][-1,1]
Weight decay
the L2 penalty seen from gradient descent: each step multiplies ww by (1−ηλ)(1-\eta\lambda) before the loss gradient is applied
Regularization path
the fitted weights as a function of λ\lambda, from all-zero (large λ\lambda) to the unregularized fit (λ→0\lambda\to0)
λ_max
for L1, the smallest λ\lambda at which every weight is 0. The path starts there
Effective degrees of freedom
how many parameters the fit really uses. For L2 it is ∑ksk/(sk+λ)\sum_k s_k/(s_k+\lambda), between 0 and pp; for L1 it is roughly the number of non-zero weights

The math

Write the model as y^=b+x⊤w\hat y=b+x^\top w with pp standardized features (mean 0, variance 1). The penalized objective is the average loss plus λ\lambda times the penalty. For squared error:

J(w)=12n∑i=1n(yi−b−xi⊤w)2+λ R(w),R2(w)=12∥w∥22,R1(w)=∥w∥1J(w)=\frac{1}{2n}\sum_{i=1}^{n}\bigl(y_i-b-x_i^\top w\bigr)^2+\lambda\,R(w),\qquad R_2(w)=\tfrac12\lVert w\rVert_2^2,\quad R_1(w)=\lVert w\rVert_1
n,  pn,\;p
number of training rows and of features
xi,  yix_i,\;y_i
feature vector and target of row ii
w,  bw,\;b
weights and intercept. bb is not penalized; with centred features its optimum is yˉ\bar y
λ≥0\lambda\ge0
regularization strength
∥w∥22=∑jwj2\lVert w\rVert_2^2=\sum_j w_j^2
squared Euclidean length (L2)
∥w∥1=∑j∣wj∣\lVert w\rVert_1=\sum_j|w_j|
sum of absolute values (L1)

The same recipe works for any loss L\mathcal L: log-loss for logistic regression, hinge loss for SVMs, cross-entropy for a neural network. Only the first term changes.

Deriving the two shrinkage rules

The difference between L1 and L2 is clearest when the features are orthonormal: centred, 1nxj⊤xj=1\tfrac1n x_j^\top x_j=1 and xj⊤xk=0x_j^\top x_k=0 for j≠kj\ne k. Designed experiments have this property, and so do principal-component scores after scaling. The unregularized solution is then simply w^j=1nxj⊤(y−yˉ)\hat w_j=\tfrac1n x_j^\top(y-\bar y), and expanding the square gives

J(w)=const+∑j=1p[12 (wj−w^j)2+λ r(wj)]J(w)=\text{const}+\sum_{j=1}^{p}\Bigl[\tfrac12\,(w_j-\hat w_j)^2+\lambda\,r(w_j)\Bigr]
w^j\hat w_j
the ordinary least-squares weight of feature j
r(wj)r(w_j)
12wj2\tfrac12 w_j^2 for L2, ∣wj∣|w_j| for L1

The objective splits into pp separate one-dimensional problems, so we can solve each weight on its own.

Solving one coordinate

  1. 1L2. 12(w−w^)2+λ2w2\tfrac12(w-\hat w)^2+\tfrac\lambda2 w^2 is a smooth parabola. Set the derivative to zero: w−w^+λw=0w-\hat w+\lambda w=0, so w⋆=w^1+λw^\star=\dfrac{\hat w}{1+\lambda}. Every weight is divided by the same factor, and it reaches 0 only when λ→∞\lambda\to\infty.
  2. 2L1, case w>0w>0. The derivative is w−w^+λ=0w-\hat w+\lambda=0, so w=w^−λw=\hat w-\lambda. This is consistent only if w^>λ\hat w>\lambda.
  3. 3L1, case w<0w<0. By symmetry w=w^+λw=\hat w+\lambda, consistent only if w^<−λ\hat w<-\lambda.
  4. 4L1, case w=0w=0. ∣w∣|w| has no derivative at 0; its subgradient there is the interval [−1,1][-1,1]. Zero is optimal when 0∈−w^+λ[−1,1]0\in -\hat w+\lambda[-1,1], which means ∣w^∣≤λ|\hat w|\le\lambda. Every OLS weight in the band [−λ,λ][-\lambda,\lambda] is set to exactly 0.
  5. 5Together: w⋆=Sλ(w^)=sign⁡(w^)max⁡(∣w^∣−λ,0)w^\star=S_\lambda(\hat w)=\operatorname{sign}(\hat w)\max(|\hat w|-\lambda,0), the soft-thresholding operator.
wL2⋆=w^1+λwL1⋆=sign⁡(w^) max⁡(∣w^∣−λ,  0)w^\star_{\text{L2}}=\frac{\hat w}{1+\lambda}\qquad\qquad w^\star_{\text{L1}}=\operatorname{sign}(\hat w)\,\max\bigl(|\hat w|-\lambda,\;0\bigr)
w⋆w^\star
the regularized weight (orthonormal features)
1/(1+λ)1/(1+\lambda)
L2: proportional shrinkage. The same fraction is removed from every weight
∣w^∣−λ|\hat w|-\lambda
L1: constant shrinkage, clipped at 0. The same amount is removed from every weight

For general correlated features there is no per-weight formula. L2 still has a closed form, w=(1nX⊤X+λI)−11nX⊤(y−yˉ)w=(\tfrac1n X^\top X+\lambda I)^{-1}\tfrac1n X^\top(y-\bar y), and the added λI\lambda I makes the matrix invertible even when p>np>n (see Ridge regression). L1 is solved by applying soft-thresholding to one coordinate at a time, which is coordinate descent (see Lasso regression).

Why the kink makes zeros: the gradient view

For gradient-trained models such as networks and logistic regression, it helps to look at what each penalty does to a single update with step size η\eta:

L2: w←(1−ηλ) w−η∇L(w)L1: w←Sηλ(w−η∇L(w))\text{L2: } w\leftarrow(1-\eta\lambda)\,w-\eta\nabla\mathcal L(w)\qquad\qquad\text{L1: } w\leftarrow S_{\eta\lambda}\bigl(w-\eta\nabla\mathcal L(w)\bigr)
η\eta
learning rate
(1−ηλ)(1-\eta\lambda)
L2 multiplies each weight by a factor just below 1 on every step, hence the name weight decay
SηλS_{\eta\lambda}
L1's proximal step: take the ordinary gradient step, then soft-threshold by ηλ\eta\lambda (the ISTA algorithm)

The L2 pull, λw\lambda w, fades as the weight gets small, so it never quite reaches zero. The L1 pull, λsign⁡(w)\lambda\operatorname{sign}(w), stays the same size all the way down. Once a weight is at 0 it stays there while ∣∂L/∂wj∣≤λ|\partial\mathcal L/\partial w_j|\le\lambda. Gradient descent on λ∣w∣\lambda|w| without the proximal step is not enough: the weights oscillate around zero and never land on it. Use a solver that handles the kink.

What λ does

Reading the knob

λ = 0
no penalty: the ordinary least-squares or maximum-likelihood fit, with the most variance
Small λ
removes the wild, noise-driven weight directions first. L2 shrinks most along directions where the data has little spread (small eigenvalues of XᵀX/n)
Moderate λ
usually the sweet spot: test error is lowest, and L1 has dropped the irrelevant features
λ ≥ λ_max (L1)
every weight is 0 and the model predicts yˉ\bar y. λmax⁡=max⁡j∣1nxj⊤(y−yˉ)∣\lambda_{\max}=\max_j\bigl|\tfrac1n x_j^\top(y-\bar y)\bigr|
λ → ∞ (L2)
weights approach 0 but are never exactly 0
Effective size
L2 with eigenvalues sks_k of 1nX⊤X\tfrac1n X^\top X keeps df(λ)=∑ksksk+λ\mathrm{df}(\lambda)=\sum_k \tfrac{s_k}{s_k+\lambda} effective parameters. In the lab below this falls from 10 to 8.44 at λ=0.1\lambda=0.1 and 4.22 at λ=1\lambda=1

Worked example: a crop trial, by hand

An agronomist runs every combination of three two-level treatments: fertilizer x1x_1, irrigation x2x_2 and seed variety x3x_3, each coded −1-1 (low/standard) or +1+1 (high/new). That makes 23=82^3=8 plots, and yy is the yield in tonnes per hectare. Coding a full factorial design this way makes the columns orthonormal, so the formulas above apply exactly and every step can be done by hand.

Plotx₁ fertilizerx₂ irrigationx₃ varietyy (t/ha)y − ȳ
1−1−1−17.0−3.0
2−1−1+16.0−4.0
3−1+1−16.5−3.5
4−1+1+18.5−1.5
5+1−1−111.0+1.0
6+1−1+112.0+2.0
7+1+1−114.5+4.5
8+1+1+114.5+4.5
Eight plots of a 2³ factorial trial. Mean yield ȳ = 10. Highlighted: the plot with all three treatments high, the input we predict at the end.

Fit OLS, L2 and L1 at λ = 0.5

  1. 1Intercept. b=yˉ=80/8=10b=\bar y=80/8=10. It is never penalized: shrinking it would only pull every prediction toward 0 t/ha.
  2. 2Check orthonormality. Each column has four −1-1s and four +1+1s, so its mean is 0 and 18xj⊤xj=1\tfrac18 x_j^\top x_j=1. Any two columns agree on 4 rows and disagree on 4, so xj⊤xk=0x_j^\top x_k=0.
  3. 3OLS weights w^j=18∑ixij(yi−yˉ)\hat w_j=\tfrac18\sum_i x_{ij}(y_i-\bar y). w^1=18(3+4+3.5+1.5+1+2+4.5+4.5)=248=3\hat w_1=\tfrac18(3+4+3.5+1.5+1+2+4.5+4.5)=\tfrac{24}{8}=3. w^2=18(3+4−3.5−1.5−1−2+4.5+4.5)=88=1\hat w_2=\tfrac18(3+4-3.5-1.5-1-2+4.5+4.5)=\tfrac88=1. w^3=18(3−4+3.5−1.5−1+2−4.5+4.5)=28=0.25\hat w_3=\tfrac18(3-4+3.5-1.5-1+2-4.5+4.5)=\tfrac28=0.25. Training MSE =0.5625=0.5625.
  4. 4L2 at λ=0.5\lambda=0.5. Divide by 1+λ=1.51+\lambda=1.5: w=(3/1.5,  1/1.5,  0.25/1.5)=(2,  0.667,  0.167)w=(3/1.5,\;1/1.5,\;0.25/1.5)=(2,\;0.667,\;0.167). All three weights lose a third of their size.
  5. 5L1 at λ=0.5\lambda=0.5. Subtract 0.5 from each magnitude and clip: (3−0.5,  1−0.5,  max⁡(0.25−0.5,0))=(2.5,  0.5,  0)(3-0.5,\;1-0.5,\;\max(0.25-0.5,0))=(2.5,\;0.5,\;0). Variety (x3x_3) drops out of the model, and fertilizer keeps 83% of its effect.
  6. 6Training fit gets worse, as expected. Training MSE rises from 0.5625 to 1.6806 (L2) and 1.1250 (L1). The full objective still improves. Under L2, J=12(1.6806)+0.5⋅12(22+0.6672+0.1672)=1.9583J=\tfrac12(1.6806)+0.5\cdot\tfrac12(2^2+0.667^2+0.167^2)=1.9583, against 2.7969 for the OLS weights. Under L1, J=12(1.125)+0.5⋅3=2.0625J=\tfrac12(1.125)+0.5\cdot3=2.0625, against 2.4063.
  7. 7Predict a new plot with every treatment high, x=(1,1,1)x=(1,1,1): OLS 10+3+1+0.25=14.2510+3+1+0.25=14.25; L2 10+2+0.667+0.167=12.8310+2+0.667+0.167=12.83; L1 10+2.5+0.5+0=13.0010+2.5+0.5+0=13.00 t/ha.
FeatureOLS ŵL2, λ = 0.5L1, λ = 0.5L1 weight is 0 onceL2 weight at λ = 3
x₁ fertilizer3.002.0002.50λ ≥ 30.750
x₂ irrigation1.000.6670.50λ ≥ 10.250
x₃ variety0.250.1670λ ≥ 0.250.063
Per-feature results. L1 removes the same amount λ from every weight, and the weight hits zero once λ ≥ |ŵ|. L2 removes the same fraction, so no weight ever reaches zero.

Is the shrunken model better? Training error alone cannot tell you, because the regularized fit is worse on the training set by construction. The real question is how it does on new data, and with 8 plots and 3 features there is little variance to remove. The lab tab in the figure has more features than the data can support, and there the answer is a clear yes.

Watching it happen

λ0.50
OLS weight ŵ → penalized weight-3-3-2-2-1-1112233dashed: no penalty (w = ŵ)w = sign(ŵ)·max(|ŵ|−λ, 0)weights as λ grows · drag the line00112233λx1x2x3● x1 ŵ = 3 · ● x2 ŵ = 1 · ● x3 ŵ = 0.25 · faint = the other penalty
L1 · lasso at λ = 0.50
w = (2.50, 0.50, 0.00)
train MSE 1.1250 (OLS 0.5625)
zeros: 1 of 3
predict (1, 1, 1): 13.00 (OLS 14.25)
L2 · ridge: (2.00, 0.67, 0.17)
fig.Two tabs, with one λ slider and an L2/L1 toggle. Worked example: the left panel maps each OLS weight ŵ to its penalized value. L2 is the straight line ŵ/(1+λ); L1 is soft-thresholding with a dead zone |ŵ| ≤ λ, shaded. The three trial weights (3, 1, 0.25) slide along these curves. At λ = 0.5 they read (2, 0.67, 0.17) for L2 and (2.5, 0.5, 0) for L1. The right panel shows the paths: L2 curves fall off hyperbolically, while L1 lines are straight and hit zero at λ = 3, 1 and 0.25. Overfitting lab: 20 rows, 10 correlated features, true weights 2, −1.5 and 1 on x1, x4 and x7. Bars are the fitted weights, dashed boxes are OLS and white ticks are the truth. Below are train and test MSE along λ. OLS has test MSE 2.807. The best L2 fit is at λ = 0.178 (test 1.944, all 10 weights non-zero); the best L1 fit is at λ = 0.133 (test 1.348, 7 non-zero). At the default λ = 0.316, L1 keeps 4 weights.
λL2 trainL2 testL1 trainL1 testL1 non-zero
0 (OLS)0.2232.8070.2232.80710
0.0010.2232.7890.2232.78310
0.1000.3542.0000.4251.4577
0.1330.4191.9500.5661.3487
0.1780.5091.9440.6861.3696
0.3160.7822.1150.9721.6084
1.0001.7593.2224.1175.0333
3.1623.0624.6065.2466.2120
The overfitting lab, on standardized features. Test MSE is measured on 100 fresh rows from the same distribution (noise variance 1, so 1.0 is the floor). Highlighted: the best λ on the grid for each penalty.

Training error only goes up as λ\lambda grows. Test error first falls, then rises, which is the U-shape of the bias–variance trade-off. Lasso wins here because the truth is sparse. Along its path the first three features to enter are exactly the true ones: x7x_7 at λ≈1.18\lambda\approx1.18 (=λmax⁡=\lambda_{\max}), then x1x_1 and x4x_4 at λ≈1.13\lambda\approx1.13. The next one, x2x_2, does not appear until λ≈0.36\lambda\approx0.36. Note also that at λ=1\lambda=1 lasso has the right three features but test MSE 5.03. Their weights (0.20,−0.18,0.14)(0.20,-0.18,0.14) are shrunk far below the true (2,−1.5,1)(2,-1.5,1), so correct selection alone does not give accurate estimates.

λL2 bias²L2 varianceL2 expected MSEL1 bias²L1 varianceL1 expected MSE
0.0010.0011.2892.2900.0011.2852.285
0.10.1660.7261.8920.0790.5111.590
0.3160.6690.5932.2620.4970.4511.948
1.01.8360.4893.3252.8080.5064.314
Where the improvement comes from. 1000 fresh training sets of 20 rows from the same distribution, predictions at the 100 test inputs. Expected test MSE = noise 1 + bias² + variance.

This is the claim in numbers. Going from λ=0.001\lambda=0.001 to 0.10.1, L2 adds 0.165 of bias² and removes 0.563 of variance, for a net gain of 0.40. L1 adds 0.078 and removes 0.774. Past the optimum the bias term grows faster than the variance shrinks.

The algorithm, step by step

Fitting a regularized model

  1. 1Split the data into training and test sets (or set up cross-validation folds) before anything else.
  2. 2Standardize each feature with the training mean and standard deviation. The penalty treats all weights alike, so they must be on the same scale. Leave the intercept out of the penalty.
  3. 3Build a λ grid, log-spaced, from λmax⁡\lambda_{\max} (L1) or a large value (L2) down to about 10−3λmax⁡10^{-3}\lambda_{\max}.
  4. 4Fit along the path, from large to small λ\lambda, warm-starting each fit from the previous one. L2 uses the closed form, or gradient descent with weight decay. L1 uses coordinate descent: repeatedly set wj←Sλ(1nxj⊤r(j))w_j\leftarrow S_\lambda\bigl(\tfrac1n x_j^\top r_{(j)}\bigr), where r(j)r_{(j)} is the residual leaving out feature jj.
  5. 5Choose λ with the lowest cross-validated error, or the largest λ\lambda within one standard error of it (the "1-SE rule", which gives a simpler model).
  6. 6Refit on all training data at that λ, then evaluate once on the test set.

Complexity

L2, closed form
O(np2+p3)O(np^2+p^3) once. After one SVD of XX, O(p2)O(p^2) per extra λ\lambda, so the whole path is nearly free
L1, coordinate descent
O(np)O(np) per sweep over all coordinates, with a few to a few hundred sweeps per λ\lambda. Warm starts make the full path cost about as much as a few single fits
Gradient training
L2 adds O(p) per step (weight decay); L1 needs a proximal or soft-threshold step, also O(p)
Prediction
O(p)O(p), or O(#non-zero)O(\#\text{non-zero}) after L1

Variants and practical tuning

PenaltyR(w)EffectWhere you meet it
L2 (ridge, Tikhonov, weight decay)½‖w‖₂²shrinks all weights proportionally; stabilizes correlated featuresRidge(alpha) minimizes ‖y−Xw‖² + α‖w‖², so α = nλ; LogisticRegression(C) with C = 1/(nλ); weight_decay in PyTorch optimizers
L1 (lasso)‖w‖₁constant shrinkage, exact zeros, feature selectionLasso(alpha) uses the (1/2n) scaling, so α = λ; LogisticRegression(penalty='l1', solver='liblinear' or 'saga')
Elastic netρ‖w‖₁ + (1−ρ)½‖w‖₂²sparse, but keeps groups of correlated features togetherElasticNet(alpha, l1_ratio=ρ). See Elastic Net regression
Group lassoΣ_g ‖w_g‖₂drops whole groups (e.g. all dummies of one categorical)specialised packages (group-lasso, glmnet extensions)
L0 (best subset)#{j : w_j ≠ 0}hard selection with no shrinkage; combinatorial and non-convexstepwise selection, mixed-integer solvers
Penalties and how libraries spell them. Note that scikit-learn uses different scalings in different classes.

Tuning checklist

Scale first
Put StandardScaler inside a Pipeline so it is refit on each CV fold
Search on a log grid
λ matters on a multiplicative scale: try 10⁻³ to 10¹ (times λ_max for L1)
Use CV, not the test set
RidgeCV, LassoCV, LogisticRegressionCV, ElasticNetCV compute the path and cross-validate efficiently
Prefer simpler on ties
the 1-SE rule picks the largest λ whose CV error is within one standard error of the minimum
Debias L1 if you need sizes
refit OLS on the features L1 selected (the relaxed lasso) if you need accurate effect sizes, not just a selected set
Neural nets
L2 is weight_decay; with Adam use AdamW, which decouples the decay from the adaptive step

In code

python
1import numpy as np
2
3def standardize(X):
4 mu, sd = X.mean(0), X.std(0)
5 return (X - mu) / sd, mu, sd
6
7def ridge(X, y, lam):
8 """argmin (1/2n)||y - b - Xw||^2 + (lam/2)||w||^2 on standardized X."""
9 Z, mu, sd = standardize(X)
10 n, p = Z.shape
11 w = np.linalg.solve(Z.T @ Z / n + lam * np.eye(p), Z.T @ (y - y.mean()) / n)
12 return w, y.mean(), mu, sd
13
14def soft(z, t):
15 return np.sign(z) * np.maximum(np.abs(z) - t, 0.0)
16
17def lasso(X, y, lam, tol=1e-10, max_sweeps=100_000):
18 """argmin (1/2n)||y - b - Xw||^2 + lam*||w||_1 by cyclic coordinate descent."""
19 Z, mu, sd = standardize(X)
20 n, p = Z.shape
21 w = np.zeros(p)
22 r = y - y.mean() # residual for w = 0
23 for _ in range(max_sweeps):
24 biggest = 0.0
25 for j in range(p):
26 old = w[j]
27 rho = Z[:, j] @ r / n + old # z_j^T (partial residual) / n (z_j^T z_j / n = 1)
28 w[j] = soft(rho, lam)
29 if w[j] != old:
30 r -= Z[:, j] * (w[j] - old)
31 biggest = max(biggest, abs(w[j] - old))
32 if biggest < tol:
33 break
34 return w, y.mean(), mu, sd
35
36def predict(model, X):
37 w, b, mu, sd = model
38 return b + ((X - mu) / sd) @ w
39
40# Worked example: 2^3 factorial experiment, features coded -1/+1
41X = np.array([[a, b, c] for a in (-1, 1) for b in (-1, 1) for c in (-1, 1)], float)
42y = np.array([7, 6, 6.5, 8.5, 11, 12, 14.5, 14.5])
43x_new = np.array([[1, 1, 1]])
44
45for name, model in [("OLS ", ridge(X, y, 0.0)), ("ridge", ridge(X, y, 0.5)), ("lasso", lasso(X, y, 0.5))]:
46 mse = np.mean((y - predict(model, X)) ** 2)
47 print(name, "w =", np.round(model[0], 3), " train MSE = %.4f" % mse, " predict(1,1,1) = %.2f" % predict(model, x_new)[0])
From scratch: L2 by its closed form, L1 by coordinate descent with soft-thresholding, on the crop trial. Prints (3, 1, 0.25) / (2, 0.667, 0.167) / (2.5, 0.5, 0), train MSE 0.5625 / 1.6806 / 1.1250 and predictions 14.25 / 12.83 / 13.00.
python
1import numpy as np
2from sklearn.linear_model import Lasso, LassoCV, LogisticRegression, Ridge
3from sklearn.pipeline import make_pipeline
4from sklearn.preprocessing import StandardScaler
5
6# --- 1. The worked example. sklearn's Ridge minimizes ||y - Xw||^2 + alpha*||w||^2,
7# so our lambda = 0.5 is alpha = n * lambda = 4; Lasso already uses (1/2n) and alpha = lambda.
8X = np.array([[a, b, c] for a in (-1, 1) for b in (-1, 1) for c in (-1, 1)], float)
9y = np.array([7, 6, 6.5, 8.5, 11, 12, 14.5, 14.5])
10n, lam = len(y), 0.5
11ridge = Ridge(alpha=n * lam).fit(X, y)
12lasso = Lasso(alpha=lam).fit(X, y)
13print("ridge", ridge.intercept_, ridge.coef_.round(3), ridge.predict([[1, 1, 1]]).round(2)) # 10.0 [2. 0.667 0.167] [12.83]
14print("lasso", lasso.intercept_, lasso.coef_.round(3), lasso.predict([[1, 1, 1]]).round(2)) # 10.0 [2.5 0.5 0. ] [13.]
15
16# --- 2. The overfitting lab: 20 training rows, 10 correlated features, only x1, x4, x7 matter.
17rs = np.random.RandomState(2)
18p = 10
19C = 0.5 ** np.abs(np.subtract.outer(np.arange(p), np.arange(p))) # corr(x_i, x_j) = 0.5^|i-j|
20L = np.linalg.cholesky(C)
21w_true = np.zeros(p); w_true[[0, 3, 6]] = [2.0, -1.5, 1.0]
22def draw(m):
23 Xs = np.round(rs.randn(m, p) @ L.T, 2)
24 return Xs, np.round(Xs @ w_true + rs.randn(m), 2)
25Xtr, ytr = draw(20)
26Xte, yte = draw(100)
27
28mse = lambda model: np.mean((model.predict(Xte) - yte) ** 2)
29grid = 10 ** (np.arange(-24, 5) / 8) # lambda from 0.001 to 3.16
30ridges = [make_pipeline(StandardScaler(), Ridge(alpha=20 * l)).fit(Xtr, ytr) for l in grid]
31lassos = [make_pipeline(StandardScaler(), Lasso(alpha=l, tol=1e-12, max_iter=100_000)).fit(Xtr, ytr) for l in grid]
32ols = make_pipeline(StandardScaler(), Ridge(alpha=0)).fit(Xtr, ytr)
33print("OLS test MSE %.3f" % mse(ols)) # 2.807
34for name, models in [("ridge", ridges), ("lasso", lassos)]:
35 k = int(np.argmin([mse(m) for m in models]))
36 w = models[k][-1].coef_
37 print(f"{name} best lambda {grid[k]:.3f} test MSE {mse(models[k]):.3f} non-zero {np.sum(w != 0)}/10 w = {w.round(2)}")
38# ridge best lambda 0.178 test MSE 1.944 non-zero 10/10
39# lasso best lambda 0.133 test MSE 1.348 non-zero 7/10
40
41# --- 3. In practice: choose lambda by cross-validation, never on the test set.
42cv = make_pipeline(StandardScaler(), LassoCV(alphas=grid, cv=5, max_iter=100_000)).fit(Xtr, ytr)
43print("LassoCV lambda %.3f test MSE %.3f" % (cv[-1].alpha_, mse(cv))) # 0.024, 2.289
44
45# --- 4. The same penalties for classification: C = 1/(n*lambda) is the INVERSE strength.
46yc = (Xtr @ w_true > 0).astype(int)
47for pen in ["l2", "l1"]:
48 clf = make_pipeline(StandardScaler(), LogisticRegression(penalty=pen, C=0.5, solver="liblinear")).fit(Xtr, yc)
49 print(pen, "logistic non-zero weights:", np.sum(clf[-1].coef_ != 0)) # l2: 10, l1: 5
scikit-learn: the crop trial (note α = nλ for Ridge), the overfitting lab regenerated from its seed (OLS 2.807, best ridge 1.944 at λ = 0.178, best lasso 1.348 at λ = 0.133 with 7/10 non-zero), LassoCV, and L1 vs L2 logistic regression.

Use when

  • +You have many features relative to rows, or correlated features, and the unregularized fit has a large train/test gap.
  • +L1: you believe only a few features matter and want an automatically selected, cheaper, readable model.
  • +L2: many features each contribute a little, or you need a stable and unique solution (it also fixes singular XᵀX).
  • +You train neural nets or logistic regression: weight decay is a cheap, almost universal default.

Avoid when

  • −Features are on different scales and you cannot standardize them. The penalty then depends on the units.
  • −You need unbiased effect sizes or valid p-values: shrinkage biases every weight toward 0 (refit or use post-selection inference).
  • −L1 with strongly correlated features: it picks one of the group more or less arbitrarily and the choice changes between samples. Use elastic net.
  • −You have plenty of data and few features: the variance is already small, so regularization buys little.

Where it's used

Genomics and high-dimensional biology

Thousands of gene-expression features and a few hundred patients. L1 or elastic-net logistic regression picks a short list of marker genes.

Text classification

L2- or L1-regularized logistic regression on bag-of-words with a vocabulary of 10⁵+. L1 gives compact models that fit on small devices.

Deep learning

Weight decay (L2) is standard in training CNNs and transformers, typically via AdamW.

Econometrics and forecasting

Lasso chooses a few predictors from hundreds of lagged indicators for macro nowcasting and demand forecasting.

Check yourself

Orthonormal features. A weight has OLS value ŵ = 0.8, and λ = 1. What do L2 and L1 give?

Check yourself

Why can L1 produce weights that are exactly zero while L2 cannot?

Check yourself

In the overfitting lab, raising λ from 0.001 to 0.1 for L2 changes bias² from 0.001 to 0.166 and variance from 1.289 to 0.726. What happens to the expected test MSE (noise = 1)?

Check yourself

Which prior on the weights makes the MAP estimate equal to L1-regularized least squares?

Remember

  • ✓Regularization minimizes loss +  λR(w)+\;\lambda R(w). L2 uses R=12∥w∥22R=\tfrac12\lVert w\rVert_2^2 and L1 uses R=∥w∥1R=\lVert w\rVert_1. The intercept is not penalized and features must be standardized.
  • ✓With orthonormal features, L2 divides each weight by 1+λ1+\lambda (proportional shrinkage) and L1 soft-thresholds it, sign⁡(w^)max⁡(∣w^∣−λ,0)\operatorname{sign}(\hat w)\max(|\hat w|-\lambda,0), so every weight with ∣w^∣≤λ|\hat w|\le\lambda becomes exactly 0.
  • ✓Training error always rises with λ and test error is U-shaped: bias² grows while variance falls. In the lab, OLS test MSE 2.807 fell to 1.944 (L2) and 1.348 (L1, 7 of 10 weights kept).
  • ✓L2 is weight decay, a Gaussian prior and a sphere constraint. L1 is a proximal soft-threshold step, a Laplace prior and a diamond constraint whose corners produce zeros.
  • ✓Choose λ on a log grid by cross-validation inside a pipeline. Watch library conventions: Ridge α = nλ, Lasso α = λ, and C = inverse strength.
← back to ML Algorithms roadmap