Definition
Regularization adds a penalty on the size of a model's weights to its training loss: minimize . The L2 penalty shrinks every weight smoothly toward zero. The L1 penalty shrinks them too, and sets the least useful ones to exactly zero. One knob, , sets how much training fit you give up in exchange for smaller weights.
At a glance
- Task
- Not a model of its own: a modification of the training objective of any weight-based model (linear and logistic regression, SVMs, neural networks)
- Input → output
- a loss , a penalty type and a strength → a weight vector with smaller norm (and, for L1, some zeros)
- Learns
- the same parameters as the base model, just pulled toward 0
- Training cost
- L2: almost free (adds to the gradient, or to the normal equations). L1: the objective has a kink at 0, so it needs coordinate descent or proximal steps instead of plain gradient descent
- Prediction cost
- unchanged; with L1, features whose weight is 0 need not be computed at all
- Hyperparameters
- strength λ (sklearn:
alpha, orC= inverse strength); penalty typel1/l2/ elastic-net mix - Assumes
- features on comparable scales (standardize first); the intercept is not penalized; the true signal is spread out (L2) or carried by a few features (L1)
The problem
Least squares and maximum likelihood pick the weights that fit the training set best. When there are many features compared with rows, or the features are correlated, there are many weight vectors that fit almost equally well, and the fitting procedure picks one that bends to the noise in this particular sample. Its weights are large and unstable: draw another training set from the same source and they change a lot. That instability is variance, and it shows up as a big gap between training and test error.
The overfitting lab below is a concrete case: 20 training rows, 10 correlated features, and only 3 of them actually matter. Ordinary least squares reaches a training MSE of 0.223 but a test MSE of 2.807, more than twelve times worse. The extra features let it fit noise. We could delete features by hand, but which ones? Regularization makes the model pay for every unit of weight, so a weight has to earn its size by reducing the loss.
The idea
Picture every weight attached to zero by a spring. The data pulls the weights toward their least-squares values and the springs pull back. With L2 the pull back is proportional to the weight, like an ordinary spring. Big weights feel a big pull and small ones a gentle one, so every weight shrinks by the same fraction and none reaches zero. With L1 the pull is a constant whatever the weight's size, like friction. A weight whose data-pull is weaker than is dragged all the way to zero and stays there. That is why L1 does feature selection and L2 only shrinks.
Either way the model becomes a little biased, because the weights are systematically too small. In return the weights move much less from one training sample to the next, so the variance drops. When the variance saved is larger than the squared bias added, test error goes down. Most of the work is finding the where that happens.
Vocabulary
- Penalty / regularizer R(w)
- the term added to the loss; (L2, ridge, weight decay) or (L1, lasso)
- λ (regularization strength)
- the exchange rate between fit and weight size. is the unregularized model; forces
- Shrinkage
- moving estimates toward 0 (or another fixed value). It always adds bias and reduces variance
- Sparsity
- having many weights exactly 0. L1 produces it, L2 does not
- Soft-thresholding
- : subtract from the magnitude and clip at 0. This is what L1 does to each weight
- Subgradient
- the set of slopes of a kinked function at the kink. For at it is the whole interval
- Weight decay
- the L2 penalty seen from gradient descent: each step multiplies by before the loss gradient is applied
- Regularization path
- the fitted weights as a function of , from all-zero (large ) to the unregularized fit ()
- λ_max
- for L1, the smallest at which every weight is 0. The path starts there
- Effective degrees of freedom
- how many parameters the fit really uses. For L2 it is , between 0 and ; for L1 it is roughly the number of non-zero weights
The math
Write the model as with standardized features (mean 0, variance 1). The penalized objective is the average loss plus times the penalty. For squared error:
- number of training rows and of features
- feature vector and target of row
- weights and intercept. is not penalized; with centred features its optimum is
- regularization strength
- squared Euclidean length (L2)
- sum of absolute values (L1)
The same recipe works for any loss : log-loss for logistic regression, hinge loss for SVMs, cross-entropy for a neural network. Only the first term changes.
Deriving the two shrinkage rules
The difference between L1 and L2 is clearest when the features are orthonormal: centred, and for . Designed experiments have this property, and so do principal-component scores after scaling. The unregularized solution is then simply , and expanding the square gives
- the ordinary least-squares weight of feature j
- for L2, for L1
The objective splits into separate one-dimensional problems, so we can solve each weight on its own.
Solving one coordinate
- 1L2. is a smooth parabola. Set the derivative to zero: , so . Every weight is divided by the same factor, and it reaches 0 only when .
- 2L1, case . The derivative is , so . This is consistent only if .
- 3L1, case . By symmetry , consistent only if .
- 4L1, case . has no derivative at 0; its subgradient there is the interval . Zero is optimal when , which means . Every OLS weight in the band is set to exactly 0.
- 5Together: , the soft-thresholding operator.
- the regularized weight (orthonormal features)
- L2: proportional shrinkage. The same fraction is removed from every weight
- L1: constant shrinkage, clipped at 0. The same amount is removed from every weight
For general correlated features there is no per-weight formula. L2 still has a closed form, , and the added makes the matrix invertible even when (see Ridge regression). L1 is solved by applying soft-thresholding to one coordinate at a time, which is coordinate descent (see Lasso regression).
Why the kink makes zeros: the gradient view
For gradient-trained models such as networks and logistic regression, it helps to look at what each penalty does to a single update with step size :
- learning rate
- L2 multiplies each weight by a factor just below 1 on every step, hence the name weight decay
- L1's proximal step: take the ordinary gradient step, then soft-threshold by (the ISTA algorithm)
The L2 pull, , fades as the weight gets small, so it never quite reaches zero. The L1 pull, , stays the same size all the way down. Once a weight is at 0 it stays there while . Gradient descent on without the proximal step is not enough: the weights oscillate around zero and never land on it. Use a solver that handles the kink.
What λ does
Reading the knob
- λ = 0
- no penalty: the ordinary least-squares or maximum-likelihood fit, with the most variance
- Small λ
- removes the wild, noise-driven weight directions first. L2 shrinks most along directions where the data has little spread (small eigenvalues of XᵀX/n)
- Moderate λ
- usually the sweet spot: test error is lowest, and L1 has dropped the irrelevant features
- λ ≥ λ_max (L1)
- every weight is 0 and the model predicts .
- λ → ∞ (L2)
- weights approach 0 but are never exactly 0
- Effective size
- L2 with eigenvalues of keeps effective parameters. In the lab below this falls from 10 to 8.44 at and 4.22 at
Worked example: a crop trial, by hand
An agronomist runs every combination of three two-level treatments: fertilizer , irrigation and seed variety , each coded (low/standard) or (high/new). That makes plots, and is the yield in tonnes per hectare. Coding a full factorial design this way makes the columns orthonormal, so the formulas above apply exactly and every step can be done by hand.
| Plot | x₁ fertilizer | x₂ irrigation | x₃ variety | y (t/ha) | y − ȳ |
|---|---|---|---|---|---|
| 1 | −1 | −1 | −1 | 7.0 | −3.0 |
| 2 | −1 | −1 | +1 | 6.0 | −4.0 |
| 3 | −1 | +1 | −1 | 6.5 | −3.5 |
| 4 | −1 | +1 | +1 | 8.5 | −1.5 |
| 5 | +1 | −1 | −1 | 11.0 | +1.0 |
| 6 | +1 | −1 | +1 | 12.0 | +2.0 |
| 7 | +1 | +1 | −1 | 14.5 | +4.5 |
| 8 | +1 | +1 | +1 | 14.5 | +4.5 |
Fit OLS, L2 and L1 at λ = 0.5
- 1Intercept. . It is never penalized: shrinking it would only pull every prediction toward 0 t/ha.
- 2Check orthonormality. Each column has four s and four s, so its mean is 0 and . Any two columns agree on 4 rows and disagree on 4, so .
- 3OLS weights . . . . Training MSE .
- 4L2 at . Divide by : . All three weights lose a third of their size.
- 5L1 at . Subtract 0.5 from each magnitude and clip: . Variety () drops out of the model, and fertilizer keeps 83% of its effect.
- 6Training fit gets worse, as expected. Training MSE rises from 0.5625 to 1.6806 (L2) and 1.1250 (L1). The full objective still improves. Under L2, , against 2.7969 for the OLS weights. Under L1, , against 2.4063.
- 7Predict a new plot with every treatment high, : OLS ; L2 ; L1 t/ha.
| Feature | OLS ŵ | L2, λ = 0.5 | L1, λ = 0.5 | L1 weight is 0 once | L2 weight at λ = 3 |
|---|---|---|---|---|---|
| x₁ fertilizer | 3.00 | 2.000 | 2.50 | λ ≥ 3 | 0.750 |
| x₂ irrigation | 1.00 | 0.667 | 0.50 | λ ≥ 1 | 0.250 |
| x₃ variety | 0.25 | 0.167 | 0 | λ ≥ 0.25 | 0.063 |
Is the shrunken model better? Training error alone cannot tell you, because the regularized fit is worse on the training set by construction. The real question is how it does on new data, and with 8 plots and 3 features there is little variance to remove. The lab tab in the figure has more features than the data can support, and there the answer is a clear yes.
Watching it happen
| λ | L2 train | L2 test | L1 train | L1 test | L1 non-zero |
|---|---|---|---|---|---|
| 0 (OLS) | 0.223 | 2.807 | 0.223 | 2.807 | 10 |
| 0.001 | 0.223 | 2.789 | 0.223 | 2.783 | 10 |
| 0.100 | 0.354 | 2.000 | 0.425 | 1.457 | 7 |
| 0.133 | 0.419 | 1.950 | 0.566 | 1.348 | 7 |
| 0.178 | 0.509 | 1.944 | 0.686 | 1.369 | 6 |
| 0.316 | 0.782 | 2.115 | 0.972 | 1.608 | 4 |
| 1.000 | 1.759 | 3.222 | 4.117 | 5.033 | 3 |
| 3.162 | 3.062 | 4.606 | 5.246 | 6.212 | 0 |
Training error only goes up as grows. Test error first falls, then rises, which is the U-shape of the bias–variance trade-off. Lasso wins here because the truth is sparse. Along its path the first three features to enter are exactly the true ones: at (), then and at . The next one, , does not appear until . Note also that at lasso has the right three features but test MSE 5.03. Their weights are shrunk far below the true , so correct selection alone does not give accurate estimates.
| λ | L2 bias² | L2 variance | L2 expected MSE | L1 bias² | L1 variance | L1 expected MSE |
|---|---|---|---|---|---|---|
| 0.001 | 0.001 | 1.289 | 2.290 | 0.001 | 1.285 | 2.285 |
| 0.1 | 0.166 | 0.726 | 1.892 | 0.079 | 0.511 | 1.590 |
| 0.316 | 0.669 | 0.593 | 2.262 | 0.497 | 0.451 | 1.948 |
| 1.0 | 1.836 | 0.489 | 3.325 | 2.808 | 0.506 | 4.314 |
This is the claim in numbers. Going from to , L2 adds 0.165 of bias² and removes 0.563 of variance, for a net gain of 0.40. L1 adds 0.078 and removes 0.774. Past the optimum the bias term grows faster than the variance shrinks.
The algorithm, step by step
Fitting a regularized model
- 1Split the data into training and test sets (or set up cross-validation folds) before anything else.
- 2Standardize each feature with the training mean and standard deviation. The penalty treats all weights alike, so they must be on the same scale. Leave the intercept out of the penalty.
- 3Build a λ grid, log-spaced, from (L1) or a large value (L2) down to about .
- 4Fit along the path, from large to small , warm-starting each fit from the previous one. L2 uses the closed form, or gradient descent with weight decay. L1 uses coordinate descent: repeatedly set , where is the residual leaving out feature .
- 5Choose λ with the lowest cross-validated error, or the largest within one standard error of it (the "1-SE rule", which gives a simpler model).
- 6Refit on all training data at that λ, then evaluate once on the test set.
Complexity
- L2, closed form
- once. After one SVD of , per extra , so the whole path is nearly free
- L1, coordinate descent
- per sweep over all coordinates, with a few to a few hundred sweeps per . Warm starts make the full path cost about as much as a few single fits
- Gradient training
- L2 adds O(p) per step (weight decay); L1 needs a proximal or soft-threshold step, also O(p)
- Prediction
- , or after L1
Variants and practical tuning
| Penalty | R(w) | Effect | Where you meet it |
|---|---|---|---|
| L2 (ridge, Tikhonov, weight decay) | ½‖w‖₂² | shrinks all weights proportionally; stabilizes correlated features | Ridge(alpha) minimizes ‖y−Xw‖² + α‖w‖², so α = nλ; LogisticRegression(C) with C = 1/(nλ); weight_decay in PyTorch optimizers |
| L1 (lasso) | ‖w‖₁ | constant shrinkage, exact zeros, feature selection | Lasso(alpha) uses the (1/2n) scaling, so α = λ; LogisticRegression(penalty='l1', solver='liblinear' or 'saga') |
| Elastic net | ρ‖w‖₁ + (1−ρ)½‖w‖₂² | sparse, but keeps groups of correlated features together | ElasticNet(alpha, l1_ratio=ρ). See Elastic Net regression |
| Group lasso | Σ_g ‖w_g‖₂ | drops whole groups (e.g. all dummies of one categorical) | specialised packages (group-lasso, glmnet extensions) |
| L0 (best subset) | #{j : w_j ≠ 0} | hard selection with no shrinkage; combinatorial and non-convex | stepwise selection, mixed-integer solvers |
Tuning checklist
- Scale first
- Put
StandardScalerinside aPipelineso it is refit on each CV fold - Search on a log grid
- λ matters on a multiplicative scale: try 10⁻³ to 10¹ (times λ_max for L1)
- Use CV, not the test set
RidgeCV,LassoCV,LogisticRegressionCV,ElasticNetCVcompute the path and cross-validate efficiently- Prefer simpler on ties
- the 1-SE rule picks the largest λ whose CV error is within one standard error of the minimum
- Debias L1 if you need sizes
- refit OLS on the features L1 selected (the relaxed lasso) if you need accurate effect sizes, not just a selected set
- Neural nets
- L2 is
weight_decay; with Adam use AdamW, which decouples the decay from the adaptive step
In code
1import numpy as np23def standardize(X):4 mu, sd = X.mean(0), X.std(0)5 return (X - mu) / sd, mu, sd67def ridge(X, y, lam):8 """argmin (1/2n)||y - b - Xw||^2 + (lam/2)||w||^2 on standardized X."""9 Z, mu, sd = standardize(X)10 n, p = Z.shape11 w = np.linalg.solve(Z.T @ Z / n + lam * np.eye(p), Z.T @ (y - y.mean()) / n)12 return w, y.mean(), mu, sd1314def soft(z, t):15 return np.sign(z) * np.maximum(np.abs(z) - t, 0.0)1617def lasso(X, y, lam, tol=1e-10, max_sweeps=100_000):18 """argmin (1/2n)||y - b - Xw||^2 + lam*||w||_1 by cyclic coordinate descent."""19 Z, mu, sd = standardize(X)20 n, p = Z.shape21 w = np.zeros(p)22 r = y - y.mean() # residual for w = 023 for _ in range(max_sweeps):24 biggest = 0.025 for j in range(p):26 old = w[j]27 rho = Z[:, j] @ r / n + old # z_j^T (partial residual) / n (z_j^T z_j / n = 1)28 w[j] = soft(rho, lam)29 if w[j] != old:30 r -= Z[:, j] * (w[j] - old)31 biggest = max(biggest, abs(w[j] - old))32 if biggest < tol:33 break34 return w, y.mean(), mu, sd3536def predict(model, X):37 w, b, mu, sd = model38 return b + ((X - mu) / sd) @ w3940# Worked example: 2^3 factorial experiment, features coded -1/+141X = np.array([[a, b, c] for a in (-1, 1) for b in (-1, 1) for c in (-1, 1)], float)42y = np.array([7, 6, 6.5, 8.5, 11, 12, 14.5, 14.5])43x_new = np.array([[1, 1, 1]])4445for name, model in [("OLS ", ridge(X, y, 0.0)), ("ridge", ridge(X, y, 0.5)), ("lasso", lasso(X, y, 0.5))]:46 mse = np.mean((y - predict(model, X)) ** 2)47 print(name, "w =", np.round(model[0], 3), " train MSE = %.4f" % mse, " predict(1,1,1) = %.2f" % predict(model, x_new)[0])1import numpy as np2from sklearn.linear_model import Lasso, LassoCV, LogisticRegression, Ridge3from sklearn.pipeline import make_pipeline4from sklearn.preprocessing import StandardScaler56# --- 1. The worked example. sklearn's Ridge minimizes ||y - Xw||^2 + alpha*||w||^2,7# so our lambda = 0.5 is alpha = n * lambda = 4; Lasso already uses (1/2n) and alpha = lambda.8X = np.array([[a, b, c] for a in (-1, 1) for b in (-1, 1) for c in (-1, 1)], float)9y = np.array([7, 6, 6.5, 8.5, 11, 12, 14.5, 14.5])10n, lam = len(y), 0.511ridge = Ridge(alpha=n * lam).fit(X, y)12lasso = Lasso(alpha=lam).fit(X, y)13print("ridge", ridge.intercept_, ridge.coef_.round(3), ridge.predict([[1, 1, 1]]).round(2)) # 10.0 [2. 0.667 0.167] [12.83]14print("lasso", lasso.intercept_, lasso.coef_.round(3), lasso.predict([[1, 1, 1]]).round(2)) # 10.0 [2.5 0.5 0. ] [13.]1516# --- 2. The overfitting lab: 20 training rows, 10 correlated features, only x1, x4, x7 matter.17rs = np.random.RandomState(2)18p = 1019C = 0.5 ** np.abs(np.subtract.outer(np.arange(p), np.arange(p))) # corr(x_i, x_j) = 0.5^|i-j|20L = np.linalg.cholesky(C)21w_true = np.zeros(p); w_true[[0, 3, 6]] = [2.0, -1.5, 1.0]22def draw(m):23 Xs = np.round(rs.randn(m, p) @ L.T, 2)24 return Xs, np.round(Xs @ w_true + rs.randn(m), 2)25Xtr, ytr = draw(20)26Xte, yte = draw(100)2728mse = lambda model: np.mean((model.predict(Xte) - yte) ** 2)29grid = 10 ** (np.arange(-24, 5) / 8) # lambda from 0.001 to 3.1630ridges = [make_pipeline(StandardScaler(), Ridge(alpha=20 * l)).fit(Xtr, ytr) for l in grid]31lassos = [make_pipeline(StandardScaler(), Lasso(alpha=l, tol=1e-12, max_iter=100_000)).fit(Xtr, ytr) for l in grid]32ols = make_pipeline(StandardScaler(), Ridge(alpha=0)).fit(Xtr, ytr)33print("OLS test MSE %.3f" % mse(ols)) # 2.80734for name, models in [("ridge", ridges), ("lasso", lassos)]:35 k = int(np.argmin([mse(m) for m in models]))36 w = models[k][-1].coef_37 print(f"{name} best lambda {grid[k]:.3f} test MSE {mse(models[k]):.3f} non-zero {np.sum(w != 0)}/10 w = {w.round(2)}")38# ridge best lambda 0.178 test MSE 1.944 non-zero 10/1039# lasso best lambda 0.133 test MSE 1.348 non-zero 7/104041# --- 3. In practice: choose lambda by cross-validation, never on the test set.42cv = make_pipeline(StandardScaler(), LassoCV(alphas=grid, cv=5, max_iter=100_000)).fit(Xtr, ytr)43print("LassoCV lambda %.3f test MSE %.3f" % (cv[-1].alpha_, mse(cv))) # 0.024, 2.2894445# --- 4. The same penalties for classification: C = 1/(n*lambda) is the INVERSE strength.46yc = (Xtr @ w_true > 0).astype(int)47for pen in ["l2", "l1"]:48 clf = make_pipeline(StandardScaler(), LogisticRegression(penalty=pen, C=0.5, solver="liblinear")).fit(Xtr, yc)49 print(pen, "logistic non-zero weights:", np.sum(clf[-1].coef_ != 0)) # l2: 10, l1: 5Use when
- +You have many features relative to rows, or correlated features, and the unregularized fit has a large train/test gap.
- +L1: you believe only a few features matter and want an automatically selected, cheaper, readable model.
- +L2: many features each contribute a little, or you need a stable and unique solution (it also fixes singular XᵀX).
- +You train neural nets or logistic regression: weight decay is a cheap, almost universal default.
Avoid when
- −Features are on different scales and you cannot standardize them. The penalty then depends on the units.
- −You need unbiased effect sizes or valid p-values: shrinkage biases every weight toward 0 (refit or use post-selection inference).
- −L1 with strongly correlated features: it picks one of the group more or less arbitrarily and the choice changes between samples. Use elastic net.
- −You have plenty of data and few features: the variance is already small, so regularization buys little.
Where it's used
Genomics and high-dimensional biology
Thousands of gene-expression features and a few hundred patients. L1 or elastic-net logistic regression picks a short list of marker genes.
Text classification
L2- or L1-regularized logistic regression on bag-of-words with a vocabulary of 10⁵+. L1 gives compact models that fit on small devices.
Deep learning
Weight decay (L2) is standard in training CNNs and transformers, typically via AdamW.
Econometrics and forecasting
Lasso chooses a few predictors from hundreds of lagged indicators for macro nowcasting and demand forecasting.
Check yourself
Orthonormal features. A weight has OLS value ŵ = 0.8, and λ = 1. What do L2 and L1 give?
Check yourself
Why can L1 produce weights that are exactly zero while L2 cannot?
Check yourself
In the overfitting lab, raising λ from 0.001 to 0.1 for L2 changes bias² from 0.001 to 0.166 and variance from 1.289 to 0.726. What happens to the expected test MSE (noise = 1)?
Check yourself
Which prior on the weights makes the MAP estimate equal to L1-regularized least squares?
Remember
- ✓Regularization minimizes loss . L2 uses and L1 uses . The intercept is not penalized and features must be standardized.
- ✓With orthonormal features, L2 divides each weight by (proportional shrinkage) and L1 soft-thresholds it, , so every weight with becomes exactly 0.
- ✓Training error always rises with λ and test error is U-shaped: bias² grows while variance falls. In the lab, OLS test MSE 2.807 fell to 1.944 (L2) and 1.348 (L1, 7 of 10 weights kept).
- ✓L2 is weight decay, a Gaussian prior and a sphere constraint. L1 is a proximal soft-threshold step, a Laplace prior and a diamond constraint whose corners produce zeros.
- ✓Choose λ on a log grid by cross-validation inside a pipeline. Watch library conventions: Ridge α = nλ, Lasso α = λ, and C = inverse strength.