Definition
Ridge regression is linear least squares with a price on large weights: it minimizes the sum of squared errors plus times the sum of squared coefficients. The penalty pulls every weight toward zero — smoothly, never exactly to zero — and in exchange the solution always exists, is unique, and changes much less when the data changes.
At a glance
- Task
- Regression (numeric target); the same penalty is used in classifiers as L2 regularization
- Input → output
- feature vector →
- Learns
- a weight per feature and an unpenalized intercept
- Training
- closed form: one solve, ; or in the kernel/dual form when
- Prediction
- — one dot product
- Hyperparameters
- penalty strength (scikit-learn: alpha); whether to standardize; whether to fit the intercept
- Assumes
- a linear relationship, features on comparable scales; Gaussian noise if you want the Bayesian reading
The problem
Ordinary least squares (see Multiple linear regression) solves . That formula is unbiased, but it has a weak spot: when two features carry nearly the same information, is almost singular, its inverse is enormous, and the weights become wild. OLS can no longer tell which of the two features deserves the credit, so it gives one a huge positive weight and the other a compensating negative one — any split fits the training data about equally well.
Our running example has exactly this disease. Two temperature sensors, A and B, read almost the same value each day (correlation 0.996), and we predict ice-cream sales from both. OLS returns : hotter on sensor B means fewer sales, which is nonsense. Worse, the fit is fragile — add 1 to a single sales figure (day 3) and OLS jumps to ; the first weight moves by 0.80 because of one unit of noise.
The idea
Keep the squared error, but add a charge for every unit of squared weight. A wild solution like has ; a sensible one like has and fits almost as well. Once weight costs something, the model prefers the cheap solution that shares credit between correlated features instead of playing them against each other.
Geometrically, the loss surface of OLS has a long, flat valley along the direction where the two sensors disagree — the data says almost nothing about that direction. The penalty is a round bowl centred at the origin. Adding the bowl lifts the flat valley floor, so the minimum is pinned down in the directions the data cannot determine, while the directions it can determine are barely touched.
Vocabulary
- Penalty / regularizer
- the extra term added to the loss
- λ (alpha)
- penalty strength. is OLS; forces and predicts
- Shrinkage
- pulling estimated weights toward zero relative to OLS
- Collinearity
- features that are (nearly) linear combinations of each other, making (nearly) singular
- Condition number
- ratio of the largest to smallest eigenvalue of ; large means unstable OLS (489 in our example)
- Ridge trace
- plot of each weight against
- Effective degrees of freedom
- : how many parameters the fit is really using, from down to 0
The math
Centre every feature and the target first (subtract their means). Then the intercept separates out as , and the penalty applies only to .
- matrix of centred features (row is )
- length-n vector of centred targets
- length-p weight vector (intercept excluded)
- penalty strength
- , the squared Euclidean length of
Deriving the closed form. is a sum of two convex quadratics, so it has a single minimum where the gradient vanishes. Expand and differentiate:
- identity matrix
- ridge solution for this
The only change from OLS is on the diagonal — the "ridge" that gives the method its name. is positive semi-definite, so its eigenvalues are ; adding raises every one of them by . For any the matrix is therefore strictly positive definite and always invertible, even when features are perfectly collinear or when .
What the shrinkage actually does. Write the centred data's singular value decomposition (columns of are the principal directions, the singular values; are the eigenvalues of ). Substituting into the closed form gives
- -th right / left singular vectors of
- -th singular value; is the data's spread along (times )
- the OLS coefficient along
- shrinkage factor, between 0 and 1
So ridge takes the OLS solution, rotates into principal directions, and multiplies each component by . Directions with large spread () keep almost all of their coefficient; directions with tiny spread () are squashed toward zero. These low-spread directions are exactly where OLS was guessing. In our sensors data (the "average temperature" direction) and (the "sensor A minus sensor B" direction); at the factors are and .
- effective degrees of freedom: at , falling smoothly to 0 as
- trace (sum of the diagonal)
Effect of . Raising (1) shrinks every coordinate along by a larger factor, so decreases monotonically; (2) increases bias — ; (3) decreases variance — , whose eigenvalues are always smaller than OLS's . A classical result (Hoerl & Kennard, 1970) is that some always has lower total mean squared error of than OLS.
- a weight budget; each corresponds to one (larger ↔ smaller )
This constrained form is the same problem (by Lagrange duality): the solution is where the smallest loss ellipse touches the round ball . Because the ball has no corners, the touching point almost never lies on an axis — ridge does not set weights to exactly zero. Compare this with the diamond-shaped L1 ball in Lasso regression.
Worked example: two sensors, one temperature
Eight days. Sensors A and B both measure the afternoon temperature (scaled units); the target is ice-cream sales (hundreds of cones). The means are and .
| Day | Sensor A (x₁) | Sensor B (x₂) | Sales y | x₁ − 4.5 | x₂ − 4.5 | y − 13 |
|---|---|---|---|---|---|---|
| 1 | 1 | 1.2 | 5 | −3.5 | −3.3 | −8 |
| 2 | 2 | 1.8 | 8 | −2.5 | −2.7 | −5 |
| 3 | 3 | 3.3 | 8 | −1.5 | −1.2 | −5 |
| 4 | 4 | 3.9 | 13 | −0.5 | −0.6 | 0 |
| 5 | 5 | 4.8 | 13 | 0.5 | 0.3 | 0 |
| 6 | 6 | 6.3 | 17 | 1.5 | 1.8 | 4 |
| 7 | 7 | 6.9 | 18 | 2.5 | 2.4 | 5 |
| 8 | 8 | 7.8 | 22 | 3.5 | 3.3 | 9 |
Ridge with λ = 2, by hand
- 1Gram matrix and right-hand side from the centred columns: , , ; , . So , .
- 2OLS for comparison. — tiny next to the entries, the signature of collinearity. , .
- 3Add the ridge. , with determinant — thirteen times larger, so the inverse is tame.
- 4Solve with the 2×2 inverse : and . Both positive, nearly equal: the credit is shared.
- 5Intercept (not penalized): .
- 6Predict a new day where the sensors disagree a little, : ridge gives . OLS gives — it lowers the forecast because sensor B read higher, an artefact of the negative weight.
| λ | w₁ | w₂ | ‖w‖₂ | train MSE | LOO MSE | df(λ) | sd(w₁) |
|---|---|---|---|---|---|---|---|
| 0 (OLS) | 3.526 | −1.228 | 3.734 | 0.855 | 2.113 | 2.000 | 1.638 |
| 0.25 | 2.125 | 0.202 | 2.134 | 0.939 | 1.772 | 1.398 | 0.654 |
| 1 | 1.513 | 0.807 | 1.715 | 1.031 | 1.811 | 1.131 | 0.238 |
| 2 | 1.345 | 0.952 | 1.648 | 1.070 | 1.861 | 1.053 | 0.139 |
| 10 | 1.102 | 0.998 | 1.486 | 1.418 | 2.625 | 0.908 | 0.071 |
| 100 | 0.541 | 0.520 | 0.750 | 9.685 | 16.050 | 0.452 | 0.035 |
Read the table top to bottom. The first small already kills the negative weight. By the model uses about one effective parameter (): essentially "average the two sensors, times 2.3". The noise sensitivity of falls twelvefold, from 1.638 to 0.139. Past the penalty starts shrinking the useful direction too, and at both weights are halved and the error explodes — that is underfitting.
The algorithm, step by step
Fit ridge regression
- 1Standardize each feature to mean 0 and standard deviation 1 (remember the means and scales); centre . The penalty is only fair if all features are on the same scale.
- 2Choose a grid of values, log-spaced (e.g. ).
- 3For each , solve with a Cholesky factorization — or compute one SVD of once and get every cheaply from the shrinkage formula.
- 4Score each by cross-validation. For ridge, leave-one-out residuals have a closed form, with the diagonal of the hat matrix (plus for the intercept), so LOO costs no extra fits.
- 5Refit on all training data with the best ; set and undo the standardization to report weights in original units.
- 6Predict for new inputs (after applying the same standardization).
Cost
- Primal solve
- form in , factor in — best when
- Dual / kernel solve
- : — best when (e.g. genomics, text); with a kernel this is kernel ridge regression
- Whole λ path
- one SVD , then per
- Huge or streaming data
- gradient descent / SGD on J; the penalty adds 2λw to the gradient ("weight decay")
Tuning and variants
| Knob / variant | What it does | Guidance |
|---|---|---|
| λ (alpha) | Strength of shrinkage; sets df(λ) between p and 0 | Search a log grid 1e-3…1e3 with CV; RidgeCV does efficient LOO by default |
| Standardization | Makes the penalty unit-free | Always, unless all features already share units |
| Intercept | Left unpenalized so shifting y shifts predictions | Keep fit_intercept=True; centre data if you code it yourself |
| Solver | cholesky, svd, sparse_cg, lsqr, sag/saga | cholesky/svd for small dense data, sag/saga or sparse_cg for large or sparse |
| Generalized ridge | Penalty ‖Γw‖², e.g. a different λ per feature or a smoothness penalty on neighbouring weights | Use when some coefficients should be shrunk more, or toward each other |
| Kernel ridge | Same objective in a feature space defined by a kernel | Nonlinear fits on small-to-medium n |
| Elastic Net | Adds an L1 term to get some exact zeros | When you also need feature selection — see Elastic Net regression |
In code
1import numpy as np23class RidgeScratch:4 def __init__(self, lam=1.0):5 self.lam = lam67 def fit(self, X, y):8 X, y = np.asarray(X, float), np.asarray(y, float)9 self.x_mean, self.y_mean = X.mean(0), y.mean()10 Xc, yc = X - self.x_mean, y - self.y_mean # centre: intercept is not penalised11 d = X.shape[1]12 A = Xc.T @ Xc + self.lam * np.eye(d) # X^T X + lambda I (always invertible for lam > 0)13 self.w = np.linalg.solve(A, Xc.T @ yc)14 self.b = self.y_mean - self.x_mean @ self.w15 # leave-one-out residuals in closed form: e_i / (1 - h_ii)16 H = Xc @ np.linalg.solve(A, Xc.T) + 1 / len(y)17 e = y - self.predict(X)18 self.loo_mse = np.mean((e / (1 - np.diag(H))) ** 2)19 # SVD view: shrinkage factor per principal direction, effective degrees of freedom20 s = np.linalg.svd(Xc, compute_uv=False)21 self.shrink = s**2 / (s**2 + self.lam)22 self.df = self.shrink.sum()23 return self2425 def predict(self, X):26 return np.asarray(X, float) @ self.w + self.b2728X = np.array([[1, 1.2], [2, 1.8], [3, 3.3], [4, 3.9], [5, 4.8], [6, 6.3], [7, 6.9], [8, 7.8]])29y = np.array([5, 8, 8, 13, 13, 17, 18, 22])3031for lam in [0, 2]:32 m = RidgeScratch(lam).fit(X, y)33 print(f"lam={lam}: w={m.w.round(4)} b={m.b:.4f} LOO-MSE={m.loo_mse:.4f} "34 f"df={m.df:.4f} shrink={m.shrink.round(4)} predict(6,7)={m.predict([[6, 7]])[0]:.3f}")3536# lam=0: w=[ 3.5263 -1.2281] b=2.6579 LOO-MSE=2.1134 df=2.0000 shrink=[1. 1.] predict(6,7)=15.21937# lam=2: w=[1.3449 0.9515] b=2.6659 LOO-MSE=1.8612 df=1.0533 shrink=[0.9761 0.0772] predict(6,7)=17.3961import numpy as np2from sklearn.linear_model import LinearRegression, Ridge, RidgeCV3from sklearn.pipeline import make_pipeline4from sklearn.preprocessing import StandardScaler56X = np.array([[1, 1.2], [2, 1.8], [3, 3.3], [4, 3.9], [5, 4.8], [6, 6.3], [7, 6.9], [8, 7.8]])7y = np.array([5, 8, 8, 13, 13, 17, 18, 22])8x_new = [[6, 7]]910ols = LinearRegression().fit(X, y)11ridge = Ridge(alpha=2).fit(X, y) # sklearn's alpha is the article's lambda12print("OLS ", ols.coef_.round(4), round(ols.intercept_, 4), ols.predict(x_new).round(3))13print("Ridge", ridge.coef_.round(4), round(ridge.intercept_, 4), ridge.predict(x_new).round(3))1415# choose lambda by efficient leave-one-out CV over a log grid16alphas = [m * 10.0**e for e in range(-2, 3) for m in (1, 1.25, 1.6, 2, 2.5, 3.15, 4, 5, 6.3, 8)]17cv = RidgeCV(alphas=alphas, scoring="neg_mean_squared_error").fit(X, y)18print("RidgeCV alpha =", cv.alpha_, "LOO MSE =", round(-cv.best_score_, 4), "coef", cv.coef_.round(4))1920# in real work: standardise first, so the penalty treats every feature equally21model = make_pipeline(StandardScaler(), RidgeCV(alphas=np.logspace(-3, 3, 61))).fit(X, y)22print("pipeline alpha =", round(model[-1].alpha_, 4), "predict", model.predict(x_new).round(3))2324# OLS [ 3.5263 -1.2281] 2.6579 [15.219]25# Ridge [1.3449 0.9515] 2.6659 [17.396]26# RidgeCV alpha = 0.25 LOO MSE = 1.7725 coef [2.1249 0.2016]27# pipeline alpha = 0.0501 predict [16.725]The best alpha after standardizing (0.0501) is far smaller than on raw data (0.25), because standardized columns have smaller sums of squares — the same amount of shrinkage needs a smaller number. One more reason not to copy λ values between setups.
Use when
- +Features are correlated or numerous (even p > n) and you want stable, well-defined weights.
- +You believe many features each contribute a little, rather than a few dominating.
- +You want a fast, closed-form, convex baseline with one hyperparameter and cheap leave-one-out CV.
- +You need smooth predictions that change little when the training data changes.
Avoid when
- −You need a sparse model: ridge keeps every feature with a non-zero weight — use Lasso or Elastic Net.
- −The relationship is strongly nonlinear (use kernel ridge, trees or boosting).
- −You need unbiased coefficient estimates for inference; ridge weights are deliberately biased toward zero.
- −Features cannot be put on a common scale meaningfully, so a single λ has no sensible interpretation.
Where it's used
Genomics and polygenic scores
Tens of thousands of correlated genetic markers, a few thousand people: ridge (GBLUP in animal breeding) spreads small effects across many markers.
Economics and marketing mix
Ad spends across channels move together; ridge keeps channel effects from flipping sign when budgets are collinear.
Chemometrics / spectroscopy
Hundreds of adjacent wavelengths are nearly collinear; ridge (and its cousin PLS) predicts concentrations stably.
Weight decay in neural networks
The same L2 penalty, applied by every deep-learning optimizer as weight decay, and the closed-form readout in kernel and random-feature models.
Check yourself
A principal direction of the centred data has . With , by what factor does ridge multiply the OLS coefficient along that direction?
Check yourself
You duplicate a feature, so the design has two identical columns. What does ridge (λ > 0) do with them?
Check yourself
As λ → ∞ (intercept not penalized), ridge predictions tend to…
Check yourself
Why does adding λI fix collinearity?
Remember
- ✓Ridge minimizes ; the closed form is , always solvable for .
- ✓In principal directions it multiplies each OLS component by : low-variance directions — where collinearity lives — are shrunk most.
- ✓Correlated features share credit instead of getting huge opposite-sign weights; weights shrink smoothly but never to exactly zero.
- ✓Standardize features and leave the intercept unpenalized; choose λ by cross-validation (leave-one-out is free for ridge), never by training error.
- ✓Equivalent views: an L2-ball constraint, a Gaussian prior with , and weight decay in gradient descent.