Ridge regression

Add λ‖w‖₂² to the squared error; weights shrink toward zero, which tames collinearity and lowers variance.

16 min read1551 wordscore8 viewsupdated 2026-10-07

Definition

Ridge regression is linear least squares with a price on large weights: it minimizes the sum of squared errors plus λ\lambda times the sum of squared coefficients. The penalty pulls every weight toward zero — smoothly, never exactly to zero — and in exchange the solution always exists, is unique, and changes much less when the data changes.

At a glance

Task
Regression (numeric target); the same penalty is used in classifiers as L2 regularization
Input → output
feature vector x∈Rpx\in\mathbb{R}^p → y^=b+w⊤x\hat y = b + w^\top x
Learns
a weight wjw_j per feature and an unpenalized intercept bb
Training
closed form: one p×pp\times p solve, O(np2+p3)O(np^2+p^3); or O(n2p)O(n^2p) in the kernel/dual form when p≫np\gg n
Prediction
O(p)O(p) — one dot product
Hyperparameters
penalty strength λ\lambda (scikit-learn: alpha); whether to standardize; whether to fit the intercept
Assumes
a linear relationship, features on comparable scales; Gaussian noise if you want the Bayesian reading

The problem

Ordinary least squares (see Multiple linear regression) solves w^=(X⊤X)−1X⊤y\hat w=(X^\top X)^{-1}X^\top y. That formula is unbiased, but it has a weak spot: when two features carry nearly the same information, X⊤XX^\top X is almost singular, its inverse is enormous, and the weights become wild. OLS can no longer tell which of the two features deserves the credit, so it gives one a huge positive weight and the other a compensating negative one — any split fits the training data about equally well.

Our running example has exactly this disease. Two temperature sensors, A and B, read almost the same value each day (correlation 0.996), and we predict ice-cream sales from both. OLS returns w=(3.526, −1.228)w=(3.526,\,-1.228): hotter on sensor B means fewer sales, which is nonsense. Worse, the fit is fragile — add 1 to a single sales figure (day 3) and OLS jumps to w=(2.724, −0.439)w=(2.724,\,-0.439); the first weight moves by 0.80 because of one unit of noise.

The idea

Keep the squared error, but add a charge for every unit of squared weight. A wild solution like (3.5,−1.2)(3.5,-1.2) has ∥w∥22=13.9\|w\|_2^2=13.9; a sensible one like (1.3,1.0)(1.3,1.0) has ∥w∥22=2.7\|w\|_2^2=2.7 and fits almost as well. Once weight costs something, the model prefers the cheap solution that shares credit between correlated features instead of playing them against each other.

Geometrically, the loss surface of OLS has a long, flat valley along the direction where the two sensors disagree — the data says almost nothing about that direction. The penalty is a round bowl centred at the origin. Adding the bowl lifts the flat valley floor, so the minimum is pinned down in the directions the data cannot determine, while the directions it can determine are barely touched.

Vocabulary

Penalty / regularizer
the extra term λ∥w∥22=λ∑jwj2\lambda\|w\|_2^2=\lambda\sum_j w_j^2 added to the loss
λ (alpha)
penalty strength. λ=0\lambda=0 is OLS; λ→∞\lambda\to\infty forces w→0w\to 0 and predicts yˉ\bar y
Shrinkage
pulling estimated weights toward zero relative to OLS
Collinearity
features that are (nearly) linear combinations of each other, making X⊤XX^\top X (nearly) singular
Condition number
ratio of the largest to smallest eigenvalue of X⊤XX^\top X; large means unstable OLS (489 in our example)
Ridge trace
plot of each weight wj(λ)w_j(\lambda) against λ\lambda
Effective degrees of freedom
df(λ)=tr⁡ X(X⊤X+λI)−1X⊤\text{df}(\lambda)=\operatorname{tr}\,X(X^\top X+\lambda I)^{-1}X^\top: how many parameters the fit is really using, from pp down to 0

The math

Centre every feature and the target first (subtract their means). Then the intercept separates out as b=yˉ−xˉ⊤wb=\bar y-\bar x^\top w, and the penalty applies only to ww.

J(w)=∥y−Xw∥22⏟squared error+λ∥w∥22⏟penalty=∑i=1n(yi−xi⊤w)2+λ∑j=1pwj2J(w)=\underbrace{\|y-Xw\|_2^2}_{\text{squared error}}+\underbrace{\lambda\|w\|_2^2}_{\text{penalty}}=\sum_{i=1}^n\big(y_i-x_i^\top w\big)^2+\lambda\sum_{j=1}^p w_j^2
XX
n×pn\times p matrix of centred features (row ii is xi⊤x_i^\top)
yy
length-n vector of centred targets
ww
length-p weight vector (intercept excluded)
λ≥0\lambda\ge 0
penalty strength
∥w∥22\|w\|_2^2
w⊤ww^\top w, the squared Euclidean length of ww

Deriving the closed form. JJ is a sum of two convex quadratics, so it has a single minimum where the gradient vanishes. Expand J=y⊤y−2w⊤X⊤y+w⊤X⊤Xw+λw⊤wJ=y^\top y-2w^\top X^\top y+w^\top X^\top Xw+\lambda w^\top w and differentiate:

∇wJ=−2X⊤y+2X⊤Xw+2λw=0    ⟹    (X⊤X+λI) w^=X⊤y    ⟹    w^λ=(X⊤X+λI)−1X⊤y\nabla_w J=-2X^\top y+2X^\top Xw+2\lambda w=0\;\;\Longrightarrow\;\;(X^\top X+\lambda I)\,\hat w=X^\top y\;\;\Longrightarrow\;\;\hat w_\lambda=(X^\top X+\lambda I)^{-1}X^\top y
II
p×pp\times p identity matrix
w^λ\hat w_\lambda
ridge solution for this λ\lambda

The only change from OLS is +λI+\lambda I on the diagonal — the "ridge" that gives the method its name. X⊤XX^\top X is positive semi-definite, so its eigenvalues are ≥0\ge 0; adding λI\lambda I raises every one of them by λ\lambda. For any λ>0\lambda>0 the matrix is therefore strictly positive definite and always invertible, even when features are perfectly collinear or when p>np>n.

What the shrinkage actually does. Write the centred data's singular value decomposition X=UDV⊤X=UDV^\top (columns vjv_j of VV are the principal directions, djd_j the singular values; dj2d_j^2 are the eigenvalues of X⊤XX^\top X). Substituting into the closed form gives

w^λ=∑j=1pvj dj2dj2+λ⋅uj⊤ydj,Xw^λ=∑j=1puj dj2dj2+λ uj⊤y\hat w_\lambda=\sum_{j=1}^{p} v_j\,\frac{d_j^2}{d_j^2+\lambda}\cdot\frac{u_j^\top y}{d_j},\qquad X\hat w_\lambda=\sum_{j=1}^{p}u_j\,\frac{d_j^2}{d_j^2+\lambda}\,u_j^\top y
vj,  ujv_j,\;u_j
jj-th right / left singular vectors of XX
djd_j
jj-th singular value; dj2d_j^2 is the data's spread along vjv_j (times nn)
uj⊤ydj\tfrac{u_j^\top y}{d_j}
the OLS coefficient along vjv_j
dj2dj2+λ\tfrac{d_j^2}{d_j^2+\lambda}
shrinkage factor, between 0 and 1

So ridge takes the OLS solution, rotates into principal directions, and multiplies each component by dj2/(dj2+λ)d_j^2/(d_j^2+\lambda). Directions with large spread (dj2≫λd_j^2\gg\lambda) keep almost all of their coefficient; directions with tiny spread (dj2≪λd_j^2\ll\lambda) are squashed toward zero. These low-spread directions are exactly where OLS was guessing. In our sensors data d12=81.79d_1^2=81.79 (the "average temperature" direction) and d22=0.167d_2^2=0.167 (the "sensor A minus sensor B" direction); at λ=2\lambda=2 the factors are 0.9760.976 and 0.0770.077.

df(λ)=tr⁡ ⁣[X(X⊤X+λI)−1X⊤]=∑j=1pdj2dj2+λ\text{df}(\lambda)=\operatorname{tr}\!\big[X(X^\top X+\lambda I)^{-1}X^\top\big]=\sum_{j=1}^p\frac{d_j^2}{d_j^2+\lambda}
df(λ)\text{df}(\lambda)
effective degrees of freedom: pp at λ=0\lambda=0, falling smoothly to 0 as λ→∞\lambda\to\infty
tr⁡\operatorname{tr}
trace (sum of the diagonal)

Effect of λ\lambda. Raising λ\lambda (1) shrinks every coordinate along vjv_j by a larger factor, so ∥w^λ∥2\|\hat w_\lambda\|_2 decreases monotonically; (2) increases bias — E[w^λ]=(X⊤X+λI)−1X⊤X w∗≠w∗E[\hat w_\lambda]=(X^\top X+\lambda I)^{-1}X^\top X\,w^*\ne w^*; (3) decreases variance — Var⁡(w^λ)=σ2(X⊤X+λI)−1X⊤X(X⊤X+λI)−1\operatorname{Var}(\hat w_\lambda)=\sigma^2(X^\top X+\lambda I)^{-1}X^\top X(X^\top X+\lambda I)^{-1}, whose eigenvalues σ2dj2/(dj2+λ)2\sigma^2 d_j^2/(d_j^2+\lambda)^2 are always smaller than OLS's σ2/dj2\sigma^2/d_j^2. A classical result (Hoerl & Kennard, 1970) is that some λ>0\lambda>0 always has lower total mean squared error of w^\hat w than OLS.

min⁡w∥y−Xw∥22    subject to    ∥w∥22≤t\min_w \|y-Xw\|_2^2\;\;\text{subject to}\;\;\|w\|_2^2\le t
tt
a weight budget; each λ\lambda corresponds to one tt (larger λ\lambda ↔ smaller tt)

This constrained form is the same problem (by Lagrange duality): the solution is where the smallest loss ellipse touches the round ball ∥w∥2≤t\|w\|_2\le\sqrt t. Because the ball has no corners, the touching point almost never lies on an axis — ridge does not set weights to exactly zero. Compare this with the diamond-shaped L1 ball in Lasso regression.

Worked example: two sensors, one temperature

Eight days. Sensors A and B both measure the afternoon temperature (scaled units); the target is ice-cream sales (hundreds of cones). The means are xˉA=xˉB=4.5\bar x_A=\bar x_B=4.5 and yˉ=13\bar y=13.

DaySensor A (x₁)Sensor B (x₂)Sales yx₁ − 4.5x₂ − 4.5y − 13
111.25−3.5−3.3−8
221.88−2.5−2.7−5
333.38−1.5−1.2−5
443.913−0.5−0.60
554.8130.50.30
666.3171.51.84
776.9182.52.45
887.8223.53.39
Training data. The two sensors agree to within 0.3 on every day (correlation 0.996).

Ridge with λ = 2, by hand

  1. 1Gram matrix and right-hand side from the centred columns: ∑c12=42\sum c_1^2=42, ∑c1c2=40.8\sum c_1c_2=40.8, ∑c22=39.96\sum c_2^2=39.96; ∑c1yc=98\sum c_1y_c=98, ∑c2yc=94.8\sum c_2y_c=94.8. So X⊤X=(4240.840.839.96)X^\top X=\begin{pmatrix}42&40.8\\40.8&39.96\end{pmatrix}, X⊤y=(9894.8)X^\top y=\begin{pmatrix}98\\94.8\end{pmatrix}.
  2. 2OLS for comparison. det⁡(X⊤X)=42⋅39.96−40.82=1678.32−1664.64=13.68\det(X^\top X)=42\cdot39.96-40.8^2=1678.32-1664.64=13.68 — tiny next to the entries, the signature of collinearity. w1=(39.96⋅98−40.8⋅94.8)/13.68=48.24/13.68=3.526w_1=(39.96\cdot98-40.8\cdot94.8)/13.68=48.24/13.68=3.526, w2=(−40.8⋅98+42⋅94.8)/13.68=−16.8/13.68=−1.228w_2=(-40.8\cdot98+42\cdot94.8)/13.68=-16.8/13.68=-1.228.
  3. 3Add the ridge. X⊤X+2I=(4440.840.841.96)X^\top X+2I=\begin{pmatrix}44&40.8\\40.8&41.96\end{pmatrix}, with determinant 44⋅41.96−40.82=1846.24−1664.64=181.644\cdot41.96-40.8^2=1846.24-1664.64=181.6 — thirteen times larger, so the inverse is tame.
  4. 4Solve with the 2×2 inverse 1det⁡(41.96−40.8−40.844)\tfrac{1}{\det}\begin{pmatrix}41.96&-40.8\\-40.8&44\end{pmatrix}: w1=(41.96⋅98−40.8⋅94.8)/181.6=244.24/181.6=1.345w_1=(41.96\cdot98-40.8\cdot94.8)/181.6=244.24/181.6=1.345 and w2=(−40.8⋅98+44⋅94.8)/181.6=172.8/181.6=0.952w_2=(-40.8\cdot98+44\cdot94.8)/181.6=172.8/181.6=0.952. Both positive, nearly equal: the credit is shared.
  5. 5Intercept (not penalized): b=yˉ−xˉAw1−xˉBw2=13−4.5 (1.345+0.952)=2.666b=\bar y-\bar x_A w_1-\bar x_B w_2=13-4.5\,(1.345+0.952)=2.666.
  6. 6Predict a new day where the sensors disagree a little, x=(6,7)x=(6,7): ridge gives 2.666+6(1.345)+7(0.952)=17.402.666+6(1.345)+7(0.952)=17.40. OLS gives 2.658+6(3.526)−7(1.228)=15.222.658+6(3.526)-7(1.228)=15.22 — it lowers the forecast because sensor B read higher, an artefact of the negative weight.
λw₁w₂‖w‖₂train MSELOO MSEdf(λ)sd(w₁)
0 (OLS)3.526−1.2283.7340.8552.1132.0001.638
0.252.1250.2022.1340.9391.7721.3980.654
11.5130.8071.7151.0311.8111.1310.238
21.3450.9521.6481.0701.8611.0530.139
101.1020.9981.4861.4182.6250.9080.071
1000.5410.5200.7509.68516.0500.4520.035
The ridge trace in numbers. Leave-one-out MSE is lowest at λ = 0.25 on this grid; the train MSE only ever rises. "sd(w₁)" is the spread of w1w_1 over 40 refits with fresh Gaussian noise (σ = 1) around the true weights (1.15, 1.15).

Read the table top to bottom. The first small λ\lambda already kills the negative weight. By λ=2\lambda=2 the model uses about one effective parameter (df=1.053\text{df}=1.053): essentially "average the two sensors, times 2.3". The noise sensitivity of w1w_1 falls twelvefold, from 1.638 to 0.139. Past λ≈10\lambda\approx 10 the penalty starts shrinking the useful direction too, and at λ=100\lambda=100 both weights are halved and the error explodes — that is underfitting.

weight plane · loss contours · ‖w‖₂ circle-2024-2024w₁ (sensor A) →w₂OLSridge trace: w₁ · w₂ vs λ-10123LOO MSE (cap 6) · min λ = 0.25024601e-21e-111e11e21e3λ (log scale) →
w = (1.345, 0.952) · b = 2.666
det(G+λI) = 181.60 · ‖w‖₂ = 1.648
train MSE 1.070 · LOO MSE 1.861
shrink d²/(d²+λ) = 0.976, 0.077 · df = 1.053
redraws: sd(w₁) 0.139 · mean ‖ŵ − w*‖² 0.045
predict x = (6, 7): 17.40
fig.The sensors data in weight space. Violet ellipses are contours of the training loss around the OLS point (3.53, −1.23); they are long along the 'A minus B' direction because the data barely constrains it. Slide λ: the green ridge solution moves along the dashed path toward the origin, always where a loss contour just touches the green L2 circle. Pink dots are 40 refits on fresh noise around the true weights (×): at λ = 0 they smear along the valley (sd of w₁ 1.638), at λ = 2 they collapse into a tight cluster (0.139). Right: the ridge trace and leave-one-out error, which is lowest at λ = 0.25 (1.772).

The algorithm, step by step

Fit ridge regression

  1. 1Standardize each feature to mean 0 and standard deviation 1 (remember the means and scales); centre yy. The penalty is only fair if all features are on the same scale.
  2. 2Choose a grid of λ\lambda values, log-spaced (e.g. 10−3…10310^{-3}\ldots10^{3}).
  3. 3For each λ\lambda, solve (X⊤X+λI)w=X⊤y(X^\top X+\lambda I)w=X^\top y with a Cholesky factorization — or compute one SVD of XX once and get every λ\lambda cheaply from the shrinkage formula.
  4. 4Score each λ\lambda by cross-validation. For ridge, leave-one-out residuals have a closed form, ei/(1−hii)e_i/(1-h_{ii}) with hiih_{ii} the diagonal of the hat matrix H=X(X⊤X+λI)−1X⊤H=X(X^\top X+\lambda I)^{-1}X^\top (plus 1/n1/n for the intercept), so LOO costs no extra fits.
  5. 5Refit on all training data with the best λ\lambda; set b=yˉ−xˉ⊤wb=\bar y-\bar x^\top w and undo the standardization to report weights in original units.
  6. 6Predict y^=b+w⊤x\hat y=b+w^\top x for new inputs (after applying the same standardization).

Cost

Primal solve
form X⊤XX^\top X in O(np2)O(np^2), factor in O(p3)O(p^3) — best when n≫pn\gg p
Dual / kernel solve
w^=X⊤(XX⊤+λI)−1y\hat w=X^\top(XX^\top+\lambda I)^{-1}y: O(n2p+n3)O(n^2p+n^3) — best when p≫np\gg n (e.g. genomics, text); with a kernel this is kernel ridge regression
Whole λ path
one SVD O(npmin⁡(n,p))O(np\min(n,p)), then O(p⋅min⁡(n,p))O(p\cdot\min(n,p)) per λ\lambda
Huge or streaming data
gradient descent / SGD on J; the penalty adds 2λw to the gradient ("weight decay")

Tuning and variants

Knob / variantWhat it doesGuidance
λ (alpha)Strength of shrinkage; sets df(λ) between p and 0Search a log grid 1e-3…1e3 with CV; RidgeCV does efficient LOO by default
StandardizationMakes the penalty unit-freeAlways, unless all features already share units
InterceptLeft unpenalized so shifting y shifts predictionsKeep fit_intercept=True; centre data if you code it yourself
Solvercholesky, svd, sparse_cg, lsqr, sag/sagacholesky/svd for small dense data, sag/saga or sparse_cg for large or sparse
Generalized ridgePenalty ‖Γw‖², e.g. a different λ per feature or a smoothness penalty on neighbouring weightsUse when some coefficients should be shrunk more, or toward each other
Kernel ridgeSame objective in a feature space defined by a kernelNonlinear fits on small-to-medium n
Elastic NetAdds an L1 term to get some exact zerosWhen you also need feature selection — see Elastic Net regression
Practical knobs for ridge and its close relatives.

In code

python
1import numpy as np
2
3class RidgeScratch:
4 def __init__(self, lam=1.0):
5 self.lam = lam
6
7 def fit(self, X, y):
8 X, y = np.asarray(X, float), np.asarray(y, float)
9 self.x_mean, self.y_mean = X.mean(0), y.mean()
10 Xc, yc = X - self.x_mean, y - self.y_mean # centre: intercept is not penalised
11 d = X.shape[1]
12 A = Xc.T @ Xc + self.lam * np.eye(d) # X^T X + lambda I (always invertible for lam > 0)
13 self.w = np.linalg.solve(A, Xc.T @ yc)
14 self.b = self.y_mean - self.x_mean @ self.w
15 # leave-one-out residuals in closed form: e_i / (1 - h_ii)
16 H = Xc @ np.linalg.solve(A, Xc.T) + 1 / len(y)
17 e = y - self.predict(X)
18 self.loo_mse = np.mean((e / (1 - np.diag(H))) ** 2)
19 # SVD view: shrinkage factor per principal direction, effective degrees of freedom
20 s = np.linalg.svd(Xc, compute_uv=False)
21 self.shrink = s**2 / (s**2 + self.lam)
22 self.df = self.shrink.sum()
23 return self
24
25 def predict(self, X):
26 return np.asarray(X, float) @ self.w + self.b
27
28X = np.array([[1, 1.2], [2, 1.8], [3, 3.3], [4, 3.9], [5, 4.8], [6, 6.3], [7, 6.9], [8, 7.8]])
29y = np.array([5, 8, 8, 13, 13, 17, 18, 22])
30
31for lam in [0, 2]:
32 m = RidgeScratch(lam).fit(X, y)
33 print(f"lam={lam}: w={m.w.round(4)} b={m.b:.4f} LOO-MSE={m.loo_mse:.4f} "
34 f"df={m.df:.4f} shrink={m.shrink.round(4)} predict(6,7)={m.predict([[6, 7]])[0]:.3f}")
35
36# lam=0: w=[ 3.5263 -1.2281] b=2.6579 LOO-MSE=2.1134 df=2.0000 shrink=[1. 1.] predict(6,7)=15.219
37# lam=2: w=[1.3449 0.9515] b=2.6659 LOO-MSE=1.8612 df=1.0533 shrink=[0.9761 0.0772] predict(6,7)=17.396
From scratch with NumPy: closed form, unpenalized intercept, closed-form leave-one-out error and the SVD shrinkage factors.
python
1import numpy as np
2from sklearn.linear_model import LinearRegression, Ridge, RidgeCV
3from sklearn.pipeline import make_pipeline
4from sklearn.preprocessing import StandardScaler
5
6X = np.array([[1, 1.2], [2, 1.8], [3, 3.3], [4, 3.9], [5, 4.8], [6, 6.3], [7, 6.9], [8, 7.8]])
7y = np.array([5, 8, 8, 13, 13, 17, 18, 22])
8x_new = [[6, 7]]
9
10ols = LinearRegression().fit(X, y)
11ridge = Ridge(alpha=2).fit(X, y) # sklearn's alpha is the article's lambda
12print("OLS ", ols.coef_.round(4), round(ols.intercept_, 4), ols.predict(x_new).round(3))
13print("Ridge", ridge.coef_.round(4), round(ridge.intercept_, 4), ridge.predict(x_new).round(3))
14
15# choose lambda by efficient leave-one-out CV over a log grid
16alphas = [m * 10.0**e for e in range(-2, 3) for m in (1, 1.25, 1.6, 2, 2.5, 3.15, 4, 5, 6.3, 8)]
17cv = RidgeCV(alphas=alphas, scoring="neg_mean_squared_error").fit(X, y)
18print("RidgeCV alpha =", cv.alpha_, "LOO MSE =", round(-cv.best_score_, 4), "coef", cv.coef_.round(4))
19
20# in real work: standardise first, so the penalty treats every feature equally
21model = make_pipeline(StandardScaler(), RidgeCV(alphas=np.logspace(-3, 3, 61))).fit(X, y)
22print("pipeline alpha =", round(model[-1].alpha_, 4), "predict", model.predict(x_new).round(3))
23
24# OLS [ 3.5263 -1.2281] 2.6579 [15.219]
25# Ridge [1.3449 0.9515] 2.6659 [17.396]
26# RidgeCV alpha = 0.25 LOO MSE = 1.7725 coef [2.1249 0.2016]
27# pipeline alpha = 0.0501 predict [16.725]
scikit-learn on the same data: OLS vs Ridge, then RidgeCV choosing λ by efficient leave-one-out.

The best alpha after standardizing (0.0501) is far smaller than on raw data (0.25), because standardized columns have smaller sums of squares — the same amount of shrinkage needs a smaller number. One more reason not to copy λ values between setups.

Use when

  • +Features are correlated or numerous (even p > n) and you want stable, well-defined weights.
  • +You believe many features each contribute a little, rather than a few dominating.
  • +You want a fast, closed-form, convex baseline with one hyperparameter and cheap leave-one-out CV.
  • +You need smooth predictions that change little when the training data changes.

Avoid when

  • −You need a sparse model: ridge keeps every feature with a non-zero weight — use Lasso or Elastic Net.
  • −The relationship is strongly nonlinear (use kernel ridge, trees or boosting).
  • −You need unbiased coefficient estimates for inference; ridge weights are deliberately biased toward zero.
  • −Features cannot be put on a common scale meaningfully, so a single λ has no sensible interpretation.

Where it's used

Genomics and polygenic scores

Tens of thousands of correlated genetic markers, a few thousand people: ridge (GBLUP in animal breeding) spreads small effects across many markers.

Economics and marketing mix

Ad spends across channels move together; ridge keeps channel effects from flipping sign when budgets are collinear.

Chemometrics / spectroscopy

Hundreds of adjacent wavelengths are nearly collinear; ridge (and its cousin PLS) predicts concentrations stably.

Weight decay in neural networks

The same L2 penalty, applied by every deep-learning optimizer as weight decay, and the closed-form readout in kernel and random-feature models.

Check yourself

A principal direction of the centred data has dj2=8d_j^2=8. With λ=2\lambda=2, by what factor does ridge multiply the OLS coefficient along that direction?

Check yourself

You duplicate a feature, so the design has two identical columns. What does ridge (λ > 0) do with them?

Check yourself

As λ → ∞ (intercept not penalized), ridge predictions tend to…

Check yourself

Why does adding λI fix collinearity?

Remember

  • ✓Ridge minimizes ∥y−Xw∥2+λ∥w∥22\|y-Xw\|^2+\lambda\|w\|_2^2; the closed form is w^=(X⊤X+λI)−1X⊤y\hat w=(X^\top X+\lambda I)^{-1}X^\top y, always solvable for λ>0\lambda>0.
  • ✓In principal directions it multiplies each OLS component by dj2/(dj2+λ)d_j^2/(d_j^2+\lambda): low-variance directions — where collinearity lives — are shrunk most.
  • ✓Correlated features share credit instead of getting huge opposite-sign weights; weights shrink smoothly but never to exactly zero.
  • ✓Standardize features and leave the intercept unpenalized; choose λ by cross-validation (leave-one-out is free for ridge), never by training error.
  • ✓Equivalent views: an L2-ball constraint, a Gaussian prior with λ=σ2/τ2\lambda=\sigma^2/\tau^2, and weight decay in gradient descent.
← back to ML Algorithms roadmap