# Ridge regression

> Add λ‖w‖₂² to the squared error; weights shrink toward zero, which tames collinearity and lowers variance.

Source: https://www.mlfoundry.in/learn/ridge-regression  
Section: ML Algorithms › Supervised · Regression  
Level: core · 16 min read · updated 2026-10-07

**Ridge regression:** is linear least squares with a price on large weights: it minimizes the sum of squared errors **plus** $\lambda$ times the sum of squared coefficients. The penalty pulls every weight toward zero — smoothly, never exactly to zero — and in exchange the solution always exists, is unique, and changes much less when the data changes.

- **Task:** Regression (numeric target); the same penalty is used in classifiers as L2 regularization
- **Input → output:** feature vector $x\in\mathbb{R}^p$ → $\hat y = b + w^\top x$
- **Learns:** a weight $w_j$ per feature and an unpenalized intercept $b$
- **Training:** closed form: one $p\times p$ solve, $O(np^2+p^3)$; or $O(n^2p)$ in the kernel/dual form when $p\gg n$
- **Prediction:** $O(p)$ — one dot product
- **Hyperparameters:** penalty strength $\lambda$ (scikit-learn: alpha); whether to standardize; whether to fit the intercept
- **Assumes:** a linear relationship, features on comparable scales; Gaussian noise if you want the Bayesian reading

## The problem

Ordinary least squares (see **Multiple linear regression**) solves $\hat w=(X^\top X)^{-1}X^\top y$. That formula is unbiased, but it has a weak spot: when two features carry nearly the same information, $X^\top X$ is almost singular, its inverse is enormous, and the weights become wild. OLS can no longer tell *which* of the two features deserves the credit, so it gives one a huge positive weight and the other a compensating negative one — any split fits the training data about equally well.

Our running example has exactly this disease. Two temperature sensors, A and B, read almost the same value each day (correlation 0.996), and we predict ice-cream sales from both. OLS returns $w=(3.526,\,-1.228)$: *hotter on sensor B means fewer sales*, which is nonsense. Worse, the fit is fragile — add 1 to a single sales figure (day 3) and OLS jumps to $w=(2.724,\,-0.439)$; the first weight moves by 0.80 because of one unit of noise.

> **Variance, not bias, is the issue:** OLS is unbiased: averaged over many noisy datasets its weights are right. But each single estimate can be far off, because the variance of $\hat w$ is $\sigma^2(X^\top X)^{-1}$ and that matrix explodes along directions where the data barely varies. Ridge accepts a little bias to cut that variance a lot. The general bias–variance argument is in **L1 & L2 regularization**; here we focus on what the squared (L2) penalty does specifically.

## The idea

Keep the squared error, but add a charge for every unit of squared weight. A wild solution like $(3.5,-1.2)$ has $\|w\|_2^2=13.9$; a sensible one like $(1.3,1.0)$ has $\|w\|_2^2=2.7$ and fits almost as well. Once weight costs something, the model prefers the cheap solution that **shares** credit between correlated features instead of playing them against each other.

Geometrically, the loss surface of OLS has a long, flat valley along the direction where the two sensors disagree — the data says almost nothing about that direction. The penalty is a round bowl centred at the origin. Adding the bowl lifts the flat valley floor, so the minimum is pinned down in the directions the data cannot determine, while the directions it *can* determine are barely touched.

**Vocabulary**

- **Penalty / regularizer:** the extra term $\lambda\|w\|_2^2=\lambda\sum_j w_j^2$ added to the loss
- **λ (alpha):** penalty strength. $\lambda=0$ is OLS; $\lambda\to\infty$ forces $w\to 0$ and predicts $\bar y$
- **Shrinkage:** pulling estimated weights toward zero relative to OLS
- **Collinearity:** features that are (nearly) linear combinations of each other, making $X^\top X$ (nearly) singular
- **Condition number:** ratio of the largest to smallest eigenvalue of $X^\top X$; large means unstable OLS (489 in our example)
- **Ridge trace:** plot of each weight $w_j(\lambda)$ against $\lambda$
- **Effective degrees of freedom:** $\text{df}(\lambda)=\operatorname{tr}\,X(X^\top X+\lambda I)^{-1}X^\top$: how many parameters the fit is really using, from $p$ down to 0

## The math

Centre every feature and the target first (subtract their means). Then the intercept separates out as $b=\bar y-\bar x^\top w$, and the penalty applies only to $w$.

$$
J(w)=\underbrace{\|y-Xw\|_2^2}_{\text{squared error}}+\underbrace{\lambda\|w\|_2^2}_{\text{penalty}}=\sum_{i=1}^n\big(y_i-x_i^\top w\big)^2+\lambda\sum_{j=1}^p w_j^2
$$

- $X$: $n\times p$ matrix of centred features (row $i$ is $x_i^\top$)
- $y$: length-n vector of centred targets
- $w$: length-p weight vector (intercept excluded)
- $\lambda\ge 0$: penalty strength
- $\|w\|_2^2$: $w^\top w$, the squared Euclidean length of $w$

**Deriving the closed form.** $J$ is a sum of two convex quadratics, so it has a single minimum where the gradient vanishes. Expand $J=y^\top y-2w^\top X^\top y+w^\top X^\top Xw+\lambda w^\top w$ and differentiate:

$$
\nabla_w J=-2X^\top y+2X^\top Xw+2\lambda w=0\;\;\Longrightarrow\;\;(X^\top X+\lambda I)\,\hat w=X^\top y\;\;\Longrightarrow\;\;\hat w_\lambda=(X^\top X+\lambda I)^{-1}X^\top y
$$

- $I$: $p\times p$ identity matrix
- $\hat w_\lambda$: ridge solution for this $\lambda$

The only change from OLS is $+\lambda I$ on the diagonal — the "ridge" that gives the method its name. $X^\top X$ is positive semi-definite, so its eigenvalues are $\ge 0$; adding $\lambda I$ raises every one of them by $\lambda$. For any $\lambda>0$ the matrix is therefore **strictly positive definite and always invertible**, even when features are perfectly collinear or when $p>n$.

**What the shrinkage actually does.** Write the centred data's singular value decomposition $X=UDV^\top$ (columns $v_j$ of $V$ are the principal directions, $d_j$ the singular values; $d_j^2$ are the eigenvalues of $X^\top X$). Substituting into the closed form gives

$$
\hat w_\lambda=\sum_{j=1}^{p} v_j\,\frac{d_j^2}{d_j^2+\lambda}\cdot\frac{u_j^\top y}{d_j},\qquad X\hat w_\lambda=\sum_{j=1}^{p}u_j\,\frac{d_j^2}{d_j^2+\lambda}\,u_j^\top y
$$

- $v_j,\;u_j$: $j$-th right / left singular vectors of $X$
- $d_j$: $j$-th singular value; $d_j^2$ is the data's spread along $v_j$ (times $n$)
- $\tfrac{u_j^\top y}{d_j}$: the OLS coefficient along $v_j$
- $\tfrac{d_j^2}{d_j^2+\lambda}$: **shrinkage factor**, between 0 and 1

So ridge takes the OLS solution, rotates into principal directions, and multiplies each component by $d_j^2/(d_j^2+\lambda)$. Directions with large spread ($d_j^2\gg\lambda$) keep almost all of their coefficient; directions with tiny spread ($d_j^2\ll\lambda$) are squashed toward zero. These low-spread directions are exactly where OLS was guessing. In our sensors data $d_1^2=81.79$ (the "average temperature" direction) and $d_2^2=0.167$ (the "sensor A minus sensor B" direction); at $\lambda=2$ the factors are $0.976$ and $0.077$.

$$
\text{df}(\lambda)=\operatorname{tr}\!\big[X(X^\top X+\lambda I)^{-1}X^\top\big]=\sum_{j=1}^p\frac{d_j^2}{d_j^2+\lambda}
$$

- $\text{df}(\lambda)$: effective degrees of freedom: $p$ at $\lambda=0$, falling smoothly to 0 as $\lambda\to\infty$
- $\operatorname{tr}$: trace (sum of the diagonal)

**Effect of $\lambda$.** Raising $\lambda$ (1) shrinks every coordinate along $v_j$ by a larger factor, so $\|\hat w_\lambda\|_2$ decreases monotonically; (2) increases bias — $E[\hat w_\lambda]=(X^\top X+\lambda I)^{-1}X^\top X\,w^*\ne w^*$; (3) decreases variance — $\operatorname{Var}(\hat w_\lambda)=\sigma^2(X^\top X+\lambda I)^{-1}X^\top X(X^\top X+\lambda I)^{-1}$, whose eigenvalues $\sigma^2 d_j^2/(d_j^2+\lambda)^2$ are always smaller than OLS's $\sigma^2/d_j^2$. A classical result (Hoerl & Kennard, 1970) is that *some* $\lambda>0$ always has lower total mean squared error of $\hat w$ than OLS.

$$
\min_w \|y-Xw\|_2^2\;\;\text{subject to}\;\;\|w\|_2^2\le t
$$

- $t$: a weight budget; each $\lambda$ corresponds to one $t$ (larger $\lambda$ ↔ smaller $t$)

This constrained form is the same problem (by Lagrange duality): the solution is where the smallest loss ellipse touches the round ball $\|w\|_2\le\sqrt t$. Because the ball has no corners, the touching point almost never lies on an axis — **ridge does not set weights to exactly zero**. Compare this with the diamond-shaped L1 ball in **Lasso regression**.

> **Ridge is a Gaussian prior:** If $y\mid w\sim\mathcal N(Xw,\sigma^2I)$ and each weight has prior $w_j\sim\mathcal N(0,\tau^2)$, the posterior mode (MAP) is exactly the ridge solution with $\lambda=\sigma^2/\tau^2$. A strong belief that weights are small (small $\tau$) is a large $\lambda$. **Bayesian regression** keeps the whole posterior instead of only its peak, and can even learn $\lambda$ from the data.

## Worked example: two sensors, one temperature

Eight days. Sensors A and B both measure the afternoon temperature (scaled units); the target is ice-cream sales (hundreds of cones). The means are $\bar x_A=\bar x_B=4.5$ and $\bar y=13$.

Training data. The two sensors agree to within 0.3 on every day (correlation 0.996).

| Day | Sensor A (x₁) | Sensor B (x₂) | Sales y | x₁ − 4.5 | x₂ − 4.5 | y − 13 |
| --- | --- | --- | --- | --- | --- | --- |
| 1 | 1 | 1.2 | 5 | −3.5 | −3.3 | −8 |
| 2 | 2 | 1.8 | 8 | −2.5 | −2.7 | −5 |
| 3 | 3 | 3.3 | 8 | −1.5 | −1.2 | −5 |
| 4 | 4 | 3.9 | 13 | −0.5 | −0.6 | 0 |
| 5 | 5 | 4.8 | 13 | 0.5 | 0.3 | 0 |
| 6 | 6 | 6.3 | 17 | 1.5 | 1.8 | 4 |
| 7 | 7 | 6.9 | 18 | 2.5 | 2.4 | 5 |
| 8 | 8 | 7.8 | 22 | 3.5 | 3.3 | 9 |

**Ridge with λ = 2, by hand**

1. **Gram matrix and right-hand side** from the centred columns: $\sum c_1^2=42$, $\sum c_1c_2=40.8$, $\sum c_2^2=39.96$; $\sum c_1y_c=98$, $\sum c_2y_c=94.8$. So $X^\top X=\begin{pmatrix}42&40.8\\40.8&39.96\end{pmatrix}$, $X^\top y=\begin{pmatrix}98\\94.8\end{pmatrix}$.
2. **OLS for comparison.** $\det(X^\top X)=42\cdot39.96-40.8^2=1678.32-1664.64=13.68$ — tiny next to the entries, the signature of collinearity. $w_1=(39.96\cdot98-40.8\cdot94.8)/13.68=48.24/13.68=3.526$, $w_2=(-40.8\cdot98+42\cdot94.8)/13.68=-16.8/13.68=-1.228$.
3. **Add the ridge.** $X^\top X+2I=\begin{pmatrix}44&40.8\\40.8&41.96\end{pmatrix}$, with determinant $44\cdot41.96-40.8^2=1846.24-1664.64=181.6$ — thirteen times larger, so the inverse is tame.
4. **Solve** with the 2×2 inverse $\tfrac{1}{\det}\begin{pmatrix}41.96&-40.8\\-40.8&44\end{pmatrix}$: $w_1=(41.96\cdot98-40.8\cdot94.8)/181.6=244.24/181.6=1.345$ and $w_2=(-40.8\cdot98+44\cdot94.8)/181.6=172.8/181.6=0.952$. Both positive, nearly equal: the credit is shared.
5. **Intercept** (not penalized): $b=\bar y-\bar x_A w_1-\bar x_B w_2=13-4.5\,(1.345+0.952)=2.666$.
6. **Predict** a new day where the sensors disagree a little, $x=(6,7)$: ridge gives $2.666+6(1.345)+7(0.952)=17.40$. OLS gives $2.658+6(3.526)-7(1.228)=15.22$ — it *lowers* the forecast because sensor B read higher, an artefact of the negative weight.

The ridge trace in numbers. Leave-one-out MSE is lowest at λ = 0.25 on this grid; the train MSE only ever rises. "sd(w₁)" is the spread of $w_1$ over 40 refits with fresh Gaussian noise (σ = 1) around the true weights (1.15, 1.15).

| λ | w₁ | w₂ | ‖w‖₂ | train MSE | LOO MSE | df(λ) | sd(w₁) |
| --- | --- | --- | --- | --- | --- | --- | --- |
| 0 (OLS) | 3.526 | −1.228 | 3.734 | 0.855 | 2.113 | 2.000 | 1.638 |
| 0.25 | 2.125 | 0.202 | 2.134 | 0.939 | **1.772** | 1.398 | 0.654 |
| 1 | 1.513 | 0.807 | 1.715 | 1.031 | 1.811 | 1.131 | 0.238 |
| **2** | **1.345** | **0.952** | 1.648 | 1.070 | 1.861 | 1.053 | 0.139 |
| 10 | 1.102 | 0.998 | 1.486 | 1.418 | 2.625 | 0.908 | 0.071 |
| 100 | 0.541 | 0.520 | 0.750 | 9.685 | 16.050 | 0.452 | 0.035 |

Read the table top to bottom. The first small $\lambda$ already kills the negative weight. By $\lambda=2$ the model uses about one effective parameter ($\text{df}=1.053$): essentially "average the two sensors, times 2.3". The noise sensitivity of $w_1$ falls twelvefold, from 1.638 to 0.139. Past $\lambda\approx 10$ the penalty starts shrinking the *useful* direction too, and at $\lambda=100$ both weights are halved and the error explodes — that is underfitting.

> Figure: The sensors data in weight space. Violet ellipses are contours of the training loss around the OLS point (3.53, −1.23); they are long along the 'A minus B' direction because the data barely constrains it. Slide λ: the green ridge solution moves along the dashed path toward the origin, always where a loss contour just touches the green L2 circle. Pink dots are 40 refits on fresh noise around the true weights (×): at λ = 0 they smear along the valley (sd of w₁ 1.638), at λ = 2 they collapse into a tight cluster (0.139). Right: the ridge trace and leave-one-out error, which is lowest at λ = 0.25 (1.772).

> **Shrinkage is uneven by design:** Watch the green point in the figure as $\lambda$ grows from 0 to 2: it travels a long way along the valley (the "A minus B" component is multiplied by 0.077) but hardly moves toward the origin along the "average" direction (factor 0.976). Ridge is not a uniform shrink; it removes the components the data cannot support and keeps the rest.

## The algorithm, step by step

**Fit ridge regression**

1. **Standardize** each feature to mean 0 and standard deviation 1 (remember the means and scales); centre $y$. The penalty is only fair if all features are on the same scale.
2. **Choose a grid** of $\lambda$ values, log-spaced (e.g. $10^{-3}\ldots10^{3}$).
3. **For each $\lambda$**, solve $(X^\top X+\lambda I)w=X^\top y$ with a Cholesky factorization — or compute one SVD of $X$ once and get every $\lambda$ cheaply from the shrinkage formula.
4. **Score** each $\lambda$ by cross-validation. For ridge, leave-one-out residuals have a closed form, $e_i/(1-h_{ii})$ with $h_{ii}$ the diagonal of the hat matrix $H=X(X^\top X+\lambda I)^{-1}X^\top$ (plus $1/n$ for the intercept), so LOO costs no extra fits.
5. **Refit** on all training data with the best $\lambda$; set $b=\bar y-\bar x^\top w$ and undo the standardization to report weights in original units.
6. **Predict** $\hat y=b+w^\top x$ for new inputs (after applying the same standardization).

**Cost**

- **Primal solve:** form $X^\top X$ in $O(np^2)$, factor in $O(p^3)$ — best when $n\gg p$
- **Dual / kernel solve:** $\hat w=X^\top(XX^\top+\lambda I)^{-1}y$: $O(n^2p+n^3)$ — best when $p\gg n$ (e.g. genomics, text); with a kernel this is **kernel ridge regression**
- **Whole λ path:** one SVD $O(np\min(n,p))$, then $O(p\cdot\min(n,p))$ per $\lambda$
- **Huge or streaming data:** gradient descent / SGD on J; the penalty adds 2λw to the gradient ("weight decay")

## Tuning and variants

Practical knobs for ridge and its close relatives.

| Knob / variant | What it does | Guidance |
| --- | --- | --- |
| λ (alpha) | Strength of shrinkage; sets df(λ) between p and 0 | Search a log grid 1e-3…1e3 with CV; RidgeCV does efficient LOO by default |
| Standardization | Makes the penalty unit-free | Always, unless all features already share units |
| Intercept | Left unpenalized so shifting y shifts predictions | Keep fit_intercept=True; centre data if you code it yourself |
| Solver | cholesky, svd, sparse_cg, lsqr, sag/saga | cholesky/svd for small dense data, sag/saga or sparse_cg for large or sparse |
| Generalized ridge | Penalty ‖Γw‖², e.g. a different λ per feature or a smoothness penalty on neighbouring weights | Use when some coefficients should be shrunk more, or toward each other |
| Kernel ridge | Same objective in a feature space defined by a kernel | Nonlinear fits on small-to-medium n |
| Elastic Net | Adds an L1 term to get some exact zeros | When you also need feature selection — see Elastic Net regression |

> **Forgetting to standardize:** The penalty $\lambda\sum w_j^2$ compares weights across features, so their units matter. If sensor B were recorded in tenths of a unit (values 10× larger), its weight would only need to be a tenth as large, and therefore costs a hundredth as much penalty. Refitting the example that way at $\lambda=2$ gives $w=(0.531,\,0.183)$ — in original units sensor B now carries 1.83 against A's 0.53, and the prediction for $x=(6,7)$ moves from 17.40 to 18.37. Same data, different model, just from a change of units. Standardize first.

> **Choosing λ on the test set, or by train error:** Training error always increases with $\lambda$ (0.855 → 1.070 → 9.685 in our table), so it can never pick $\lambda$. Choose it by cross-validation on the training data, inside the same pipeline as the scaler, and touch the test set once at the end.

> **λ is not comparable across libraries:** scikit-learn's Ridge minimizes ‖y − Xw‖² + alpha‖w‖² (no 1/n), while glmnet and many deep-learning optimizers scale the loss by 1/n or 1/(2n). The same number means a different penalty strength; translate before you copy a value.

## In code

```python
import numpy as np

class RidgeScratch:
    def __init__(self, lam=1.0):
        self.lam = lam

    def fit(self, X, y):
        X, y = np.asarray(X, float), np.asarray(y, float)
        self.x_mean, self.y_mean = X.mean(0), y.mean()
        Xc, yc = X - self.x_mean, y - self.y_mean          # centre: intercept is not penalised
        d = X.shape[1]
        A = Xc.T @ Xc + self.lam * np.eye(d)               # X^T X + lambda I  (always invertible for lam > 0)
        self.w = np.linalg.solve(A, Xc.T @ yc)
        self.b = self.y_mean - self.x_mean @ self.w
        # leave-one-out residuals in closed form: e_i / (1 - h_ii)
        H = Xc @ np.linalg.solve(A, Xc.T) + 1 / len(y)
        e = y - self.predict(X)
        self.loo_mse = np.mean((e / (1 - np.diag(H))) ** 2)
        # SVD view: shrinkage factor per principal direction, effective degrees of freedom
        s = np.linalg.svd(Xc, compute_uv=False)
        self.shrink = s**2 / (s**2 + self.lam)
        self.df = self.shrink.sum()
        return self

    def predict(self, X):
        return np.asarray(X, float) @ self.w + self.b

X = np.array([[1, 1.2], [2, 1.8], [3, 3.3], [4, 3.9], [5, 4.8], [6, 6.3], [7, 6.9], [8, 7.8]])
y = np.array([5, 8, 8, 13, 13, 17, 18, 22])

for lam in [0, 2]:
    m = RidgeScratch(lam).fit(X, y)
    print(f"lam={lam}: w={m.w.round(4)} b={m.b:.4f} LOO-MSE={m.loo_mse:.4f} "
          f"df={m.df:.4f} shrink={m.shrink.round(4)} predict(6,7)={m.predict([[6, 7]])[0]:.3f}")

# lam=0: w=[ 3.5263 -1.2281] b=2.6579 LOO-MSE=2.1134 df=2.0000 shrink=[1. 1.] predict(6,7)=15.219
# lam=2: w=[1.3449 0.9515] b=2.6659 LOO-MSE=1.8612 df=1.0533 shrink=[0.9761 0.0772] predict(6,7)=17.396
```

From scratch with NumPy: closed form, unpenalized intercept, closed-form leave-one-out error and the SVD shrinkage factors.

```python
import numpy as np
from sklearn.linear_model import LinearRegression, Ridge, RidgeCV
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X = np.array([[1, 1.2], [2, 1.8], [3, 3.3], [4, 3.9], [5, 4.8], [6, 6.3], [7, 6.9], [8, 7.8]])
y = np.array([5, 8, 8, 13, 13, 17, 18, 22])
x_new = [[6, 7]]

ols = LinearRegression().fit(X, y)
ridge = Ridge(alpha=2).fit(X, y)            # sklearn's alpha is the article's lambda
print("OLS  ", ols.coef_.round(4), round(ols.intercept_, 4), ols.predict(x_new).round(3))
print("Ridge", ridge.coef_.round(4), round(ridge.intercept_, 4), ridge.predict(x_new).round(3))

# choose lambda by efficient leave-one-out CV over a log grid
alphas = [m * 10.0**e for e in range(-2, 3) for m in (1, 1.25, 1.6, 2, 2.5, 3.15, 4, 5, 6.3, 8)]
cv = RidgeCV(alphas=alphas, scoring="neg_mean_squared_error").fit(X, y)
print("RidgeCV alpha =", cv.alpha_, "LOO MSE =", round(-cv.best_score_, 4), "coef", cv.coef_.round(4))

# in real work: standardise first, so the penalty treats every feature equally
model = make_pipeline(StandardScaler(), RidgeCV(alphas=np.logspace(-3, 3, 61))).fit(X, y)
print("pipeline alpha =", round(model[-1].alpha_, 4), "predict", model.predict(x_new).round(3))

# OLS   [ 3.5263 -1.2281] 2.6579 [15.219]
# Ridge [1.3449 0.9515] 2.6659 [17.396]
# RidgeCV alpha = 0.25 LOO MSE = 1.7725 coef [2.1249 0.2016]
# pipeline alpha = 0.0501 predict [16.725]
```

scikit-learn on the same data: OLS vs Ridge, then RidgeCV choosing λ by efficient leave-one-out.

The best alpha after standardizing (0.0501) is far smaller than on raw data (0.25), because standardized columns have smaller sums of squares — the same amount of shrinkage needs a smaller number. One more reason not to copy λ values between setups.

**Strengths**

- Features are correlated or numerous (even p > n) and you want stable, well-defined weights.
- You believe many features each contribute a little, rather than a few dominating.
- You want a fast, closed-form, convex baseline with one hyperparameter and cheap leave-one-out CV.
- You need smooth predictions that change little when the training data changes.

**Limitations**

- You need a sparse model: ridge keeps every feature with a non-zero weight — use Lasso or Elastic Net.
- The relationship is strongly nonlinear (use kernel ridge, trees or boosting).
- You need unbiased coefficient estimates for inference; ridge weights are deliberately biased toward zero.
- Features cannot be put on a common scale meaningfully, so a single λ has no sensible interpretation.

- **Genomics and polygenic scores:** Tens of thousands of correlated genetic markers, a few thousand people: ridge (GBLUP in animal breeding) spreads small effects across many markers.
- **Economics and marketing mix:** Ad spends across channels move together; ridge keeps channel effects from flipping sign when budgets are collinear.
- **Chemometrics / spectroscopy:** Hundreds of adjacent wavelengths are nearly collinear; ridge (and its cousin PLS) predicts concentrations stably.
- **Weight decay in neural networks:** The same L2 penalty, applied by every deep-learning optimizer as weight decay, and the closed-form readout in kernel and random-feature models.

**Question:** A principal direction of the centred data has $d_j^2=8$. With $\lambda=2$, by what factor does ridge multiply the OLS coefficient along that direction?

- A. 0.2
- B. 0.25
- C. 0.8
- D. 1 — ridge only shrinks small directions

**Answer:** C. 0.8 — The shrinkage factor is $d_j^2/(d_j^2+\lambda)=8/10=0.8$. Every direction is shrunk; small-spread ones much more.

**Question:** You duplicate a feature, so the design has two identical columns. What does ridge (λ > 0) do with them?

- A. Fails, because XᵀX is singular
- B. Puts all the weight on the first copy
- C. Gives both copies exactly the same weight
- D. Sets one copy to exactly zero

**Answer:** C. Gives both copies exactly the same weight — Adding $\lambda I$ makes the matrix invertible, and for a fixed total $w_1+w_2$ the penalty $w_1^2+w_2^2$ is smallest when they are equal. On the sales data with sensor A duplicated, Ridge(alpha=2) gives 1.1395 to each copy — exactly half of what a single copy gets with alpha = 1. Setting one copy to zero is what Lasso tends to do.

**Question:** As λ → ∞ (intercept not penalized), ridge predictions tend to…

- A. 0
- B. the mean of the training targets
- C. the OLS predictions
- D. they diverge

**Answer:** B. the mean of the training targets — All weights are forced to 0, leaving $\hat y=b=\bar y$. On the worked example, Ridge(alpha=1e9) predicts 13.000 — the mean sales — for every day.

**Question:** Why does adding λI fix collinearity?

- A. It removes the correlated features
- B. It raises every eigenvalue of XᵀX by λ, so none is near zero and the inverse is bounded
- C. It makes the features orthogonal
- D. It increases the number of training points

**Answer:** B. It raises every eigenvalue of XᵀX by λ, so none is near zero and the inverse is bounded — Eigenvalues of $X^\top X+\lambda I$ are $d_j^2+\lambda\ge\lambda$. In the example the smallest eigenvalue goes from 0.167 to 2.167 and the determinant from 13.68 to 181.6.

**Key points**

- Ridge minimizes $\|y-Xw\|^2+\lambda\|w\|_2^2$; the closed form is $\hat w=(X^\top X+\lambda I)^{-1}X^\top y$, always solvable for $\lambda>0$.
- In principal directions it multiplies each OLS component by $d_j^2/(d_j^2+\lambda)$: low-variance directions — where collinearity lives — are shrunk most.
- Correlated features share credit instead of getting huge opposite-sign weights; weights shrink smoothly but never to exactly zero.
- Standardize features and leave the intercept unpenalized; choose λ by cross-validation (leave-one-out is free for ridge), never by training error.
- Equivalent views: an L2-ball constraint, a Gaussian prior with $\lambda=\sigma^2/\tau^2$, and weight decay in gradient descent.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
