# The Bias–Variance Trade-off

> Expected test error = bias² + variance + irreducible noise; model complexity trades the first two.

Source: https://www.mlfoundry.in/learn/bias-variance-trade-off  
Section: ML Algorithms › Evaluation & Generalization  
Level: core · 20 min read · updated 2026-10-08

**The bias–variance decomposition:** splits the expected squared error of a learning algorithm at a point $x$, averaged over random training sets and over the noise, into three parts: **bias²** (how far the *average* fitted model is from the truth), **variance** (how much the fitted model moves from one training set to the next) and **irreducible noise** $\sigma^2$ (what no model can predict).

- **Applies to:** squared-error regression exactly; classification losses have analogous but messier decompositions
- **Random over:** the training set $\mathcal{D}$ (and the test noise), with the test input $x$ held fixed
- **Bias²:** $\big(\mathbb{E}_\mathcal{D}[\hat f(x)]-f(x)\big)^2$: systematic error, *underfitting*
- **Variance:** $\mathrm{Var}_\mathcal{D}[\hat f(x)]$: sensitivity to the sample, *overfitting*
- **Noise:** $\sigma^2=\mathrm{Var}[y\mid x]$: a floor no model beats
- **Knobs:** polynomial degree, k in kNN, tree depth, regularization λ, number of features, ensemble size, training-set size n
- **Diagnosed by:** training vs validation error, validation curves, learning curves

## The intuition: four dartboards

Imagine you could rerun your whole pipeline on many fresh training sets from the same source, and each time throw one dart: the trained model's prediction at a fixed test input. The bullseye is the true value $f(x)$. Two different things can go wrong.

Each throw is the prediction of a model trained on a different random training set.

| Dartboard | Low variance (tight cluster) | High variance (scattered) |
| --- | --- | --- |
| **Low bias** (centred on the bullseye) | the goal: right on average, and every time | right on average, but any single model can be far off |
| **High bias** (centred off the bullseye) | consistently wrong in the same way | wrong on average *and* unreliable |

**Bias** is the distance from the centre of the cluster to the bullseye. It comes from assumptions the model cannot escape: a straight line fitted to a sine wave misses the curve no matter how much data you give it. That is **underfitting**: training and test error are both high.

**Variance** is the spread of the cluster. A very flexible model, such as a degree-9 polynomial through 10 points, bends to follow whatever noise happened to land in *its* training set, so the curves from different sets disagree wildly. That is **overfitting**: training error is tiny, test error is large.

Making a model more flexible usually moves bias down and variance up. The trade-off is choosing the flexibility where their **sum** is smallest. The noise $\sigma^2$ adds a fixed amount on top of every model.

## Setup and notation

Data come from $y=f(x)+\varepsilon$ with $\mathbb{E}[\varepsilon]=0$, $\mathrm{Var}[\varepsilon]=\sigma^2$, noise independent of everything else. A learning algorithm maps a training set $\mathcal{D}=\{(x_i,y_i)\}_{i=1}^n$ to a function $\hat f_\mathcal{D}$. Fix a test input $x$ and draw a fresh test label $y=f(x)+\varepsilon$, independent of $\mathcal{D}$. We want the **expected prediction error** at $x$, averaged over both sources of randomness:

$$
\mathrm{Err}(x)=\mathbb{E}_{\mathcal{D},\varepsilon}\big[(y-\hat f_\mathcal{D}(x))^2\big],\qquad \bar f(x)=\mathbb{E}_\mathcal{D}\big[\hat f_\mathcal{D}(x)\big]
$$

- $f(x)$: true regression function, $\mathbb{E}[y\mid x]$
- $\hat f_\mathcal{D}(x)$: prediction of the model trained on training set $\mathcal{D}$ (random, because $\mathcal{D}$ is)
- $\bar f(x)$: the average prediction over all possible training sets: the centre of the dart cluster
- $\varepsilon$: test-label noise, mean 0, variance $\sigma^2$, independent of $\mathcal{D}$

## Deriving the decomposition

The whole proof is "add and subtract the right thing, expand the square, and show the cross terms have mean zero". We do it twice: once to peel off the noise, once to split the rest into bias and variance. Write $\hat f$ for $\hat f_\mathcal{D}(x)$.

**Step 1: separate the noise**

1. Add and subtract $f(x)$: $\;y-\hat f=\big(y-f(x)\big)+\big(f(x)-\hat f\big)=\varepsilon+\big(f(x)-\hat f\big).$
2. Square: $\;(y-\hat f)^2=\varepsilon^2+2\,\varepsilon\big(f(x)-\hat f\big)+\big(f(x)-\hat f\big)^2.$
3. Take expectations. $\mathbb{E}[\varepsilon^2]=\sigma^2$. The cross term factorizes because the test noise $\varepsilon$ is independent of the training set and hence of $\hat f$: $\;\mathbb{E}\big[\varepsilon(f(x)-\hat f)\big]=\mathbb{E}[\varepsilon]\cdot\mathbb{E}\big[f(x)-\hat f\big]=0\cdot(\dots)=0.$
4. So $\;\mathrm{Err}(x)=\sigma^2+\mathbb{E}_\mathcal{D}\big[(f(x)-\hat f)^2\big]$: noise plus the mean squared error of the estimator $\hat f$ as an estimate of the number $f(x)$.

**Step 2: split the estimator's error**

1. Add and subtract the average prediction $\bar f=\mathbb{E}_\mathcal{D}[\hat f]$: $\;f(x)-\hat f=\big(f(x)-\bar f\big)+\big(\bar f-\hat f\big).$
2. Square: $\;(f(x)-\hat f)^2=\big(f(x)-\bar f\big)^2+2\big(f(x)-\bar f\big)\big(\bar f-\hat f\big)+\big(\bar f-\hat f\big)^2.$
3. The first factor of the cross term, $f(x)-\bar f$, is a **constant** (no randomness left in it), so it comes out of the expectation: $\;2\big(f(x)-\bar f\big)\,\mathbb{E}_\mathcal{D}\big[\bar f-\hat f\big]=2\big(f(x)-\bar f\big)\big(\bar f-\bar f\big)=0.$
4. What remains: $\;\mathbb{E}_\mathcal{D}\big[(f(x)-\hat f)^2\big]=\big(\bar f-f(x)\big)^2+\mathbb{E}_\mathcal{D}\big[(\hat f-\bar f)^2\big]$, which is bias² plus variance.

$$
\mathbb{E}_{\mathcal{D},\varepsilon}\big[(y-\hat f_\mathcal{D}(x))^2\big]=\underbrace{\big(\bar f(x)-f(x)\big)^2}_{\text{bias}^2}+\underbrace{\mathbb{E}_\mathcal{D}\big[(\hat f_\mathcal{D}(x)-\bar f(x))^2\big]}_{\text{variance}}+\underbrace{\sigma^2}_{\text{noise}}
$$

- $\text{bias}^2$: squared distance from the average model to the truth; zero if the model class can represent f and the method is unbiased
- $\text{variance}$: spread of the fitted models around their own average; zero for a model that ignores the data
- $\sigma^2$: irreducible error: even the true $f$ has expected squared error $\sigma^2$

Two facts make the cross terms vanish: the **test noise is independent of the training set**, and **$\bar f$ is the mean of $\hat f$**, so deviations from it average to zero. All three terms are non-negative, so no model can beat $\sigma^2$. To get the error over the whole input distribution, average over $x$: $\;\mathbb{E}_x[\mathrm{Err}(x)]=\mathbb{E}_x[\text{bias}^2(x)]+\mathbb{E}_x[\text{var}(x)]+\sigma^2.$

> **It is the same as the MSE of any estimator:** Step 2 is the classic statistics identity $\mathrm{MSE}(\hat\theta)=\mathrm{Bias}(\hat\theta)^2+\mathrm{Var}(\hat\theta)$, applied to the "parameter" $\theta=f(x)$. Machine learning adds Step 1: a prediction is scored against a *noisy* label, not against $f(x)$ itself.

## See it: many training sets, one picture

The figure below runs the thought experiment literally. The true function is $f(x)=\sin(2\pi x)$ on $[0,1]$, with noise $\sigma=0.3$. Each training set has $n$ inputs on a fixed, equally spaced grid $x_i=i/(n-1)$; only the noise is redrawn. For each of 30 training sets the model is refitted, and every fit is drawn. The bias, variance and total on the right are **exact** (computed in closed form, see below), averaged over 201 test points.

> Figure: Left: 30 fits (violet), their average (blue, the estimate of the mean model), the true sin(2πx) (dashed) and the ±1 sd band around the exact mean model (orange). One training set and its fit are in mint. Right: bias², variance, σ² and their sum versus complexity, with the minimum ringed. Degree 1 misses the curve the same way every time; degree 9 at n = 10 passes through every point and swings wildly between them. Raise n to watch variance shrink while bias stays put, or switch to kNN, where variance is exactly σ²/k.

## Worked example: polynomial degree 1, 3 and 9

Take the setting from the figure with $n=10$ training points, $\sigma^2=0.09$, and fit least-squares polynomials of degree 1, 3 and 9. Two ways to get the numbers. **Monte Carlo:** draw 20,000 training sets, fit each one, then measure the spread and the offset of the 20,000 predictions at every test point. **Closed form:** least squares is linear in $y$, so the mean and variance of $\hat f(x)$ follow from one matrix (next section). Both agree:

$f(x)=\sin(2\pi x)$, $n=10$, $\sigma=0.3$; averaged over 201 test points in $[0,1]$. MC = 20,000 simulated training sets (seed 1); "direct" is the average of $(y_\text{new}-\hat f)^2$ with fresh test noise, computed without using the decomposition at all.

| Degree | Bias² | Variance | Noise σ² | Total (exact) | MC bias² / var | MC direct test MSE | Verdict |
| --- | --- | --- | --- | --- | --- | --- | --- |
| 1 | 0.2235 | 0.0164 | 0.09 | 0.3299 | 0.2238 / 0.0163 | 0.3299 | underfits: bias dominates |
| **3** | **0.0066** | **0.0299** | 0.09 | **0.1265** | 0.0066 / 0.0298 | 0.1264 | **best of the three** |
| 9 | < 0.0001 | 0.5874 | 0.09 | 0.6774 | 0.0001 / 0.5768 | 0.6673 | overfits: variance dominates |

Reading the table: the line has bias² 0.22, so even with infinite data its error would stay near $0.22+0.09$. The degree-9 polynomial is essentially unbiased (on average it reproduces the sine to within $10^{-4}$) yet its variance is 0.59, 6.5 times the noise. The cubic pays a small bias (0.007) for a small variance (0.03) and wins with a total of 0.127, only 0.037 above the floor $\sigma^2$. Sweeping every degree, the minimum is at degree 3.

Where the error lives: single test points (exact, n = 10). Variance is not uniform in x: high-degree fits explode near the edges of the data.

| Test point | Degree | f(x) | Mean fit f̄(x) | Bias² | Variance |
| --- | --- | --- | --- | --- | --- |
| x = 0.25 (peak) | 1 | 1.000 | 0.337 | 0.4393 | 0.0145 |
| x = 0.25 | 3 | 1.000 | 0.961 | 0.0015 | 0.0298 |
| x = 0.25 | 9 | 1.000 | 1.000 | 0.0000 | 0.0866 |
| x = 0.95 (edge) | 1 | −0.309 | −0.607 | 0.0888 | 0.0269 |
| x = 0.95 | 3 | −0.309 | −0.402 | 0.0086 | 0.0397 |
| x = 0.95 | 9 | −0.309 | −0.309 | 0.0000 | **3.9322** |

Two curiosities show up when you sweep all degrees. Degree 2 has exactly the same bias as degree 1 (0.2235), and degree 4 the same as degree 3: $\sin(2\pi x)$ is antisymmetric about $x=\tfrac12$ and the design is symmetric, so the added even term has expected coefficient 0 and only adds variance. And the training error of degree 9 is exactly 0 (10 coefficients, 10 points), while its test error is 0.68, the worst in the table.

More data shrinks variance and leaves bias alone (exact, $\sigma=0.3$).

| n | Degree 3: bias² | Degree 3: variance | Degree 3: total | Degree 9: variance | Degree 9: total |
| --- | --- | --- | --- | --- | --- |
| 10 | 0.0066 | 0.0299 | 0.1265 | 0.5874 | 0.6774 |
| 15 | 0.0056 | 0.0209 | 0.1165 | 0.0638 | 0.1538 |
| 25 | 0.0050 | 0.0132 | 0.1081 | 0.0329 | 0.1229 |
| 50 | 0.0047 | 0.0069 | 0.1016 | 0.0166 | 0.1066 |

At $n=10$ the cubic beats degree 9 by a factor of five; at $n=50$ the gap is under 5%. This is why "the right complexity" is not a property of the problem alone: it depends on how much data you have. The best degree drifts upward as $n$ grows.

## Closed forms: linear smoothers

Least squares, ridge and kNN all predict with a **fixed linear combination of the training labels**. With the inputs $x_1,\dots,x_n$ held fixed (only the noise is random), write

$$
\hat f(x_0)=\sum_{i=1}^n \ell_i(x_0)\,y_i\quad\Rightarrow\quad \bar f(x_0)=\sum_{i}\ell_i(x_0)f(x_i),\qquad \mathrm{Var}[\hat f(x_0)]=\sigma^2\sum_i \ell_i(x_0)^2
$$

- $\ell_i(x_0)$: weight the method puts on training label $y_i$ when predicting at $x_0$ (depends on the inputs only)
- $\sum_i\ell_i^2$: $=\|\ell(x_0)\|^2$; independent noise makes variances add with squared weights

The mean follows from $\mathbb{E}[y_i]=f(x_i)$ and linearity; the variance from $\mathrm{Var}[\sum_i\ell_i\varepsilon_i]=\sum_i\ell_i^2\sigma^2$ for independent noise. This is exactly how the figure and the "exact" column above are computed. Three cases are worth working out by hand.

## Least squares: variance = σ² p / n

For a model with $p$ features (here $p=d+1$ polynomial coefficients) and design matrix $\Phi\in\mathbb{R}^{n\times p}$, the fitted values at the training inputs are $\hat{\mathbf y}=H\mathbf y$ with the **hat matrix** $H=\Phi(\Phi^\top\Phi)^{-1}\Phi^\top$. $H$ is symmetric and idempotent ($H^2=H$), so the squared weights in row $i$ sum to $\sum_j H_{ij}^2=(H^2)_{ii}=H_{ii}$. Averaging over the training inputs:

$$
\frac1n\sum_{i=1}^n\mathrm{Var}[\hat f(x_i)]=\frac{\sigma^2}{n}\sum_i H_{ii}=\frac{\sigma^2}{n}\operatorname{tr}H=\sigma^2\,\frac{p}{n}
$$

- $H$: hat matrix, projection onto the column space of Φ
- $\operatorname{tr}H=p$: trace of a rank-p projection; the number of parameters

Every extra parameter costs $\sigma^2/n$ of in-sample variance, whether or not it is needed. Checked numerically for $n=10$: degree 1, 3, 9 give 0.018, 0.036, 0.090, exactly $0.09\cdot p/10$. Away from the training inputs variance is larger (0.59 rather than 0.09 for degree 9), because polynomials extrapolate badly between and beyond the outer points.

## kNN regression: variance = σ² / k

kNN regression averages the labels of the $k$ training points nearest to $x_0$, so $\ell_i=1/k$ on those neighbours $N_k(x_0)$ and 0 elsewhere:

$$
\mathrm{Err}(x_0)=\sigma^2+\Big(f(x_0)-\frac1k\sum_{i\in N_k(x_0)}f(x_i)\Big)^2+\frac{\sigma^2}{k}
$$

- $N_k(x_0)$: indices of the $k$ training inputs closest to $x_0$
- $\sigma^2/k$: variance: $k$ squared weights of $1/k^2$, times $\sigma^2$, exact for any $f$

The variance is exact and does not depend on $f$. The bias grows with $k$ because the neighbourhood reaches further from $x_0$. In 1-D with spacing $h$ and $f$ smooth, a Taylor expansion gives bias $\approx \tfrac{1}{2}f''(x_0)\cdot\overline{(x_i-x_0)^2}\approx f''(x_0)\,k^2h^2/24$ for an interior point. So the bias grows like $k^2$ while the variance falls like $1/k$. Small $k$ is the flexible end of the dial.

kNN on the same sine, $n=50$ equally spaced points, $\sigma=0.3$; exact values, confirmed by 20,000-set Monte Carlo to within 0.0003.

| k | Bias² | Variance = σ²/k | Total |
| --- | --- | --- | --- |
| 1 | 0.0007 | 0.0900 | 0.1807 |
| 3 | 0.0009 | 0.0300 | 0.1209 |
| **5** | **0.0027** | **0.0180** | **0.1107** |
| 10 | 0.0180 | 0.0090 | 0.1170 |
| 20 | 0.0659 | 0.0045 | 0.1604 |
| 50 (= n) | 0.4975 | 0.0018 | 0.5893 |

At $k=n$ every prediction is the global mean; its bias² $0.4975$ is the variance of $\sin(2\pi x)$ over the grid, close to $\tfrac12$. The minimum sits at $k\approx 5$–6 (total 0.109 at $k=6$).

## Ridge: variance falls as λ grows

Ridge regression fits $\hat w=(\Phi^\top\Phi+\lambda I)^{-1}\Phi^\top\mathbf y$. Take the SVD $\Phi=UDV^\top$ with singular values $d_j$. The fitted values are $S_\lambda\mathbf y$ with $S_\lambda=U\,\mathrm{diag}\big(\tfrac{d_j^2}{d_j^2+\lambda}\big)U^\top$: ridge shrinks the component of $\mathbf y$ along each direction $u_j$ by $s_j=d_j^2/(d_j^2+\lambda)\in(0,1]$. With $\mathbf f=(f(x_1),\dots,f(x_n))$, the in-sample terms are

$$
\overline{\text{bias}^2}=\frac1n\Big[\sum_j\Big(\frac{\lambda}{d_j^2+\lambda}\Big)^2(u_j^\top\mathbf f)^2+\|(I-UU^\top)\mathbf f\|^2\Big],\qquad \overline{\text{var}}=\frac{\sigma^2}{n}\sum_j\Big(\frac{d_j^2}{d_j^2+\lambda}\Big)^2
$$

- $d_j,\ u_j$: singular values and left singular vectors of the design matrix
- $\lambda/(d_j^2+\lambda)$: $=1-s_j$, the fraction of signal along $u_j$ that ridge throws away
- $\|(I-UU^\top)\mathbf f\|^2$: the part of f the features cannot represent at all; no λ fixes it
- $\mathrm{df}(\lambda)$: $=\sum_j s_j=\operatorname{tr}S_\lambda$, effective number of parameters

Derivation in one line each: $\mathbb{E}[S_\lambda\mathbf y]-\mathbf f=-(I-S_\lambda)\mathbf f$, and $I-S_\lambda$ scales the $u_j$ component by $1-s_j$ and keeps the part outside the column space. $\mathrm{Cov}[S_\lambda\mathbf y]=\sigma^2S_\lambda^2$, whose trace is $\sigma^2\sum_j s_j^2$. As $\lambda$ goes from 0 to $\infty$, every $s_j$ falls from 1 to 0: **variance decreases monotonically and bias increases monotonically.** At $\lambda=0$ this is the OLS result $\sigma^2p/n$. Directions with small $d_j$ (poorly determined by the data) are shrunk first, which is where most of the variance lived.

Ridge on the degree-9 fit ($n=10$, Legendre features in $t=2x-1$, all 10 coefficients penalized). In-sample values match the SVD formulas and a 20,000-set Monte Carlo; the last column averages over the 201 test points.

| λ | df(λ) | In-sample bias² | In-sample variance | Test-grid total |
| --- | --- | --- | --- | --- |
| 0 (OLS) | 10.00 | 0.0000 | 0.0900 | 0.6774 |
| 0.01 | 9.55 | 0.0000 | 0.0829 | 0.4134 |
| 0.1 | 8.26 | 0.0012 | 0.0669 | 0.1825 |
| **0.3** | 7.15 | 0.0082 | 0.0533 | **0.1530** |
| 1 | 5.42 | 0.0497 | 0.0345 | 0.1724 |
| 10 | 1.97 | 0.3038 | 0.0069 | 0.4278 |

A modest penalty cuts the degree-9 test error from 0.68 to 0.15, close to the cubic's 0.127, without touching the degree. Regularization is a second, continuous complexity dial. See the L1/L2 regularization article for the geometry.

## A modern caveat: double descent

The U-shaped total-error curve is the classical picture for models whose capacity is below the number of samples. Large neural networks, random-feature models and even plain least squares behave differently once the parameter count $p$ passes $n$. Test error rises to a peak at the **interpolation threshold** $p\approx n$ (where the fit is forced through every point and is extremely unstable, as degree 9 at $n=10$ is above), and then **falls again** as $p$ grows further. With more parameters than points there are infinitely many interpolating solutions. Gradient descent and the minimum-norm solver pick the smoothest one, an *implicit* regularization that lowers variance.

> **The decomposition still holds:** Double descent does not break $\text{bias}^2+\text{variance}+\sigma^2$, which is an identity. It breaks the assumption that "number of parameters" measures complexity. Past the threshold, the effective complexity of the minimum-norm interpolator *drops* as $p$ grows, so variance falls. A well-tuned explicit penalty (ridge with the best $\lambda$) largely removes the peak.

## Diagnosing bias and variance in practice

On real data you cannot redraw training sets or see $f$, so bias and variance are never measured directly. You read them off two error curves instead: **training error** (an optimistic proxy that ignores variance) and **validation error** (an honest estimate of $\mathrm{Err}$).

**Reading the curves**

- **Validation curve:** error vs a complexity knob at fixed n. Training error falls steadily; validation error is U-shaped. Left of the minimum is the high-bias region, right of it the high-variance region.
- **Learning curve:** error vs training-set size n at fixed complexity. Training error rises with n (harder to memorize), validation error falls; they converge to the model's bias² + σ².
- **High bias:** both curves plateau early, close together, at an unacceptably high level. More data will not help.
- **High variance:** a large gap between training and validation error that narrows slowly as n grows. More data will help.

Remedies, and which term they attack.

| Symptom | Diagnosis | Remedies | Side effect |
| --- | --- | --- | --- |
| Train and validation error both high, close together | **high bias** (underfitting) | more flexible model (higher degree, deeper trees, smaller k), more or better features, interactions, less regularization, train longer / boosting | variance rises |
| Train error low, validation error much higher | **high variance** (overfitting) | more training data, stronger regularization (ridge, lasso, dropout, weight decay), simpler model, feature selection, bagging / random forests, early stopping, data augmentation | bias rises (except more data and bagging, which mostly leave it alone) |
| Both low and close | near the noise floor | stop. Further gains need less noisy labels or new informative features | none |

> **Two tools beat the trade-off:** **More data** lowers variance ($\sigma^2p/n$, $\sigma^2/k$ with $k$ fixed) without changing bias. **Bagging** averages $B$ models trained on bootstrap resamples. If the models' predictions have variance $v$ and pairwise correlation $\rho$, the average has variance $\rho v+(1-\rho)v/B$, while its bias stays that of a single model. Random forests add feature subsampling to lower $\rho$. Boosting goes the other way: it combines high-bias weak learners to lower bias.

## In code

```python
import numpy as np

f = lambda x: np.sin(2 * np.pi * x)
sigma, n, R = 0.3, 10, 20_000
x_train = np.linspace(0, 1, n)            # fixed design: only the noise is redrawn
x_test = np.linspace(0, 1, 201)
rng = np.random.default_rng(1)

def fit_predict(y, degree):
    coef = np.polynomial.legendre.legfit(2 * x_train - 1, y, degree)   # least squares
    return np.polynomial.legendre.legval(2 * x_test - 1, coef)

for degree in [1, 3, 9]:
    Y = f(x_train) + sigma * rng.standard_normal((R, n))    # R training sets
    F = np.array([fit_predict(y, degree) for y in Y])        # R x 201 predictions
    bias2 = ((F.mean(0) - f(x_test)) ** 2).mean()
    var = F.var(0).mean()
    y_new = f(x_test) + sigma * rng.standard_normal(F.shape) # fresh test noise
    mse = ((y_new - F) ** 2).mean()
    print(f"degree {degree}: bias^2 {bias2:.4f}  var {var:.4f}  "
          f"sum {bias2 + var + sigma**2:.4f}  direct test MSE {mse:.4f}")

# Closed form: every fit is linear in y, f_hat(x_test) = L @ y
def smoother(degree):
    P = np.polynomial.legendre.legvander(2 * x_train - 1, degree)
    G = np.polynomial.legendre.legvander(2 * x_test - 1, degree)
    return G @ np.linalg.pinv(P)

for degree in [1, 3, 9]:
    L = smoother(degree)
    bias2 = ((L @ f(x_train) - f(x_test)) ** 2).mean()
    var = sigma**2 * (L**2).sum(1).mean()
    print(f"degree {degree}: exact bias^2 {bias2:.4f}  var {var:.4f}  total {bias2 + var + sigma**2:.4f}")
```

Monte-Carlo decomposition and the exact linear-smoother version. Output: degree 1: 0.2238/0.0163 (direct MSE 0.3299); degree 3: 0.0066/0.0298 (0.1264); degree 9: 0.0001/0.5768 (0.6673). Exact: 0.3299, 0.1265, 0.6774.

```python
import numpy as np
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures, StandardScaler
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import validation_curve, learning_curve, KFold

rng = np.random.default_rng(0)
X = rng.uniform(0, 1, size=(60, 1))
y = np.sin(2 * np.pi * X[:, 0]) + rng.normal(0, 0.3, size=60)

model = make_pipeline(PolynomialFeatures(include_bias=False), StandardScaler(), LinearRegression())
cv = KFold(5, shuffle=True, random_state=0)
degrees = np.arange(1, 13)

train, val = validation_curve(model, X, y, param_name="polynomialfeatures__degree",
                              param_range=degrees, cv=cv, scoring="neg_mean_squared_error")
for d, tr, va in zip(degrees, -train.mean(1), -val.mean(1)):
    print(f"degree {d:2d}  train MSE {tr:.3f}  val MSE {va:.3f}")

for d in [1, 9]:                                   # learning curves: high bias vs high variance
    model.set_params(polynomialfeatures__degree=d)
    sizes, tr, va = learning_curve(model, X, y, train_sizes=[12, 24, 36, 48],
                                   cv=cv, scoring="neg_mean_squared_error")
    print(d, sizes, (-tr.mean(1)).round(3), (-va.mean(1)).round(3))
```

On real data: validation and learning curves with scikit-learn (60 random points from the same sine). Validation MSE bottoms out at degree 7 (0.098). Degree 1 plateaus at ≈0.29 train / 0.315 validation (bias). Degree 9 has train 0.063 vs validation 0.119 at n = 48, and validation 639 at n = 12 (variance).

> **Why the best degree differs from the worked example:** With 60 points (48 per training fold) instead of 10, variance is smaller and the optimum moves to a more flexible model: degree 7 here vs degree 3 at n = 10. Degrees 4 to 10 are all within about 0.02 of each other, so in practice you would pick the simplest model within one standard error of the minimum.

**Strengths**

- Choosing model complexity: it explains why validation error is U-shaped and where to look for the minimum.
- Deciding what to try next: more data and bagging help high variance; features and flexibility help high bias. The decomposition tells you which.
- Understanding ensembles and regularization: bagging, random forests, ridge, dropout and early stopping are all variance reduction; boosting is bias reduction.

**Limitations**

- It is exact only for squared error. 0–1 loss has no clean additive split, and bias and variance can interact (more variance can even help a biased classifier).
- You cannot compute it on real data: it needs the true f and many independent training sets. Use validation and learning curves instead.
- Parameter count is a poor complexity measure for overparameterized models (double descent). Use effective degrees of freedom or norms instead.

- **Hyperparameter tuning:** Every validation curve over depth, k, λ, degree or number of epochs is a bias–variance sweep.
- **Data-collection decisions:** A learning curve that is still falling says more data will pay; a flat one says spend the effort on features or a richer model.
- **Ensemble design:** Bag low-bias, high-variance learners (deep trees); boost high-bias, low-variance ones (stumps).
- **Small-data science:** With few samples, strong priors and regularization (Bayesian or ridge models) trade a little bias for a large cut in variance.

> **Training error lies:** Training error ignores variance entirely and keeps falling as complexity grows. Degree 9 at n = 10 has training error 0 and the worst test error. Always compare against held-out validation or cross-validation error.

> **Tuning on the test set:** If you pick the complexity that minimizes *test* error, that error is no longer an unbiased estimate: you have fitted the hyperparameter to the test noise. Select on validation or CV folds; touch the test set once.

> **Variance is not "noise in the data":** $\sigma^2$ is noise in the labels and cannot be reduced by the model. Variance is the *model's* instability across training sets, and it is reducible. Don't blame a large train–validation gap on "noisy data" before trying regularization or more samples.

> **Variance is not uniform in x:** The average hides where the error lives. Flexible models are most unstable at the edges of the data and in sparse regions (variance 3.93 at x = 0.95 for degree 9 vs 0.087 at x = 0.25), which is exactly where extrapolation happens in production.

**Question:** Training MSE is 0.02 and validation MSE is 0.35; the label noise is believed to be about 0.05. What should you try first?

- A. A more flexible model
- B. More training data or stronger regularization
- C. Removing regularization
- D. Nothing: this is irreducible noise

**Answer:** B. More training data or stronger regularization — A large gap between training and validation error, with training error below the noise level, is the signature of high variance. More data, regularization, bagging or a simpler model attack variance; more flexibility would make it worse.

**Question:** In the derivation, why does the cross term $2\,\mathbb{E}\big[(f(x)-\bar f)(\bar f-\hat f)\big]$ vanish?

- A. Because the noise has mean zero
- B. Because $f(x)-\bar f$ is a constant and $\mathbb{E}[\hat f]=\bar f$
- C. Because the model is unbiased
- D. Because $\hat f$ and $f$ are independent

**Answer:** B. Because $f(x)-\bar f$ is a constant and $\mathbb{E}[\hat f]=\bar f$ — $f(x)-\bar f$ contains no randomness, so it factors out, leaving $\mathbb{E}[\bar f-\hat f]=\bar f-\bar f=0$ by the definition of $\bar f$. The noise-mean-zero argument is used for the *other* cross term, in Step 1.

**Question:** kNN regression with noise variance $\sigma^2=0.09$. What is the variance of the prediction at any point when $k=9$?

- A. 0.09
- B. 0.03
- C. 0.01
- D. 0.0011

**Answer:** C. 0.01 — The prediction averages 9 independent noisy labels, so its variance is $\sigma^2/k=0.09/9=0.01$, regardless of $f$ or of where the neighbours lie.

**Question:** You double the training-set size and keep the same polynomial degree. What happens, in expectation?

- A. Bias falls, variance unchanged
- B. Variance falls, bias (roughly) unchanged
- C. Both rise
- D. The noise term σ² halves

**Answer:** B. Variance falls, bias (roughly) unchanged — For least squares the in-sample variance is $\sigma^2p/n$, so it halves; the bias is a property of the model class and barely moves (degree 3: 0.0066 at n = 10, 0.0047 at n = 50). $\sigma^2$ belongs to the data and never changes.

**Key points**

- Expected squared error at $x$ = bias² + variance + $\sigma^2$. Derive it by adding and subtracting $f(x)$, then $\bar f(x)$. The cross terms vanish by independence of the test noise and by the definition of $\bar f$.
- Bias is the error of the average model (underfitting); variance is the spread across training sets (overfitting); σ² is a floor no model beats.
- Closed forms for linear smoothers: least squares has in-sample variance $\sigma^2p/n$, kNN has variance exactly $\sigma^2/k$ with bias growing in $k$, and ridge trades variance for bias monotonically as $\lambda$ grows.
- On sin(2πx) with n = 10, σ = 0.3: degree 1 totals 0.330 (bias), degree 9 totals 0.677 (variance), degree 3 is best at 0.127.
- More data and bagging cut variance without adding bias; diagnose with validation and learning curves, never with training error.
- Double descent: past the interpolation threshold, parameter count stops measuring complexity, but the decomposition itself still holds.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
