The Bias–Variance Trade-off

Expected test error = bias² + variance + irreducible noise; model complexity trades the first two.

20 min read1990 wordscore0 viewsupdated 2026-10-08

Definition

The bias–variance decomposition splits the expected squared error of a learning algorithm at a point xx, averaged over random training sets and over the noise, into three parts: bias² (how far the average fitted model is from the truth), variance (how much the fitted model moves from one training set to the next) and irreducible noise σ2\sigma^2 (what no model can predict).

At a glance

Applies to
squared-error regression exactly; classification losses have analogous but messier decompositions
Random over
the training set D\mathcal{D} (and the test noise), with the test input xx held fixed
Bias²
(ED[f^(x)]−f(x))2\big(\mathbb{E}_\mathcal{D}[\hat f(x)]-f(x)\big)^2: systematic error, underfitting
Variance
VarD[f^(x)]\mathrm{Var}_\mathcal{D}[\hat f(x)]: sensitivity to the sample, overfitting
Noise
σ2=Var[y∣x]\sigma^2=\mathrm{Var}[y\mid x]: a floor no model beats
Knobs
polynomial degree, k in kNN, tree depth, regularization λ, number of features, ensemble size, training-set size n
Diagnosed by
training vs validation error, validation curves, learning curves

The intuition: four dartboards

Imagine you could rerun your whole pipeline on many fresh training sets from the same source, and each time throw one dart: the trained model's prediction at a fixed test input. The bullseye is the true value f(x)f(x). Two different things can go wrong.

DartboardLow variance (tight cluster)High variance (scattered)
Low bias (centred on the bullseye)the goal: right on average, and every timeright on average, but any single model can be far off
High bias (centred off the bullseye)consistently wrong in the same waywrong on average and unreliable
Each throw is the prediction of a model trained on a different random training set.

Bias is the distance from the centre of the cluster to the bullseye. It comes from assumptions the model cannot escape: a straight line fitted to a sine wave misses the curve no matter how much data you give it. That is underfitting: training and test error are both high.

Variance is the spread of the cluster. A very flexible model, such as a degree-9 polynomial through 10 points, bends to follow whatever noise happened to land in its training set, so the curves from different sets disagree wildly. That is overfitting: training error is tiny, test error is large.

Making a model more flexible usually moves bias down and variance up. The trade-off is choosing the flexibility where their sum is smallest. The noise σ2\sigma^2 adds a fixed amount on top of every model.

Setup and notation

Data come from y=f(x)+εy=f(x)+\varepsilon with E[ε]=0\mathbb{E}[\varepsilon]=0, Var[ε]=σ2\mathrm{Var}[\varepsilon]=\sigma^2, noise independent of everything else. A learning algorithm maps a training set D={(xi,yi)}i=1n\mathcal{D}=\{(x_i,y_i)\}_{i=1}^n to a function f^D\hat f_\mathcal{D}. Fix a test input xx and draw a fresh test label y=f(x)+εy=f(x)+\varepsilon, independent of D\mathcal{D}. We want the expected prediction error at xx, averaged over both sources of randomness:

Err(x)=ED,ε[(y−f^D(x))2],fˉ(x)=ED[f^D(x)]\mathrm{Err}(x)=\mathbb{E}_{\mathcal{D},\varepsilon}\big[(y-\hat f_\mathcal{D}(x))^2\big],\qquad \bar f(x)=\mathbb{E}_\mathcal{D}\big[\hat f_\mathcal{D}(x)\big]
f(x)f(x)
true regression function, E[y∣x]\mathbb{E}[y\mid x]
f^D(x)\hat f_\mathcal{D}(x)
prediction of the model trained on training set D\mathcal{D} (random, because D\mathcal{D} is)
fˉ(x)\bar f(x)
the average prediction over all possible training sets: the centre of the dart cluster
ε\varepsilon
test-label noise, mean 0, variance σ2\sigma^2, independent of D\mathcal{D}

Deriving the decomposition

The whole proof is "add and subtract the right thing, expand the square, and show the cross terms have mean zero". We do it twice: once to peel off the noise, once to split the rest into bias and variance. Write f^\hat f for f^D(x)\hat f_\mathcal{D}(x).

Step 1: separate the noise

  1. 1Add and subtract f(x)f(x):   y−f^=(y−f(x))+(f(x)−f^)=ε+(f(x)−f^).\;y-\hat f=\big(y-f(x)\big)+\big(f(x)-\hat f\big)=\varepsilon+\big(f(x)-\hat f\big).
  2. 2Square:   (y−f^)2=ε2+2 ε(f(x)−f^)+(f(x)−f^)2.\;(y-\hat f)^2=\varepsilon^2+2\,\varepsilon\big(f(x)-\hat f\big)+\big(f(x)-\hat f\big)^2.
  3. 3Take expectations. E[ε2]=σ2\mathbb{E}[\varepsilon^2]=\sigma^2. The cross term factorizes because the test noise ε\varepsilon is independent of the training set and hence of f^\hat f:   E[ε(f(x)−f^)]=E[ε]⋅E[f(x)−f^]=0⋅(… )=0.\;\mathbb{E}\big[\varepsilon(f(x)-\hat f)\big]=\mathbb{E}[\varepsilon]\cdot\mathbb{E}\big[f(x)-\hat f\big]=0\cdot(\dots)=0.
  4. 4So   Err(x)=σ2+ED[(f(x)−f^)2]\;\mathrm{Err}(x)=\sigma^2+\mathbb{E}_\mathcal{D}\big[(f(x)-\hat f)^2\big]: noise plus the mean squared error of the estimator f^\hat f as an estimate of the number f(x)f(x).

Step 2: split the estimator's error

  1. 1Add and subtract the average prediction fˉ=ED[f^]\bar f=\mathbb{E}_\mathcal{D}[\hat f]:   f(x)−f^=(f(x)−fˉ)+(fˉ−f^).\;f(x)-\hat f=\big(f(x)-\bar f\big)+\big(\bar f-\hat f\big).
  2. 2Square:   (f(x)−f^)2=(f(x)−fˉ)2+2(f(x)−fˉ)(fˉ−f^)+(fˉ−f^)2.\;(f(x)-\hat f)^2=\big(f(x)-\bar f\big)^2+2\big(f(x)-\bar f\big)\big(\bar f-\hat f\big)+\big(\bar f-\hat f\big)^2.
  3. 3The first factor of the cross term, f(x)−fˉf(x)-\bar f, is a constant (no randomness left in it), so it comes out of the expectation:   2(f(x)−fˉ) ED[fˉ−f^]=2(f(x)−fˉ)(fˉ−fˉ)=0.\;2\big(f(x)-\bar f\big)\,\mathbb{E}_\mathcal{D}\big[\bar f-\hat f\big]=2\big(f(x)-\bar f\big)\big(\bar f-\bar f\big)=0.
  4. 4What remains:   ED[(f(x)−f^)2]=(fˉ−f(x))2+ED[(f^−fˉ)2]\;\mathbb{E}_\mathcal{D}\big[(f(x)-\hat f)^2\big]=\big(\bar f-f(x)\big)^2+\mathbb{E}_\mathcal{D}\big[(\hat f-\bar f)^2\big], which is bias² plus variance.
ED,ε[(y−f^D(x))2]=(fˉ(x)−f(x))2⏟bias2+ED[(f^D(x)−fˉ(x))2]⏟variance+σ2⏟noise\mathbb{E}_{\mathcal{D},\varepsilon}\big[(y-\hat f_\mathcal{D}(x))^2\big]=\underbrace{\big(\bar f(x)-f(x)\big)^2}_{\text{bias}^2}+\underbrace{\mathbb{E}_\mathcal{D}\big[(\hat f_\mathcal{D}(x)-\bar f(x))^2\big]}_{\text{variance}}+\underbrace{\sigma^2}_{\text{noise}}
bias2\text{bias}^2
squared distance from the average model to the truth; zero if the model class can represent f and the method is unbiased
variance\text{variance}
spread of the fitted models around their own average; zero for a model that ignores the data
σ2\sigma^2
irreducible error: even the true ff has expected squared error σ2\sigma^2

Two facts make the cross terms vanish: the test noise is independent of the training set, and fˉ\bar f is the mean of f^\hat f, so deviations from it average to zero. All three terms are non-negative, so no model can beat σ2\sigma^2. To get the error over the whole input distribution, average over xx:   Ex[Err(x)]=Ex[bias2(x)]+Ex[var(x)]+σ2.\;\mathbb{E}_x[\mathrm{Err}(x)]=\mathbb{E}_x[\text{bias}^2(x)]+\mathbb{E}_x[\text{var}(x)]+\sigma^2.

See it: many training sets, one picture

The figure below runs the thought experiment literally. The true function is f(x)=sin⁡(2πx)f(x)=\sin(2\pi x) on [0,1][0,1], with noise σ=0.3\sigma=0.3. Each training set has nn inputs on a fixed, equally spaced grid xi=i/(n−1)x_i=i/(n-1); only the noise is redrawn. For each of 30 training sets the model is refitted, and every fit is drawn. The bias, variance and total on the right are exact (computed in closed form, see below), averaged over 201 test points.

30 training sets · fits · mean fit · true f-2-101200.51x →bias² + variance + σ² = total00.20.40.60123456789degree · more flexible →
modeln
worked example
exact (avg over x): 0.0066 + 0.0299 + 0.0900 = 0.1265
from these 30 fits: bias² 0.0079 · variance 0.0273
parameters p = 4 · in-sample var σ²p/n = 0.0360
σ = 0.3 · n = 10 · mint: one training set and its fit
fig.Left: 30 fits (violet), their average (blue, the estimate of the mean model), the true sin(2πx) (dashed) and the ±1 sd band around the exact mean model (orange). One training set and its fit are in mint. Right: bias², variance, σ² and their sum versus complexity, with the minimum ringed. Degree 1 misses the curve the same way every time; degree 9 at n = 10 passes through every point and swings wildly between them. Raise n to watch variance shrink while bias stays put, or switch to kNN, where variance is exactly σ²/k.

Worked example: polynomial degree 1, 3 and 9

Take the setting from the figure with n=10n=10 training points, σ2=0.09\sigma^2=0.09, and fit least-squares polynomials of degree 1, 3 and 9. Two ways to get the numbers. Monte Carlo: draw 20,000 training sets, fit each one, then measure the spread and the offset of the 20,000 predictions at every test point. Closed form: least squares is linear in yy, so the mean and variance of f^(x)\hat f(x) follow from one matrix (next section). Both agree:

DegreeBias²VarianceNoise σ²Total (exact)MC bias² / varMC direct test MSEVerdict
10.22350.01640.090.32990.2238 / 0.01630.3299underfits: bias dominates
30.00660.02990.090.12650.0066 / 0.02980.1264best of the three
9< 0.00010.58740.090.67740.0001 / 0.57680.6673overfits: variance dominates
f(x)=sin⁡(2πx)f(x)=\sin(2\pi x), n=10n=10, σ=0.3\sigma=0.3; averaged over 201 test points in [0,1][0,1]. MC = 20,000 simulated training sets (seed 1); "direct" is the average of (ynew−f^)2(y_\text{new}-\hat f)^2 with fresh test noise, computed without using the decomposition at all.

Reading the table: the line has bias² 0.22, so even with infinite data its error would stay near 0.22+0.090.22+0.09. The degree-9 polynomial is essentially unbiased (on average it reproduces the sine to within 10−410^{-4}) yet its variance is 0.59, 6.5 times the noise. The cubic pays a small bias (0.007) for a small variance (0.03) and wins with a total of 0.127, only 0.037 above the floor σ2\sigma^2. Sweeping every degree, the minimum is at degree 3.

Test pointDegreef(x)Mean fit f̄(x)Bias²Variance
x = 0.25 (peak)11.0000.3370.43930.0145
x = 0.2531.0000.9610.00150.0298
x = 0.2591.0001.0000.00000.0866
x = 0.95 (edge)1−0.309−0.6070.08880.0269
x = 0.953−0.309−0.4020.00860.0397
x = 0.959−0.309−0.3090.00003.9322
Where the error lives: single test points (exact, n = 10). Variance is not uniform in x: high-degree fits explode near the edges of the data.

Two curiosities show up when you sweep all degrees. Degree 2 has exactly the same bias as degree 1 (0.2235), and degree 4 the same as degree 3: sin⁡(2πx)\sin(2\pi x) is antisymmetric about x=12x=\tfrac12 and the design is symmetric, so the added even term has expected coefficient 0 and only adds variance. And the training error of degree 9 is exactly 0 (10 coefficients, 10 points), while its test error is 0.68, the worst in the table.

nDegree 3: bias²Degree 3: varianceDegree 3: totalDegree 9: varianceDegree 9: total
100.00660.02990.12650.58740.6774
150.00560.02090.11650.06380.1538
250.00500.01320.10810.03290.1229
500.00470.00690.10160.01660.1066
More data shrinks variance and leaves bias alone (exact, σ=0.3\sigma=0.3).

At n=10n=10 the cubic beats degree 9 by a factor of five; at n=50n=50 the gap is under 5%. This is why "the right complexity" is not a property of the problem alone: it depends on how much data you have. The best degree drifts upward as nn grows.

Closed forms: linear smoothers

Least squares, ridge and kNN all predict with a fixed linear combination of the training labels. With the inputs x1,…,xnx_1,\dots,x_n held fixed (only the noise is random), write

f^(x0)=∑i=1nℓi(x0) yi⇒fˉ(x0)=∑iℓi(x0)f(xi),Var[f^(x0)]=σ2∑iℓi(x0)2\hat f(x_0)=\sum_{i=1}^n \ell_i(x_0)\,y_i\quad\Rightarrow\quad \bar f(x_0)=\sum_{i}\ell_i(x_0)f(x_i),\qquad \mathrm{Var}[\hat f(x_0)]=\sigma^2\sum_i \ell_i(x_0)^2
ℓi(x0)\ell_i(x_0)
weight the method puts on training label yiy_i when predicting at x0x_0 (depends on the inputs only)
∑iℓi2\sum_i\ell_i^2
=∥ℓ(x0)∥2=\|\ell(x_0)\|^2; independent noise makes variances add with squared weights

The mean follows from E[yi]=f(xi)\mathbb{E}[y_i]=f(x_i) and linearity; the variance from Var[∑iℓiεi]=∑iℓi2σ2\mathrm{Var}[\sum_i\ell_i\varepsilon_i]=\sum_i\ell_i^2\sigma^2 for independent noise. This is exactly how the figure and the "exact" column above are computed. Three cases are worth working out by hand.

Least squares: variance = σ² p / n

For a model with pp features (here p=d+1p=d+1 polynomial coefficients) and design matrix Φ∈Rn×p\Phi\in\mathbb{R}^{n\times p}, the fitted values at the training inputs are y^=Hy\hat{\mathbf y}=H\mathbf y with the hat matrix H=Φ(Φ⊤Φ)−1Φ⊤H=\Phi(\Phi^\top\Phi)^{-1}\Phi^\top. HH is symmetric and idempotent (H2=HH^2=H), so the squared weights in row ii sum to ∑jHij2=(H2)ii=Hii\sum_j H_{ij}^2=(H^2)_{ii}=H_{ii}. Averaging over the training inputs:

1n∑i=1nVar[f^(xi)]=σ2n∑iHii=σ2ntr⁡H=σ2 pn\frac1n\sum_{i=1}^n\mathrm{Var}[\hat f(x_i)]=\frac{\sigma^2}{n}\sum_i H_{ii}=\frac{\sigma^2}{n}\operatorname{tr}H=\sigma^2\,\frac{p}{n}
HH
hat matrix, projection onto the column space of Φ
tr⁡H=p\operatorname{tr}H=p
trace of a rank-p projection; the number of parameters

Every extra parameter costs σ2/n\sigma^2/n of in-sample variance, whether or not it is needed. Checked numerically for n=10n=10: degree 1, 3, 9 give 0.018, 0.036, 0.090, exactly 0.09⋅p/100.09\cdot p/10. Away from the training inputs variance is larger (0.59 rather than 0.09 for degree 9), because polynomials extrapolate badly between and beyond the outer points.

kNN regression: variance = σ² / k

kNN regression averages the labels of the kk training points nearest to x0x_0, so ℓi=1/k\ell_i=1/k on those neighbours Nk(x0)N_k(x_0) and 0 elsewhere:

Err(x0)=σ2+(f(x0)−1k∑i∈Nk(x0)f(xi))2+σ2k\mathrm{Err}(x_0)=\sigma^2+\Big(f(x_0)-\frac1k\sum_{i\in N_k(x_0)}f(x_i)\Big)^2+\frac{\sigma^2}{k}
Nk(x0)N_k(x_0)
indices of the kk training inputs closest to x0x_0
σ2/k\sigma^2/k
variance: kk squared weights of 1/k21/k^2, times σ2\sigma^2, exact for any ff

The variance is exact and does not depend on ff. The bias grows with kk because the neighbourhood reaches further from x0x_0. In 1-D with spacing hh and ff smooth, a Taylor expansion gives bias ≈12f′′(x0)⋅(xi−x0)2‾≈f′′(x0) k2h2/24\approx \tfrac{1}{2}f''(x_0)\cdot\overline{(x_i-x_0)^2}\approx f''(x_0)\,k^2h^2/24 for an interior point. So the bias grows like k2k^2 while the variance falls like 1/k1/k. Small kk is the flexible end of the dial.

kBias²Variance = σ²/kTotal
10.00070.09000.1807
30.00090.03000.1209
50.00270.01800.1107
100.01800.00900.1170
200.06590.00450.1604
50 (= n)0.49750.00180.5893
kNN on the same sine, n=50n=50 equally spaced points, σ=0.3\sigma=0.3; exact values, confirmed by 20,000-set Monte Carlo to within 0.0003.

At k=nk=n every prediction is the global mean; its bias² 0.49750.4975 is the variance of sin⁡(2πx)\sin(2\pi x) over the grid, close to 12\tfrac12. The minimum sits at k≈5k\approx 5–6 (total 0.109 at k=6k=6).

Ridge: variance falls as λ grows

Ridge regression fits w^=(Φ⊤Φ+λI)−1Φ⊤y\hat w=(\Phi^\top\Phi+\lambda I)^{-1}\Phi^\top\mathbf y. Take the SVD Φ=UDV⊤\Phi=UDV^\top with singular values djd_j. The fitted values are SλyS_\lambda\mathbf y with Sλ=U diag(dj2dj2+λ)U⊤S_\lambda=U\,\mathrm{diag}\big(\tfrac{d_j^2}{d_j^2+\lambda}\big)U^\top: ridge shrinks the component of y\mathbf y along each direction uju_j by sj=dj2/(dj2+λ)∈(0,1]s_j=d_j^2/(d_j^2+\lambda)\in(0,1]. With f=(f(x1),…,f(xn))\mathbf f=(f(x_1),\dots,f(x_n)), the in-sample terms are

bias2‾=1n[∑j(λdj2+λ)2(uj⊤f)2+∥(I−UU⊤)f∥2],var‾=σ2n∑j(dj2dj2+λ)2\overline{\text{bias}^2}=\frac1n\Big[\sum_j\Big(\frac{\lambda}{d_j^2+\lambda}\Big)^2(u_j^\top\mathbf f)^2+\|(I-UU^\top)\mathbf f\|^2\Big],\qquad \overline{\text{var}}=\frac{\sigma^2}{n}\sum_j\Big(\frac{d_j^2}{d_j^2+\lambda}\Big)^2
dj, ujd_j,\ u_j
singular values and left singular vectors of the design matrix
λ/(dj2+λ)\lambda/(d_j^2+\lambda)
=1−sj=1-s_j, the fraction of signal along uju_j that ridge throws away
∥(I−UU⊤)f∥2\|(I-UU^\top)\mathbf f\|^2
the part of f the features cannot represent at all; no λ fixes it
df(λ)\mathrm{df}(\lambda)
=∑jsj=tr⁡Sλ=\sum_j s_j=\operatorname{tr}S_\lambda, effective number of parameters

Derivation in one line each: E[Sλy]−f=−(I−Sλ)f\mathbb{E}[S_\lambda\mathbf y]-\mathbf f=-(I-S_\lambda)\mathbf f, and I−SλI-S_\lambda scales the uju_j component by 1−sj1-s_j and keeps the part outside the column space. Cov[Sλy]=σ2Sλ2\mathrm{Cov}[S_\lambda\mathbf y]=\sigma^2S_\lambda^2, whose trace is σ2∑jsj2\sigma^2\sum_j s_j^2. As λ\lambda goes from 0 to ∞\infty, every sjs_j falls from 1 to 0: variance decreases monotonically and bias increases monotonically. At λ=0\lambda=0 this is the OLS result σ2p/n\sigma^2p/n. Directions with small djd_j (poorly determined by the data) are shrunk first, which is where most of the variance lived.

λdf(λ)In-sample bias²In-sample varianceTest-grid total
0 (OLS)10.000.00000.09000.6774
0.019.550.00000.08290.4134
0.18.260.00120.06690.1825
0.37.150.00820.05330.1530
15.420.04970.03450.1724
101.970.30380.00690.4278
Ridge on the degree-9 fit (n=10n=10, Legendre features in t=2x−1t=2x-1, all 10 coefficients penalized). In-sample values match the SVD formulas and a 20,000-set Monte Carlo; the last column averages over the 201 test points.

A modest penalty cuts the degree-9 test error from 0.68 to 0.15, close to the cubic's 0.127, without touching the degree. Regularization is a second, continuous complexity dial. See the L1/L2 regularization article for the geometry.

A modern caveat: double descent

The U-shaped total-error curve is the classical picture for models whose capacity is below the number of samples. Large neural networks, random-feature models and even plain least squares behave differently once the parameter count pp passes nn. Test error rises to a peak at the interpolation threshold p≈np\approx n (where the fit is forced through every point and is extremely unstable, as degree 9 at n=10n=10 is above), and then falls again as pp grows further. With more parameters than points there are infinitely many interpolating solutions. Gradient descent and the minimum-norm solver pick the smoothest one, an implicit regularization that lowers variance.

Diagnosing bias and variance in practice

On real data you cannot redraw training sets or see ff, so bias and variance are never measured directly. You read them off two error curves instead: training error (an optimistic proxy that ignores variance) and validation error (an honest estimate of Err\mathrm{Err}).

Reading the curves

Validation curve
error vs a complexity knob at fixed n. Training error falls steadily; validation error is U-shaped. Left of the minimum is the high-bias region, right of it the high-variance region.
Learning curve
error vs training-set size n at fixed complexity. Training error rises with n (harder to memorize), validation error falls; they converge to the model's bias² + σ².
High bias
both curves plateau early, close together, at an unacceptably high level. More data will not help.
High variance
a large gap between training and validation error that narrows slowly as n grows. More data will help.
SymptomDiagnosisRemediesSide effect
Train and validation error both high, close togetherhigh bias (underfitting)more flexible model (higher degree, deeper trees, smaller k), more or better features, interactions, less regularization, train longer / boostingvariance rises
Train error low, validation error much higherhigh variance (overfitting)more training data, stronger regularization (ridge, lasso, dropout, weight decay), simpler model, feature selection, bagging / random forests, early stopping, data augmentationbias rises (except more data and bagging, which mostly leave it alone)
Both low and closenear the noise floorstop. Further gains need less noisy labels or new informative featuresnone
Remedies, and which term they attack.

In code

python
1import numpy as np
2
3f = lambda x: np.sin(2 * np.pi * x)
4sigma, n, R = 0.3, 10, 20_000
5x_train = np.linspace(0, 1, n) # fixed design: only the noise is redrawn
6x_test = np.linspace(0, 1, 201)
7rng = np.random.default_rng(1)
8
9def fit_predict(y, degree):
10 coef = np.polynomial.legendre.legfit(2 * x_train - 1, y, degree) # least squares
11 return np.polynomial.legendre.legval(2 * x_test - 1, coef)
12
13for degree in [1, 3, 9]:
14 Y = f(x_train) + sigma * rng.standard_normal((R, n)) # R training sets
15 F = np.array([fit_predict(y, degree) for y in Y]) # R x 201 predictions
16 bias2 = ((F.mean(0) - f(x_test)) ** 2).mean()
17 var = F.var(0).mean()
18 y_new = f(x_test) + sigma * rng.standard_normal(F.shape) # fresh test noise
19 mse = ((y_new - F) ** 2).mean()
20 print(f"degree {degree}: bias^2 {bias2:.4f} var {var:.4f} "
21 f"sum {bias2 + var + sigma**2:.4f} direct test MSE {mse:.4f}")
22
23# Closed form: every fit is linear in y, f_hat(x_test) = L @ y
24def smoother(degree):
25 P = np.polynomial.legendre.legvander(2 * x_train - 1, degree)
26 G = np.polynomial.legendre.legvander(2 * x_test - 1, degree)
27 return G @ np.linalg.pinv(P)
28
29for degree in [1, 3, 9]:
30 L = smoother(degree)
31 bias2 = ((L @ f(x_train) - f(x_test)) ** 2).mean()
32 var = sigma**2 * (L**2).sum(1).mean()
33 print(f"degree {degree}: exact bias^2 {bias2:.4f} var {var:.4f} total {bias2 + var + sigma**2:.4f}")
Monte-Carlo decomposition and the exact linear-smoother version. Output: degree 1: 0.2238/0.0163 (direct MSE 0.3299); degree 3: 0.0066/0.0298 (0.1264); degree 9: 0.0001/0.5768 (0.6673). Exact: 0.3299, 0.1265, 0.6774.
python
1import numpy as np
2from sklearn.pipeline import make_pipeline
3from sklearn.preprocessing import PolynomialFeatures, StandardScaler
4from sklearn.linear_model import LinearRegression
5from sklearn.model_selection import validation_curve, learning_curve, KFold
6
7rng = np.random.default_rng(0)
8X = rng.uniform(0, 1, size=(60, 1))
9y = np.sin(2 * np.pi * X[:, 0]) + rng.normal(0, 0.3, size=60)
10
11model = make_pipeline(PolynomialFeatures(include_bias=False), StandardScaler(), LinearRegression())
12cv = KFold(5, shuffle=True, random_state=0)
13degrees = np.arange(1, 13)
14
15train, val = validation_curve(model, X, y, param_name="polynomialfeatures__degree",
16 param_range=degrees, cv=cv, scoring="neg_mean_squared_error")
17for d, tr, va in zip(degrees, -train.mean(1), -val.mean(1)):
18 print(f"degree {d:2d} train MSE {tr:.3f} val MSE {va:.3f}")
19
20for d in [1, 9]: # learning curves: high bias vs high variance
21 model.set_params(polynomialfeatures__degree=d)
22 sizes, tr, va = learning_curve(model, X, y, train_sizes=[12, 24, 36, 48],
23 cv=cv, scoring="neg_mean_squared_error")
24 print(d, sizes, (-tr.mean(1)).round(3), (-va.mean(1)).round(3))
On real data: validation and learning curves with scikit-learn (60 random points from the same sine). Validation MSE bottoms out at degree 7 (0.098). Degree 1 plateaus at ≈0.29 train / 0.315 validation (bias). Degree 9 has train 0.063 vs validation 0.119 at n = 48, and validation 639 at n = 12 (variance).

Use when

  • +Choosing model complexity: it explains why validation error is U-shaped and where to look for the minimum.
  • +Deciding what to try next: more data and bagging help high variance; features and flexibility help high bias. The decomposition tells you which.
  • +Understanding ensembles and regularization: bagging, random forests, ridge, dropout and early stopping are all variance reduction; boosting is bias reduction.

Avoid when

  • −It is exact only for squared error. 0–1 loss has no clean additive split, and bias and variance can interact (more variance can even help a biased classifier).
  • −You cannot compute it on real data: it needs the true f and many independent training sets. Use validation and learning curves instead.
  • −Parameter count is a poor complexity measure for overparameterized models (double descent). Use effective degrees of freedom or norms instead.

Where it's used

Hyperparameter tuning

Every validation curve over depth, k, λ, degree or number of epochs is a bias–variance sweep.

Data-collection decisions

A learning curve that is still falling says more data will pay; a flat one says spend the effort on features or a richer model.

Ensemble design

Bag low-bias, high-variance learners (deep trees); boost high-bias, low-variance ones (stumps).

Small-data science

With few samples, strong priors and regularization (Bayesian or ridge models) trade a little bias for a large cut in variance.

Check yourself

Training MSE is 0.02 and validation MSE is 0.35; the label noise is believed to be about 0.05. What should you try first?

Check yourself

In the derivation, why does the cross term 2 E[(f(x)−fˉ)(fˉ−f^)]2\,\mathbb{E}\big[(f(x)-\bar f)(\bar f-\hat f)\big] vanish?

Check yourself

kNN regression with noise variance σ2=0.09\sigma^2=0.09. What is the variance of the prediction at any point when k=9k=9?

Check yourself

You double the training-set size and keep the same polynomial degree. What happens, in expectation?

Remember

  • ✓Expected squared error at xx = bias² + variance + σ2\sigma^2. Derive it by adding and subtracting f(x)f(x), then fˉ(x)\bar f(x). The cross terms vanish by independence of the test noise and by the definition of fˉ\bar f.
  • ✓Bias is the error of the average model (underfitting); variance is the spread across training sets (overfitting); σ² is a floor no model beats.
  • ✓Closed forms for linear smoothers: least squares has in-sample variance σ2p/n\sigma^2p/n, kNN has variance exactly σ2/k\sigma^2/k with bias growing in kk, and ridge trades variance for bias monotonically as λ\lambda grows.
  • ✓On sin(2πx) with n = 10, σ = 0.3: degree 1 totals 0.330 (bias), degree 9 totals 0.677 (variance), degree 3 is best at 0.127.
  • ✓More data and bagging cut variance without adding bias; diagnose with validation and learning curves, never with training error.
  • ✓Double descent: past the interpolation threshold, parameter count stops measuring complexity, but the decomposition itself still holds.
← back to ML Algorithms roadmap