Cross-validation

Rotate the validation fold k times and average, so every point is used for validation exactly once.

19 min read1494 wordscore0 viewsupdated 2026-10-08

Definition

k-fold cross-validation splits the data into kk disjoint folds, trains the model kk times (each time on k−1k-1 folds) and scores it on the fold that was left out. The average of the kk held-out scores estimates how well this modelling procedure does on unseen data. Every sample is validated exactly once and used for training k−1k-1 times.

At a glance

Purpose
Estimate generalization error; compare models; choose hyperparameters
Input → output
data, a learning procedure and kk → kk fold scores, their mean and standard error
Cost
kk model fits (LOOCV: nn fits, or one fit for linear smoothers)
Typical k
5 or 10; LOOCV (k = n) for very small data or cheap closed-form models
Variants
stratified (class ratios), group (no group in two folds), time series (train on the past), repeated, nested
Assumes
folds are exchangeable: each one looks like the future data, and nothing leaks between train and validation

The problem

Training error is optimistic: the model has seen those points. A single held-out validation set fixes the bias but wastes data and is noisy: with 200 samples a 20% split scores the model on just 40 points, and a different random split can easily move accuracy by several points. On small or medium datasets we want an estimate that uses every point for validation and every point for training, without ever scoring a model on data it was fitted to.

Cross-validation does this by rotation. Cut the data into kk folds. In round jj, hold out fold jj, fit on the rest, score on fold jj. After kk rounds every point has been predicted exactly once by a model that never saw it.

k-fold cross-validation

  1. 1Shuffle the indices (unless order matters, e.g. time) and split them into kk folds F1,…,FkF_1,\dots,F_k of nearly equal size.
  2. 2For j=1…kj=1\dots k: fit the whole pipeline (scaling, feature selection, model) on D∖FjD\setminus F_j.
  3. 3Score that fitted model on FjF_j only, giving the fold loss LjL_j.
  4. 4Report the mean CVk\mathrm{CV}_k and its standard error. Then, for deployment, refit once on all the data.

The estimate and its uncertainty

Lj=1∣Fj∣∑i∈Fjℓ(yi, f^(−j)(xi))CVk=1k∑j=1kLjL_j=\frac{1}{|F_j|}\sum_{i\in F_j}\ell\big(y_i,\ \hat f^{(-j)}(x_i)\big)\qquad \mathrm{CV}_k=\frac1k\sum_{j=1}^{k}L_j
FjF_j
indices in fold jj
f^(−j)\hat f^{(-j)}
model fitted on every fold except FjF_j
ℓ\ell
per-sample loss (squared error, 0–1 loss, log-loss …)
LjL_j
average loss on fold jj
CVk\mathrm{CV}_k
the kk-fold estimate of test error

With equal fold sizes this is the same as averaging the loss over all nn out-of-fold predictions. (scikit-learn's cross_val_score returns the kk fold scores, and you average them; with unequal folds the two averages differ slightly.)

SE^=sLk,sL2=1k−1∑j=1k(Lj−CVk)2\widehat{\mathrm{SE}}=\frac{s_L}{\sqrt k},\qquad s_L^2=\frac{1}{k-1}\sum_{j=1}^{k}\big(L_j-\mathrm{CV}_k\big)^2
sLs_L
sample standard deviation of the k fold losses
SE^\widehat{\mathrm{SE}}
standard error of the fold mean: the usual error bar on a CV score

This formula treats the LjL_j as independent. They are not. Any two training sets share k−2k-2 of their kk folds, so the kk fitted models are strongly correlated, and with them the fold losses. Write σ2\sigma^2 for the variance of one LjL_j and ρ\rho for the correlation between two of them. Then

Var⁡(CVk)=σ2k(1+(k−1)ρ)\operatorname{Var}(\mathrm{CV}_k)=\frac{\sigma^2}{k}\big(1+(k-1)\rho\big)
σ2\sigma^2
variance of a single fold loss
ρ\rho
average correlation between two fold losses (positive: overlapping training sets)

With ρ>0\rho>0 the true variance is larger than σ2/k\sigma^2/k, so sL/ks_L/\sqrt k under-states the uncertainty. Bengio and Grandvalet (2004) showed that no unbiased estimator of this variance exists from a single run of k-fold CV. Read the SE as a rough, optimistic error bar. It is still useful (the "one-standard-error rule" picks the simplest model within one SE of the best), but two models whose CV scores differ by less than about one SE are effectively tied.

Choosing k: bias versus variance

Each fold model sees a fraction (k−1)/k(k-1)/k of the data. Learning curves usually still rise at that point, so fold models are slightly worse than the final model and CVk\mathrm{CV}_k is pessimistically biased. The bias is largest for small kk: 2-fold trains on only half the data.

kTrain size per foldBias (pessimism)Variance of the estimateFits
2n/2highmoderate (few, noisy folds)2
5–1080–90% of nsmallmoderate5–10
n (LOOCV)n − 1almost noneoften high: the n models are nearly identical, so their errors are highly correlatedn (or 1)
How k trades bias, variance and compute.

LOOCV is nearly unbiased, because each model trains on n−1n-1 points. But its nn training sets differ by a single point, so the fold losses are almost perfectly correlated (ρ→1\rho\to1 above), and its variance does not shrink the way 1/n1/n would suggest. For unstable learners (deep trees, 1-NN) it can be quite noisy. Each fold also scores just one point, so with 0–1 loss every LjL_j is 0 or 1. The empirical consensus (Breiman and Spector 1992; Kohavi 1995) is that 5- or 10-fold gives the best trade-off. Repeated k-fold (e.g. 5 × 2 or 10 × 10 with different shuffles) averages away the luck of one particular partition.

The LOOCV shortcut for linear smoothers

For any model whose fitted values are a fixed linear function of yy, y^=Hy\hat y = Hy, you don't need to refit nn times. This covers least squares, ridge, polynomial and spline regression, and kernel smoothers with fixed bandwidth. Deleting point ii and refitting gives exactly:

yi−f^(−i)(xi)=yi−y^i1−hiiLOOCV=1n∑i=1n(yi−y^i1−hii)2y_i-\hat f^{(-i)}(x_i)=\frac{y_i-\hat y_i}{1-h_{ii}}\qquad \mathrm{LOOCV}=\frac1n\sum_{i=1}^{n}\Big(\frac{y_i-\hat y_i}{1-h_{ii}}\Big)^2
HH
hat (smoother) matrix; for OLS H=X(X⊤X)−1X⊤H=X(X^\top X)^{-1}X^\top, for ridge X(X⊤X+λI)−1X⊤X(X^\top X+\lambda I)^{-1}X^\top
y^i\hat y_i
fitted value from the single fit on all n points
hiih_{ii}
leverage of point ii: the ii-th diagonal of HH, between 0 and 1

High-leverage points (far from xˉ\bar x) have hiih_{ii} close to 1, so their residuals are inflated the most. That is right, because the full fit was pulled toward them. For a straight line, hii=1n+(xi−xˉ)2∑m(xm−xˉ)2h_{ii}=\tfrac1n+\tfrac{(x_i-\bar x)^2}{\sum_m (x_m-\bar x)^2}. Replacing each hiih_{ii} by the average tr⁡(H)/n\operatorname{tr}(H)/n gives generalized cross-validation (GCV), 1n∑(yi−y^i)2/(1−tr⁡H/n)2\tfrac1n\sum(y_i-\hat y_i)^2/(1-\operatorname{tr}H/n)^2. RidgeCV uses this trick to try many λ\lambda for the cost of one fit.

Worked example: 10 points, 5 folds, a straight line

Fit y^=b0+b1x\hat y=b_0+b_1x to these 10 points with squared-error loss. The folds pair each point with the one five positions later: F1={1,6}F_1=\{1,6\}, F2={2,7}F_2=\{2,7\}, …, F5={5,10}F_5=\{5,10\}. This is one particular shuffle; it keeps every fold spread across the range of xx.

x12345678910
y2.13.96.27.89.112.513.815.218.619.4
fold1234512345
The data. The least-squares line on all 10 points is ŷ = 0.0267 + 1.9697x, with training MSE 0.2288.
Fold jHeld out xLine fitted on the other 8PredictionsResidualsL_j (MSE)
11, 6−0.120 + 1.978x1.858, 11.7500.242, 0.7500.3105
22, 70.056 + 1.966x3.989, 13.820−0.089, −0.0200.0042
33, 8−0.100 + 2.000x5.900, 15.9000.300, −0.7000.2900
44, 90.191 + 1.921x7.874, 17.478−0.074, 1.1220.6324
55, 100.029 + 1.997x10.012, 19.996−0.912, −0.5960.5938
One row per round: fit on the 8 training points, predict the 2 held-out points, average their squared residuals.

Mean and standard error

  1. 1CV5=15(0.3105+0.0042+0.2900+0.6324+0.5938)=0.3662\mathrm{CV}_5=\tfrac15(0.3105+0.0042+0.2900+0.6324+0.5938)=\mathbf{0.3662}.
  2. 2Deviations from the mean: −0.056, −0.362, −0.076, 0.266, 0.228-0.056,\,-0.362,\,-0.076,\,0.266,\,0.228; sL2=14∑(⋅)2=0.0657s_L^2=\tfrac14\sum(\cdot)^2=0.0657, so sL=0.2563s_L=0.2563.
  3. 3SE^=0.2563/5=0.1146\widehat{\mathrm{SE}}=0.2563/\sqrt5=\mathbf{0.1146}. Report 0.366±0.1150.366\pm0.115. The fold losses range from 0.004 to 0.632 just from which two points land together.
  4. 4Compare with the training MSE of 0.229: the honest estimate is 60% higher.

Now LOOCV with the shortcut: one fit on all 10 points, then divide each residual by 1−hii1-h_{ii}. Here xˉ=5.5\bar x=5.5 and ∑(xm−xˉ)2=82.5\sum(x_m-\bar x)^2=82.5, so for x=1x=1, h11=0.1+20.25/82.5=0.3455h_{11}=0.1+20.25/82.5=0.3455.

xresidual eᵢleverage hᵢᵢeᵢ / (1 − hᵢᵢ)brute force
10.10360.34550.15830.1583
2−0.06610.2485−0.0879−0.0879
30.26420.17580.32060.3206
4−0.10550.1273−0.1208−0.1208
5−0.77520.1030−0.8642−0.8642
60.65520.10300.73040.7304
7−0.01450.1273−0.0167−0.0167
8−0.58420.1758−0.7088−0.7088
90.84610.24851.12581.1258
10−0.32360.3455−0.4944−0.4944
Leave-one-out residuals from one fit (shortcut) versus 10 refits (brute force). They agree to 1e-14.

Squaring and averaging the last column gives LOOCV=0.3445\mathrm{LOOCV}=\mathbf{0.3445} (SE 0.1330.133 over the 10 single-point losses). It is close to the 5-fold value of 0.366 and slightly lower, as expected: each LOO model trains on 9 points instead of 8. The end points x=1,10x=1,10 have the largest leverage, so their residuals are inflated the most (by 53%). The point at x=9x=9 has the largest LOO error, 1.126.

Watch it run

▸ 5-fold cross-validationstep 1/12
fold grid · rows = iterations, columns = samples (time →)1357911131517192012345trainvalidationdata (x = time, y = target)fold error L_j (MSE ×10³) · dashed = running mean, band = ± SE0.002.915.8212345train 2.13
5 folds · n = 20
  1. 1folds = split(range(n), k, scheme)
  2. 2for j in 1..k:
  3. 3 train, val = all ∖ F_j, F_j
  4. 4 f̂ = fit_line(X[train], y[train])
  5. 5 L_j = mean((y[val] − f̂(X[val]))²)
  6. 6CV = mean(L_1 … L_k)
  7. 7SE = std(L) / √k

Shuffle the 20 samples and cut them into 5 folds of about 4. Every column is orange in exactly one row.

fig.20 points in time order, fitted by a straight line (the true trend is slightly curved). Each row of the grid is one round; orange cells are that round's validation fold. Step through: the line is refitted on the blue points only, its errors on the orange points become one bar, and the dashed mean and ± SE band settle as bars accumulate. Default 5-fold: 2.95 ± 0.67 (×10⁻³) vs training MSE 2.13. Reshuffling moves the 5-fold estimate between 2.59 and 3.15 (seeds 1–5). At k = 20 (LOOCV) you get 2.61 ± 0.57, equal to the one-fit shortcut. Switch to time series: 5 expanding windows give 5.07 ± 3.01, because each fold must extrapolate the curve into the future.

Splitting schemes

The averaging is always the same; what changes is how the folds are drawn. The rule: a validation fold should relate to its training folds the way future data will relate to the data you have.

Which splitter

KFold (shuffled)
i.i.d. rows. Always shuffle=True unless the file order is meaningless and random; sorted files (by date, by label) otherwise produce folds that differ systematically.
StratifiedKFold
classification: each fold keeps the overall class ratios. Essential for imbalanced classes. With 5% positives and 100 rows, a plain fold of 20 could contain zero positives and make recall undefined. For regression you can stratify on binned yy, as the figure does.
GroupKFold / StratifiedGroupKFold
rows come in groups (several images per patient, sessions per user, sentences per document). All rows of a group go to the same fold, so the model is never validated on a patient it trained on. LeaveOneGroupOut is the group analogue of LOOCV.
TimeSeriesSplit
ordered data. Fold jj trains on everything before a cut and validates on the next block (expanding window; max_train_size makes it a rolling window). gap drops samples between train and test when features or labels look ahead. The first block is never validated.
Repeated (Stratified)KFold
re-run k-fold with several shuffles and average; reduces the partition noise you see when pressing reshuffle.

Nested cross-validation

If you try 50 hyperparameter settings and report the best CV score, that number is optimistically biased: you selected the maximum of 50 noisy estimates, so part of its height is luck. This is the winner's curse. The selection is part of the learning procedure, so it must be inside the loop that is being scored.

Nested CV (outer k₁ × inner k₂)

  1. 1Outer loop: split into k1k_1 folds. For each outer fold jj, set it aside as the test set.
  2. 2Inner loop: on the remaining data only, run an ordinary k2k_2-fold CV for every hyperparameter setting and pick the best (this is just GridSearchCV).
  3. 3Refit the chosen setting on all of the outer-training data and score it on outer fold jj: this gives LjouterL_j^{\text{outer}}.
  4. 4Average the k1k_1 outer scores. This estimates the performance of "tune by inner CV, then train". The chosen hyperparameters may differ between outer folds, which is fine and shows how stable the choice is.
CVnested=1k1∑j=1k1L(Fj, f^λ^j(−j)),λ^j=arg⁡min⁡λ CVk2(λ; D∖Fj)\mathrm{CV}^{\text{nested}}=\frac{1}{k_1}\sum_{j=1}^{k_1}L\Big(F_j,\ \hat f_{\hat\lambda_j}^{(-j)}\Big),\qquad \hat\lambda_j=\arg\min_{\lambda}\ \mathrm{CV}_{k_2}\big(\lambda;\,D\setminus F_j\big)
λ^j\hat\lambda_j
hyperparameters chosen by inner CV using only D∖FjD\setminus F_j
f^λ^j(−j)\hat f_{\hat\lambda_j}^{(-j)}
model with those hyperparameters refitted on D∖FjD\setminus F_j
k1, k2k_1,\ k_2
outer and inner fold counts; cost ≈ k₁ · k₂ · (grid size) fits

Nested CV evaluates; it does not produce the final model. To deploy, run the inner search once on all the data and refit. Its best_score_ is the optimistic number; the nested score is the one to report. If you have plenty of data, a single held-out test set touched once does the same job more cheaply. See Train/validation/test.

In code

python
1import numpy as np
2
3x = np.arange(1, 11, dtype=float)
4y = np.array([2.1, 3.9, 6.2, 7.8, 9.1, 12.5, 13.8, 15.2, 18.6, 19.4])
5folds = [np.array([j, j + 5]) for j in range(5)] # {1,6}, {2,7}, ... {5,10}
6
7def fit(xs, ys): # least-squares line
8 b1, b0 = np.polyfit(xs, ys, 1)
9 return lambda t: b0 + b1 * t
10
11L = []
12for val in folds:
13 train = np.setdiff1d(np.arange(10), val)
14 f = fit(x[train], y[train]) # fit on train ONLY
15 L.append(np.mean((y[val] - f(x[val])) ** 2)) # score on the held-out fold
16L = np.array(L)
17print(L.round(4)) # [0.3105 0.0042 0.29 0.6324 0.5938]
18print(L.mean(), L.std(ddof=1) / np.sqrt(len(L))) # CV_5 = 0.3662, SE = 0.1146
19
20# LOOCV with one fit: e_i / (1 - h_ii)
21X = np.c_[np.ones(10), x]
22H = X @ np.linalg.solve(X.T @ X, X.T) # hat matrix
23e = y - H @ y
24print(np.mean((e / (1 - np.diag(H))) ** 2)) # 0.3445, same as 10 refits
From scratch: the worked example (5-fold, then LOOCV with the leverage shortcut).
python
1import numpy as np
2from sklearn.datasets import load_breast_cancer
3from sklearn.linear_model import LogisticRegression
4from sklearn.model_selection import (GroupKFold, KFold, StratifiedKFold,
5 TimeSeriesSplit, cross_val_score, cross_validate)
6from sklearn.pipeline import make_pipeline
7from sklearn.preprocessing import StandardScaler
8
9X, y = load_breast_cancer(return_X_y=True)
10model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)) # scaler refit per fold
11
12cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
13scores = cross_val_score(model, X, y, cv=cv, scoring="accuracy")
14print(scores.round(3), f"{scores.mean():.3f} ± {scores.std(ddof=1) / np.sqrt(len(scores)):.3f}")
15# [0.956 0.974 0.982 1. 0.982] 0.979 ± 0.007
16
17# The splitter is the only thing that changes between schemes:
18KFold(n_splits=5, shuffle=True, random_state=0) # i.i.d. regression data
19StratifiedKFold(n_splits=5) # classification: keep class ratios per fold
20GroupKFold(n_splits=5) # pass groups=...: a group never spans train and val
21TimeSeriesSplit(n_splits=5, gap=0) # train on the past, validate on the next block
22
23groups = np.arange(len(y)) // 4 # e.g. 4 scans per patient
24res = cross_validate(model, X, y, groups=groups, cv=GroupKFold(n_splits=5),
25 scoring=["accuracy", "roc_auc"], return_train_score=True)
26print(res["train_accuracy"].mean().round(3), res["test_accuracy"].mean().round(3)) # 0.989 0.975
scikit-learn: put preprocessing inside the pipeline, and choose the splitter to match the data.
python
1import numpy as np
2from sklearn.datasets import load_breast_cancer
3from sklearn.model_selection import GridSearchCV, StratifiedKFold, cross_val_score
4from sklearn.pipeline import make_pipeline
5from sklearn.preprocessing import StandardScaler
6from sklearn.svm import SVC
7
8X, y = load_breast_cancer(return_X_y=True)
9pipe = make_pipeline(StandardScaler(), SVC())
10grid = {"svc__C": [0.1, 1, 10, 100], "svc__gamma": ["scale", 0.01, 0.001]}
11
12inner = StratifiedKFold(5, shuffle=True, random_state=1) # picks C, gamma
13outer = StratifiedKFold(5, shuffle=True, random_state=2) # scores the whole procedure
14search = GridSearchCV(pipe, grid, cv=inner, scoring="accuracy")
15
16nested = cross_val_score(search, X, y, cv=outer) # search re-run inside every outer fold
17print(f"nested CV accuracy {nested.mean():.3f} ± {nested.std(ddof=1) / np.sqrt(5):.3f}") # 0.975 ± 0.012
18
19search.fit(X, y) # final model: search on all data
20print(search.best_params_, f"best inner score {search.best_score_:.3f} (optimistic)") # C=10, gamma=0.01, 0.982
Nested CV: GridSearchCV is the inner loop, cross_val_score the outer one.

Use when

  • +Data is limited and a single validation split would be too small or too noisy.
  • +You need an error bar, or a fair comparison of models on identical folds (use the same cv object, then compare per-fold differences).
  • +Choosing hyperparameters (GridSearchCV, RandomizedSearchCV), and nested CV to report the tuned procedure honestly.
  • +Linear smoothers: LOOCV / GCV is essentially free.

Avoid when

  • −Training is expensive (large deep nets): k fits cost k times as much. Use one held-out validation set instead.
  • −The data is huge: a single 1% validation split already has a tiny standard error.
  • −The folds cannot mimic deployment (distribution shift, new users, the future) unless you pick the matching group or time splitter.
  • −You need the error of the deployed model: CV scores the procedure, averaged over training sets.

Where it's used

Model selection

Choose the regularization strength, tree depth, k in kNN, or number of components by minimizing CV error (often with the one-SE rule).

Small-sample science

Medical, genomics and psychology studies with a few hundred subjects, using grouped and nested CV so subjects never leak.

Forecasting backtests

Rolling-origin evaluation of demand or price forecasts is TimeSeriesSplit under another name.

Stacking

Out-of-fold predictions from cross_val_predict are the honest inputs for training a meta-model.

Check yourself

Five fold losses are 0.3105,0.0042,0.2900,0.6324,0.59380.3105, 0.0042, 0.2900, 0.6324, 0.5938. What are CV5\mathrm{CV}_5 and its standard error?

Check yourself

Compared with 10-fold CV, leave-one-out CV typically has…

Check yourself

An OLS fit on all data has residual ei=0.30e_i=0.30 at a point with leverage hii=0.25h_{ii}=0.25. Its leave-one-out residual is…

Check yourself

You standardize all features, select the 50 most label-correlated ones on the full dataset, then run 10-fold CV. The CV accuracy is…

Remember

  • ✓k-fold CV rotates the held-out fold so each sample is validated exactly once: CVk=1k∑jLj\mathrm{CV}_k=\tfrac1k\sum_j L_j, refit on all data afterwards.
  • ✓Report mean ± sL/ks_L/\sqrt k, but remember fold losses are correlated, so this SE is an optimistic, approximate error bar.
  • ✓Small k is pessimistically biased; LOOCV is nearly unbiased but can have high variance. 5–10 folds is the usual sweet spot; repeat with new shuffles for stability.
  • ✓For linear smoothers LOOCV needs one fit: residual /(1−hii)/(1-h_{ii}); GCV replaces hiih_{ii} by tr⁡H/n\operatorname{tr}H/n.
  • ✓Match the splitter to the data: stratified for classes, group for repeated units, TimeSeriesSplit (never shuffled) for ordered data.
  • ✓Put all preprocessing inside the pipeline, and use nested CV (or an untouched test set) to report a tuned model honestly.
← back to ML Algorithms roadmap