Definition
k-fold cross-validation splits the data into disjoint folds, trains the model times (each time on folds) and scores it on the fold that was left out. The average of the held-out scores estimates how well this modelling procedure does on unseen data. Every sample is validated exactly once and used for training times.
At a glance
- Purpose
- Estimate generalization error; compare models; choose hyperparameters
- Input → output
- data, a learning procedure and → fold scores, their mean and standard error
- Cost
- model fits (LOOCV: fits, or one fit for linear smoothers)
- Typical k
- 5 or 10; LOOCV (k = n) for very small data or cheap closed-form models
- Variants
- stratified (class ratios), group (no group in two folds), time series (train on the past), repeated, nested
- Assumes
- folds are exchangeable: each one looks like the future data, and nothing leaks between train and validation
The problem
Training error is optimistic: the model has seen those points. A single held-out validation set fixes the bias but wastes data and is noisy: with 200 samples a 20% split scores the model on just 40 points, and a different random split can easily move accuracy by several points. On small or medium datasets we want an estimate that uses every point for validation and every point for training, without ever scoring a model on data it was fitted to.
Cross-validation does this by rotation. Cut the data into folds. In round , hold out fold , fit on the rest, score on fold . After rounds every point has been predicted exactly once by a model that never saw it.
k-fold cross-validation
- 1Shuffle the indices (unless order matters, e.g. time) and split them into folds of nearly equal size.
- 2For : fit the whole pipeline (scaling, feature selection, model) on .
- 3Score that fitted model on only, giving the fold loss .
- 4Report the mean and its standard error. Then, for deployment, refit once on all the data.
The estimate and its uncertainty
- indices in fold
- model fitted on every fold except
- per-sample loss (squared error, 0–1 loss, log-loss …)
- average loss on fold
- the -fold estimate of test error
With equal fold sizes this is the same as averaging the loss over all out-of-fold predictions. (scikit-learn's cross_val_score returns the fold scores, and you average them; with unequal folds the two averages differ slightly.)
- sample standard deviation of the k fold losses
- standard error of the fold mean: the usual error bar on a CV score
This formula treats the as independent. They are not. Any two training sets share of their folds, so the fitted models are strongly correlated, and with them the fold losses. Write for the variance of one and for the correlation between two of them. Then
- variance of a single fold loss
- average correlation between two fold losses (positive: overlapping training sets)
With the true variance is larger than , so under-states the uncertainty. Bengio and Grandvalet (2004) showed that no unbiased estimator of this variance exists from a single run of k-fold CV. Read the SE as a rough, optimistic error bar. It is still useful (the "one-standard-error rule" picks the simplest model within one SE of the best), but two models whose CV scores differ by less than about one SE are effectively tied.
Choosing k: bias versus variance
Each fold model sees a fraction of the data. Learning curves usually still rise at that point, so fold models are slightly worse than the final model and is pessimistically biased. The bias is largest for small : 2-fold trains on only half the data.
| k | Train size per fold | Bias (pessimism) | Variance of the estimate | Fits |
|---|---|---|---|---|
| 2 | n/2 | high | moderate (few, noisy folds) | 2 |
| 5–10 | 80–90% of n | small | moderate | 5–10 |
| n (LOOCV) | n − 1 | almost none | often high: the n models are nearly identical, so their errors are highly correlated | n (or 1) |
LOOCV is nearly unbiased, because each model trains on points. But its training sets differ by a single point, so the fold losses are almost perfectly correlated ( above), and its variance does not shrink the way would suggest. For unstable learners (deep trees, 1-NN) it can be quite noisy. Each fold also scores just one point, so with 0–1 loss every is 0 or 1. The empirical consensus (Breiman and Spector 1992; Kohavi 1995) is that 5- or 10-fold gives the best trade-off. Repeated k-fold (e.g. 5 × 2 or 10 × 10 with different shuffles) averages away the luck of one particular partition.
The LOOCV shortcut for linear smoothers
For any model whose fitted values are a fixed linear function of , , you don't need to refit times. This covers least squares, ridge, polynomial and spline regression, and kernel smoothers with fixed bandwidth. Deleting point and refitting gives exactly:
- hat (smoother) matrix; for OLS , for ridge
- fitted value from the single fit on all n points
- leverage of point : the -th diagonal of , between 0 and 1
High-leverage points (far from ) have close to 1, so their residuals are inflated the most. That is right, because the full fit was pulled toward them. For a straight line, . Replacing each by the average gives generalized cross-validation (GCV), . RidgeCV uses this trick to try many for the cost of one fit.
Worked example: 10 points, 5 folds, a straight line
Fit to these 10 points with squared-error loss. The folds pair each point with the one five positions later: , , …, . This is one particular shuffle; it keeps every fold spread across the range of .
| x | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |
|---|---|---|---|---|---|---|---|---|---|---|
| y | 2.1 | 3.9 | 6.2 | 7.8 | 9.1 | 12.5 | 13.8 | 15.2 | 18.6 | 19.4 |
| fold | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 |
| Fold j | Held out x | Line fitted on the other 8 | Predictions | Residuals | L_j (MSE) |
|---|---|---|---|---|---|
| 1 | 1, 6 | −0.120 + 1.978x | 1.858, 11.750 | 0.242, 0.750 | 0.3105 |
| 2 | 2, 7 | 0.056 + 1.966x | 3.989, 13.820 | −0.089, −0.020 | 0.0042 |
| 3 | 3, 8 | −0.100 + 2.000x | 5.900, 15.900 | 0.300, −0.700 | 0.2900 |
| 4 | 4, 9 | 0.191 + 1.921x | 7.874, 17.478 | −0.074, 1.122 | 0.6324 |
| 5 | 5, 10 | 0.029 + 1.997x | 10.012, 19.996 | −0.912, −0.596 | 0.5938 |
Mean and standard error
- 1.
- 2Deviations from the mean: ; , so .
- 3. Report . The fold losses range from 0.004 to 0.632 just from which two points land together.
- 4Compare with the training MSE of 0.229: the honest estimate is 60% higher.
Now LOOCV with the shortcut: one fit on all 10 points, then divide each residual by . Here and , so for , .
| x | residual eᵢ | leverage hᵢᵢ | eᵢ / (1 − hᵢᵢ) | brute force |
|---|---|---|---|---|
| 1 | 0.1036 | 0.3455 | 0.1583 | 0.1583 |
| 2 | −0.0661 | 0.2485 | −0.0879 | −0.0879 |
| 3 | 0.2642 | 0.1758 | 0.3206 | 0.3206 |
| 4 | −0.1055 | 0.1273 | −0.1208 | −0.1208 |
| 5 | −0.7752 | 0.1030 | −0.8642 | −0.8642 |
| 6 | 0.6552 | 0.1030 | 0.7304 | 0.7304 |
| 7 | −0.0145 | 0.1273 | −0.0167 | −0.0167 |
| 8 | −0.5842 | 0.1758 | −0.7088 | −0.7088 |
| 9 | 0.8461 | 0.2485 | 1.1258 | 1.1258 |
| 10 | −0.3236 | 0.3455 | −0.4944 | −0.4944 |
Squaring and averaging the last column gives (SE over the 10 single-point losses). It is close to the 5-fold value of 0.366 and slightly lower, as expected: each LOO model trains on 9 points instead of 8. The end points have the largest leverage, so their residuals are inflated the most (by 53%). The point at has the largest LOO error, 1.126.
Watch it run
- 1folds = split(range(n), k, scheme)
- 2for j in 1..k:
- 3 train, val = all ∖ F_j, F_j
- 4 f̂ = fit_line(X[train], y[train])
- 5 L_j = mean((y[val] − f̂(X[val]))²)
- 6CV = mean(L_1 … L_k)
- 7SE = std(L) / √k
Shuffle the 20 samples and cut them into 5 folds of about 4. Every column is orange in exactly one row.
Splitting schemes
The averaging is always the same; what changes is how the folds are drawn. The rule: a validation fold should relate to its training folds the way future data will relate to the data you have.
Which splitter
- KFold (shuffled)
- i.i.d. rows. Always
shuffle=Trueunless the file order is meaningless and random; sorted files (by date, by label) otherwise produce folds that differ systematically. - StratifiedKFold
- classification: each fold keeps the overall class ratios. Essential for imbalanced classes. With 5% positives and 100 rows, a plain fold of 20 could contain zero positives and make recall undefined. For regression you can stratify on binned , as the figure does.
- GroupKFold / StratifiedGroupKFold
- rows come in groups (several images per patient, sessions per user, sentences per document). All rows of a group go to the same fold, so the model is never validated on a patient it trained on.
LeaveOneGroupOutis the group analogue of LOOCV. - TimeSeriesSplit
- ordered data. Fold trains on everything before a cut and validates on the next block (expanding window;
max_train_sizemakes it a rolling window).gapdrops samples between train and test when features or labels look ahead. The first block is never validated. - Repeated (Stratified)KFold
- re-run k-fold with several shuffles and average; reduces the partition noise you see when pressing reshuffle.
Nested cross-validation
If you try 50 hyperparameter settings and report the best CV score, that number is optimistically biased: you selected the maximum of 50 noisy estimates, so part of its height is luck. This is the winner's curse. The selection is part of the learning procedure, so it must be inside the loop that is being scored.
Nested CV (outer k₁ × inner k₂)
- 1Outer loop: split into folds. For each outer fold , set it aside as the test set.
- 2Inner loop: on the remaining data only, run an ordinary -fold CV for every hyperparameter setting and pick the best (this is just
GridSearchCV). - 3Refit the chosen setting on all of the outer-training data and score it on outer fold : this gives .
- 4Average the outer scores. This estimates the performance of "tune by inner CV, then train". The chosen hyperparameters may differ between outer folds, which is fine and shows how stable the choice is.
- hyperparameters chosen by inner CV using only
- model with those hyperparameters refitted on
- outer and inner fold counts; cost ≈ k₁ · k₂ · (grid size) fits
Nested CV evaluates; it does not produce the final model. To deploy, run the inner search once on all the data and refit. Its best_score_ is the optimistic number; the nested score is the one to report. If you have plenty of data, a single held-out test set touched once does the same job more cheaply. See Train/validation/test.
In code
1import numpy as np23x = np.arange(1, 11, dtype=float)4y = np.array([2.1, 3.9, 6.2, 7.8, 9.1, 12.5, 13.8, 15.2, 18.6, 19.4])5folds = [np.array([j, j + 5]) for j in range(5)] # {1,6}, {2,7}, ... {5,10}67def fit(xs, ys): # least-squares line8 b1, b0 = np.polyfit(xs, ys, 1)9 return lambda t: b0 + b1 * t1011L = []12for val in folds:13 train = np.setdiff1d(np.arange(10), val)14 f = fit(x[train], y[train]) # fit on train ONLY15 L.append(np.mean((y[val] - f(x[val])) ** 2)) # score on the held-out fold16L = np.array(L)17print(L.round(4)) # [0.3105 0.0042 0.29 0.6324 0.5938]18print(L.mean(), L.std(ddof=1) / np.sqrt(len(L))) # CV_5 = 0.3662, SE = 0.11461920# LOOCV with one fit: e_i / (1 - h_ii)21X = np.c_[np.ones(10), x]22H = X @ np.linalg.solve(X.T @ X, X.T) # hat matrix23e = y - H @ y24print(np.mean((e / (1 - np.diag(H))) ** 2)) # 0.3445, same as 10 refits1import numpy as np2from sklearn.datasets import load_breast_cancer3from sklearn.linear_model import LogisticRegression4from sklearn.model_selection import (GroupKFold, KFold, StratifiedKFold,5 TimeSeriesSplit, cross_val_score, cross_validate)6from sklearn.pipeline import make_pipeline7from sklearn.preprocessing import StandardScaler89X, y = load_breast_cancer(return_X_y=True)10model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)) # scaler refit per fold1112cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)13scores = cross_val_score(model, X, y, cv=cv, scoring="accuracy")14print(scores.round(3), f"{scores.mean():.3f} ± {scores.std(ddof=1) / np.sqrt(len(scores)):.3f}")15# [0.956 0.974 0.982 1. 0.982] 0.979 ± 0.0071617# The splitter is the only thing that changes between schemes:18KFold(n_splits=5, shuffle=True, random_state=0) # i.i.d. regression data19StratifiedKFold(n_splits=5) # classification: keep class ratios per fold20GroupKFold(n_splits=5) # pass groups=...: a group never spans train and val21TimeSeriesSplit(n_splits=5, gap=0) # train on the past, validate on the next block2223groups = np.arange(len(y)) // 4 # e.g. 4 scans per patient24res = cross_validate(model, X, y, groups=groups, cv=GroupKFold(n_splits=5),25 scoring=["accuracy", "roc_auc"], return_train_score=True)26print(res["train_accuracy"].mean().round(3), res["test_accuracy"].mean().round(3)) # 0.989 0.9751import numpy as np2from sklearn.datasets import load_breast_cancer3from sklearn.model_selection import GridSearchCV, StratifiedKFold, cross_val_score4from sklearn.pipeline import make_pipeline5from sklearn.preprocessing import StandardScaler6from sklearn.svm import SVC78X, y = load_breast_cancer(return_X_y=True)9pipe = make_pipeline(StandardScaler(), SVC())10grid = {"svc__C": [0.1, 1, 10, 100], "svc__gamma": ["scale", 0.01, 0.001]}1112inner = StratifiedKFold(5, shuffle=True, random_state=1) # picks C, gamma13outer = StratifiedKFold(5, shuffle=True, random_state=2) # scores the whole procedure14search = GridSearchCV(pipe, grid, cv=inner, scoring="accuracy")1516nested = cross_val_score(search, X, y, cv=outer) # search re-run inside every outer fold17print(f"nested CV accuracy {nested.mean():.3f} ± {nested.std(ddof=1) / np.sqrt(5):.3f}") # 0.975 ± 0.0121819search.fit(X, y) # final model: search on all data20print(search.best_params_, f"best inner score {search.best_score_:.3f} (optimistic)") # C=10, gamma=0.01, 0.982Use when
- +Data is limited and a single validation split would be too small or too noisy.
- +You need an error bar, or a fair comparison of models on identical folds (use the same
cvobject, then compare per-fold differences). - +Choosing hyperparameters (
GridSearchCV,RandomizedSearchCV), and nested CV to report the tuned procedure honestly. - +Linear smoothers: LOOCV / GCV is essentially free.
Avoid when
- −Training is expensive (large deep nets): k fits cost k times as much. Use one held-out validation set instead.
- −The data is huge: a single 1% validation split already has a tiny standard error.
- −The folds cannot mimic deployment (distribution shift, new users, the future) unless you pick the matching group or time splitter.
- −You need the error of the deployed model: CV scores the procedure, averaged over training sets.
Where it's used
Model selection
Choose the regularization strength, tree depth, k in kNN, or number of components by minimizing CV error (often with the one-SE rule).
Small-sample science
Medical, genomics and psychology studies with a few hundred subjects, using grouped and nested CV so subjects never leak.
Forecasting backtests
Rolling-origin evaluation of demand or price forecasts is TimeSeriesSplit under another name.
Stacking
Out-of-fold predictions from cross_val_predict are the honest inputs for training a meta-model.
Check yourself
Five fold losses are . What are and its standard error?
Check yourself
Compared with 10-fold CV, leave-one-out CV typically has…
Check yourself
An OLS fit on all data has residual at a point with leverage . Its leave-one-out residual is…
Check yourself
You standardize all features, select the 50 most label-correlated ones on the full dataset, then run 10-fold CV. The CV accuracy is…
Remember
- ✓k-fold CV rotates the held-out fold so each sample is validated exactly once: , refit on all data afterwards.
- ✓Report mean ± , but remember fold losses are correlated, so this SE is an optimistic, approximate error bar.
- ✓Small k is pessimistically biased; LOOCV is nearly unbiased but can have high variance. 5–10 folds is the usual sweet spot; repeat with new shuffles for stability.
- ✓For linear smoothers LOOCV needs one fit: residual ; GCV replaces by .
- ✓Match the splitter to the data: stratified for classes, group for repeated units, TimeSeriesSplit (never shuffled) for ordered data.
- ✓Put all preprocessing inside the pipeline, and use nested CV (or an untouched test set) to report a tuned model honestly.