Train / validation / test

The test set estimates generalization only if it never influenced any modeling decision.

17 min read1736 wordscore0 viewsupdated 2026-10-08

Definition

A train / validation / test split divides the labelled data into three disjoint parts with three different jobs. The training set fits the model's parameters. The validation set compares candidate models and chooses hyperparameters. The test set is used once, at the very end, to estimate how the chosen model will do on new data. Each part can only answer its own question honestly if no later decision was based on it.

At a glance

Purpose
Estimate generalization error, and choose a model without fooling yourself
Typical sizes
60/20/20 or 70/15/15 for thousands of rows; 98/1/1 when you have millions, since a 1% test set is still large in absolute terms
Train set
fits parameters θ\theta (weights, split thresholds, centroids)
Validation set
chooses hyperparameters λ\lambda (degree, depth, regularization, early-stopping epoch, features)
Test set
one final, unbiased estimate. It is never used for a decision
Variants
stratified (preserve class mix), grouped (no entity in two splits), time-ordered (train on the past, test on the future)
Related
Cross-validation rotates the validation role when data is scarce; Bias–variance trade-off explains the U-shaped validation curve

The problem

A model's error on the data it was trained on is almost useless as a forecast. A degree-10 polynomial or an unpruned decision tree can push training error close to zero while doing badly on the next data point. What we want is the error on data the model has not seen. The only way to measure that is to hide some data from the model.

A second, subtler problem is that we rarely train one model. We try ten degrees, fifty learning rates, three feature sets, and keep the best. Choosing is also a kind of fitting. Whatever data we used to choose is no longer unseen, so the score of the winner on that data is too optimistic. This is why we need two held-out sets and not one.

Three sets, three jobs

The protocol

  1. 1Split once, before you look. Shuffle (or order by time, or group by entity) and cut the data into train / validation / test. Put the test set away.
  2. 2Fit on train. For every candidate hyperparameter setting λ\lambda, fit the parameters θ^λ\hat\theta_\lambda on the training set only.
  3. 3Choose on validation. Score every candidate on the validation set and keep λ^=arg⁡min⁡λR^val(θ^λ)\hat\lambda=\arg\min_\lambda \hat R_{\text{val}}(\hat\theta_\lambda). Iterate on features and model families here, as often as you like.
  4. 4Optionally refit the chosen configuration on train + validation, which gives more data and usually a slightly better model.
  5. 5Test once. Evaluate the final model on the test set and report that number with its standard error. Do not go back and change the model because of it.

Empirical risk vs true risk

Let ℓ(y^,y)\ell(\hat y, y) be the loss on one example (squared error, 0–1 error, log-loss). What we care about is the expected loss on a fresh draw from the data distribution D\mathcal D, the true risk. What we can compute is an average over a finite sample, the empirical risk.

R(h)=E(x,y)∼D[ℓ(h(x),y)]R^S(h)=1∣S∣∑(xi,yi)∈Sℓ(h(xi),yi)R(h)=\mathbb{E}_{(x,y)\sim\mathcal D}\big[\ell(h(x),y)\big]\qquad \hat R_S(h)=\frac{1}{|S|}\sum_{(x_i,y_i)\in S}\ell(h(x_i),y_i)
hh
a fixed, already-trained model (hypothesis)
R(h)R(h)
true risk: expected loss on new data from the same distribution
R^S(h)\hat R_S(h)
empirical risk: average loss over the sample SS
D\mathcal D
the (unknown) distribution the data comes from

If SS is drawn from D\mathcal D independently of hh, then each term is an independent draw of the loss. In that case R^S(h)\hat R_S(h) is an unbiased estimate, E[R^S(h)]=R(h)\mathbb E[\hat R_S(h)]=R(h), and its spread shrinks like 1/∣S∣1/\sqrt{|S|}. Everything below follows from that small qualification, independently of hh:

Which estimates are honest

Training error
R^train(h^)\hat R_{\text{train}}(\hat h) where h^\hat h was fitted to minimize it. It is biased low, often badly.
Validation error of one model
unbiased, as long as that single model was fixed before looking at the validation set.
Validation error of the winner
biased low: the winner was selected because its validation error was small.
Test error of the final model
unbiased, because the test set played no part in fitting or selection.

Why the best validation score is optimistic

Each validation score is the true risk plus noise: R^val(hk)=R(hk)+εk\hat R_{\text{val}}(h_k)=R(h_k)+\varepsilon_k. Taking the minimum over KK candidates favours whichever model drew the most negative noise. The minimum is a concave function, so Jensen's inequality says that on average it lands below the minimum of the true risks:

E[min⁡kR^val(hk)]  ≤  min⁡kE[R^val(hk)]=min⁡kR(hk)\mathbb{E}\Big[\min_{k}\hat R_{\text{val}}(h_k)\Big]\;\le\;\min_k \mathbb{E}\big[\hat R_{\text{val}}(h_k)\big]=\min_k R(h_k)
h1,…,hKh_1,\dots,h_K
the candidate models (one per hyperparameter setting)
R^val(hk)\hat R_{\text{val}}(h_k)
validation error of candidate k
R(hk)R(h_k)
its true risk

How big is the gap? Take the worst case, where all KK candidates are equally good, R(hk)=RR(h_k)=R, and their validation noise is independent with standard deviation ss. Then the expected minimum is RR minus ss times the expected maximum of KK standard normals:

E[min⁡kR^val(hk)]≈R−cK s,s=σℓnval,c2=0.56,  c10=1.54,  c100=2.51\mathbb{E}\Big[\min_k \hat R_{\text{val}}(h_k)\Big]\approx R - c_K\,s,\qquad s=\frac{\sigma_\ell}{\sqrt{n_{\text{val}}}},\qquad c_2=0.56,\;c_{10}=1.54,\;c_{100}=2.51
cKc_K
expected maximum of KK independent standard normal variables
σℓ\sigma_\ell
standard deviation of the per-example loss
nvaln_{\text{val}}
number of validation examples

The bias grows with the number of things you try (slowly, roughly like 2ln⁡K\sqrt{2\ln K}) and shrinks with the size of the validation set. Real candidates are correlated and not all equally good, so the actual gap is smaller than this worst case. The direction is always the same, though: the validation score of the winner overstates how good the winner is. A test set that took no part in the choice removes the bias.

Worked example: choosing a polynomial degree

Sixty points are collected over time, x∈[0,1]x\in[0,1], from y=sin⁡(2πx)+εy=\sin(2\pi x)+\varepsilon with noise σ=0.3\sigma=0.3. No model can get below the noise floor, so every true risk is at least σ2=0.09\sigma^2=0.09. We shuffle and split 60/20/20, which gives 36 train, 12 validation and 12 test points. Then we fit polynomials of degree 1–10 by least squares on the 36 training points. Because the data is synthetic we can also compute each fit's true risk exactly, by integrating its squared error against sin⁡(2πx)\sin(2\pi x) and adding σ2\sigma^2. Real projects never get this column. It is here so we can check the estimates.

DegreeTrain MSEValidation MSETrue risk (unknowable in practice)
10.3970.3530.303
20.3960.3550.304
30.1330.0800.101
40.1270.1050.105
50.1000.1550.109
60.0900.1310.114
70.0900.1310.114
80.0880.1300.114
90.0870.1220.113
100.0870.1220.113
Mean squared error of each degree, all fitted on the same 36 training points. Highlighted: the validation minimum, degree 3. Training error keeps falling as the degree rises; validation error does not.

Reading the table

  1. 1Training error is useless for choosing. It falls monotonically from 0.397 to 0.087, so it would always pick the largest degree. A sine needs an odd polynomial of degree at least 3, which is why degrees 1 and 2 are stuck near 0.4.
  2. 2Validation picks degree 3, with R^val=0.080\hat R_{\text{val}}=0.080. That also happens to be the degree with the lowest true risk (0.101), so the choice is correct.
  3. 3But the winning validation score is optimistic. 0.080 is below the true risk of 0.101, and even below the noise floor of 0.09 that no model can beat. Degree 3 won partly because its 12 validation points happened to be easy.
  4. 4Test once. The degree-3 model on the 12 untouched test points gives squared errors with mean 0.077 and sample standard deviation 0.083, so SE=0.083/12=0.024\text{SE}=0.083/\sqrt{12}=0.024. We report test MSE = 0.077 ± 0.024. The true value of 0.101 lies about one standard error away. With only 12 test points the estimate is honest (unbiased) but noisy, and here it was lucky too.
60 samples, shuffled (bar = y) · drag the handlestrain 36 · 60%val 12 · 20%test 12 · 20%data · chosen fit (degree 3) · true f dashed-10100.51x (collection time) →MSE vs degree: train · val · true risk0.010.030.10.313σ² floor12345678910polynomial degree →same sizes, 40 reshuffles: the winner's val MSE vs its true risk (9 different degrees won)winner's val MSEits true risk00.10.20.3≥0.4MSE · bars = means · ringed dot = this split
|
chosen degree 3 · val MSE 0.080 · true risk 0.101
test MSE (n = 12) 0.077 ± 0.024 (1 SE)
y-terciles low/mid/high · train 10/15/11 · val 5/2/5 · test 5/3/4
over 40 reshuffles: mean winner val 0.113 vs true 0.116 · optimism 0.004
fig.Sixty samples as bars (height = y), coloured by split. Drag the two handles or use the sliders to resize the splits. The middle-right panel fits degrees 1–10 on the train bars; the ring marks the validation minimum and the green diamond is the single test evaluation with ±1 SE. The bottom strip repeats the whole procedure on 40 reshuffles of the same sizes. With 12 validation points the winner's validation MSE averages 0.113 against a true risk of 0.116. Click “tiny val (3)” and the winner's validation MSE drops to an average of 0.085, below the 0.09 noise floor, while its true risk rises to 0.188. Switch to time-ordered and the polynomials have to extrapolate into the future, so the validation and test errors explode.

The bottom strip of the figure shows selection bias directly. Across 40 reshuffles at 36/12/12, the average gap between the winner's validation score and its true risk is only 0.003. Shrink validation to 3 points and the gap grows to 0.104. The winning validation score becomes optimistic in 31 of the 40 runs, and every one of the 10 degrees wins at least once. A tiny validation set lets you pick a model, but it does not reliably pick a good one, and the score it reports for that model can't be trusted.

How big must the test set be?

For a classifier the per-example loss is 0 or 1, so the test error p^\hat p is a binomial proportion and its standard error has a closed form:

SE(p^)=p(1−p)n⟹n  ≥  z2 p(1−p)h2\text{SE}(\hat p)=\sqrt{\frac{p(1-p)}{n}}\qquad\Longrightarrow\qquad n\;\ge\;\frac{z^2\,p(1-p)}{h^2}
pp
true error rate (plug in the estimate p^\hat p)
nn
number of test examples
hh
desired half-width of the confidence interval p^±h\hat p\pm h
zz
normal quantile: 1.96 for 95% confidence
True error pHalf-width hTest examples needed
10%±2 points865
10%±1 point3,458
5%±1 point1,825
Test-set size needed for a 95% interval of half-width h.

So 1,000 test examples at an 8% error rate give SE=0.08⋅0.92/1000=0.0086\text{SE}=\sqrt{0.08\cdot0.92/1000}=0.0086, which means "8% ± 1.7%" at 95%. To halve the interval you need four times as many examples. For losses that are not 0/1, such as the MSE above, use the same idea with the sample standard deviation of the per-example losses: SE=sℓ/n\text{SE}=s_\ell/\sqrt{n}.

Splitting correctly

A random split is only right if the rows are independent and identically distributed and the model will see the same distribution in deployment. When either assumption fails, the split has to copy the conditions the model will face in deployment:

Choose the split to match deployment

Stratified
For classification, split within each class so that every set keeps the overall class proportions. With 5% positives and a 100-row validation set, a plain shuffle gives anywhere from about 1 to 9 positives (mean 5, SD 2.2), and some metrics are then meaningless. Stratifying fixes the count. For regression you can stratify on quantile bins of yy, which is what the figure's stratified mode does with terciles.
Time-ordered
If the model will predict the future, train on the past and validate and test on later periods, without shuffling. A shuffled split lets the model learn from the future, for example by interpolating between neighbouring days. It reports an error you will never achieve. Leave a gap between the periods when features use look-back windows. For several folds, see TimeSeriesSplit.
Grouped
If rows come in groups (several scans per patient, many sessions per user, frames of one video), every group must fall entirely within one split. Otherwise the model can recognise the patient instead of learning the disease. Use GroupShuffleSplit (or GroupKFold) with the entity id.
Deduplicated
Exact and near duplicates (re-uploaded images, copied text, the same customer twice) that land in different splits turn test examples into training examples. Deduplicate before splitting, or group the duplicates together.

In the figure, switching to time-ordered puts the last 12 time points in the test set. Every polynomial now has to extrapolate. Validation chooses degree 1, the only degree that doesn't diverge, and its test MSE is 0.432, more than five times the shuffled estimate. That number is the honest one if the task is forecasting.

Data leakage

Definition

Data leakage is any path by which information from the validation or test set (or from the future, or from the target itself) reaches the model during training or selection. A leaked evaluation looks better than the model will ever perform in deployment.

The most common form is innocent preprocessing. You compute a scaler's mean and standard deviation, a PCA basis, an imputation value, a vocabulary or a target encoding on all the data and then split. The training features now contain statistics of the test rows. The rule is that every fitted step is fitted on the training set only and then applied unchanged to validation and test:

μ=1ntr∑i∈trainxi,σ2=1ntr∑i∈train(xi−μ)2,z=x−μσ for train, val and test alike\mu=\frac{1}{n_{\text{tr}}}\sum_{i\in\text{train}}x_i,\qquad \sigma^2=\frac{1}{n_{\text{tr}}}\sum_{i\in\text{train}}(x_i-\mu)^2,\qquad z=\frac{x-\mu}{\sigma}\ \text{for train, val and test alike}
μ, σ\mu,\ \sigma
standardization statistics, computed from training rows only
zz
the transformed feature fed to the model

Wrap the preprocessing and the model in one scikit-learn Pipeline. Calling fit on the training data then fits every step on that data alone, and the same pipeline object can be passed to cross-validation without leaking across folds.

In code

python
1import numpy as np
2from sklearn.model_selection import train_test_split
3from sklearn.pipeline import make_pipeline
4from sklearn.preprocessing import PolynomialFeatures, StandardScaler
5from sklearn.linear_model import LinearRegression
6
7rng = np.random.default_rng(7)
8x = np.sort(rng.uniform(0, 1, 60))
9y = np.sin(2 * np.pi * x) + rng.normal(0, 0.3, 60)
10X = x[:, None]
11
12# 60 / 20 / 20: carve off the test set FIRST, then split the remainder 75 / 25.
13X_rest, X_test, y_rest, y_test = train_test_split(X, y, test_size=0.20, random_state=3)
14X_tr, X_val, y_tr, y_val = train_test_split(X_rest, y_rest, test_size=0.25, random_state=3)
15print(len(X_tr), len(X_val), len(X_test)) # 36 12 12
16
17def model(d):
18 # The scaler lives inside the pipeline, so its mean/sd are learned from X_tr only.
19 return make_pipeline(PolynomialFeatures(d, include_bias=False), StandardScaler(), LinearRegression())
20
21val_mse = {}
22for d in range(1, 11):
23 m = model(d).fit(X_tr, y_tr)
24 val_mse[d] = np.mean((m.predict(X_val) - y_val) ** 2)
25best = min(val_mse, key=val_mse.get)
26
27# Touch the test set exactly once, with the chosen degree.
28final = model(best).fit(X_tr, y_tr)
29sq = (final.predict(X_test) - y_test) ** 2
30print(f"degree {best}: test MSE {sq.mean():.3f} ± {sq.std(ddof=1) / np.sqrt(len(sq)):.3f}")
The worked example's procedure with scikit-learn. This uses NumPy's random generator instead of the site's, so the data and numbers differ slightly: this run picks degree 4 with test MSE 0.061 ± 0.033.
python
1from sklearn.model_selection import train_test_split, GroupShuffleSplit, TimeSeriesSplit
2from sklearn.pipeline import make_pipeline
3from sklearn.preprocessing import StandardScaler
4from sklearn.linear_model import LogisticRegression
5
6# Stratified 60/20/20: every split keeps the class proportions of y.
7X_rest, X_test, y_rest, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=0)
8X_tr, X_val, y_tr, y_val = train_test_split(X_rest, y_rest, test_size=0.25, stratify=y_rest, random_state=0)
9
10# Grouped: no patient appears on both sides.
11tr_idx, te_idx = next(GroupShuffleSplit(n_splits=1, test_size=0.2, random_state=0)
12 .split(X, y, groups=patient_id))
13
14# Time-ordered: rows sorted by time, no shuffling; the future is held out.
15cut1, cut2 = int(0.6 * len(X)), int(0.8 * len(X))
16X_past, X_recent, X_future = X[:cut1], X[cut1:cut2], X[cut2:] # train, val, test
17# For several expanding-window folds: TimeSeriesSplit(n_splits=5, gap=lookback)
18
19# LEAKY: the scaler saw the validation and test rows.
20X_all = StandardScaler().fit_transform(X) # then split X_all ... (don't)
21
22# CORRECT: preprocessing is part of the model and is fitted on train only.
23clf = make_pipeline(StandardScaler(), LogisticRegression())
24clf.fit(X_tr, y_tr)
25print(clf.score(X_val, y_val))
Stratified, grouped and time-ordered splits, and the leakage bug next to its fix.

Use when

  • +You have plenty of data: one fixed split is simple, fast, and each model is trained only once per candidate.
  • +Training is expensive (deep networks, large ensembles), so k-fold retraining is not affordable.
  • +You need an auditable final number: a held-out test set that nobody touched is the clearest evidence you can give.
  • +Deployment has a clear structure (a future period, new customers or new sites) that a single split can copy exactly.

Avoid when

  • −Data is scarce: a 20% validation set gives a noisy choice and a 20% test set a wide interval. Use cross-validation, or nested cross-validation for the final estimate.
  • −One random split is one sample. The selected model and its score can change with the seed, as the reshuffle strip shows.
  • −Holding out 40% of the data means the deployed model was trained on less than it could have been. Refit on train + validation after choosing.
  • −Any reuse of the test set, even only to check, slowly turns it into a validation set.

Where it's used

Model and hyperparameter selection

Degree, depth, regularization strength, learning rate, architecture: choose on validation, never on test.

Early stopping

Watch validation loss during training and keep the epoch where it was lowest. That epoch is a hyperparameter chosen on validation, so the final number still comes from test.

Benchmarks and competitions

Public leaderboards act as validation sets. Hidden private test sets exist because repeatedly submitting to a public board overfits it.

Forecasting and risk models

Backtesting on later periods is the time-ordered version of a test set, and is often a regulatory requirement.

Check yourself

You try 50 hyperparameter settings and report the best validation accuracy, 91.3%. What does that number most likely do?

Check yourself

A classifier has a 10% error rate. About how many test examples do you need for a 95% interval of ±1 percentage point?

Check yourself

You standardize all 10,000 rows with StandardScaler, then split 80/20 and train. What is wrong?

Check yourself

You predict next month's sales from daily data for 2019–2024. Which split gives an honest estimate?

Remember

  • ✓Train fits parameters, validation chooses hyperparameters and models, and test gives one final estimate. Each is honest only for decisions it took no part in.
  • ✓Empirical risk on a sample independent of the model is unbiased for the true risk, with standard error sℓ/ns_\ell/\sqrt n. For error rates this is p(1−p)/n\sqrt{p(1-p)/n}.
  • ✓The minimum of many noisy validation scores is optimistically biased. The bias grows with the number of candidates and shrinks with the size of the validation set.
  • ✓Test-set size: n≥z2p(1−p)/h2n\ge z^2p(1-p)/h^2, about 3,500 examples for ±1 point at a 10% error rate.
  • ✓Match the split to deployment: stratify for rare classes, keep time order for forecasting, and split by group when rows share an entity. Remove duplicates before splitting.
  • ✓Fit every preprocessing step on training data only, ideally inside a Pipeline, and never tune against the test set.
← back to ML Algorithms roadmap