Definition
A train / validation / test split divides the labelled data into three disjoint parts with three different jobs. The training set fits the model's parameters. The validation set compares candidate models and chooses hyperparameters. The test set is used once, at the very end, to estimate how the chosen model will do on new data. Each part can only answer its own question honestly if no later decision was based on it.
At a glance
- Purpose
- Estimate generalization error, and choose a model without fooling yourself
- Typical sizes
- 60/20/20 or 70/15/15 for thousands of rows; 98/1/1 when you have millions, since a 1% test set is still large in absolute terms
- Train set
- fits parameters (weights, split thresholds, centroids)
- Validation set
- chooses hyperparameters (degree, depth, regularization, early-stopping epoch, features)
- Test set
- one final, unbiased estimate. It is never used for a decision
- Variants
- stratified (preserve class mix), grouped (no entity in two splits), time-ordered (train on the past, test on the future)
- Related
- Cross-validation rotates the validation role when data is scarce; Bias–variance trade-off explains the U-shaped validation curve
The problem
A model's error on the data it was trained on is almost useless as a forecast. A degree-10 polynomial or an unpruned decision tree can push training error close to zero while doing badly on the next data point. What we want is the error on data the model has not seen. The only way to measure that is to hide some data from the model.
A second, subtler problem is that we rarely train one model. We try ten degrees, fifty learning rates, three feature sets, and keep the best. Choosing is also a kind of fitting. Whatever data we used to choose is no longer unseen, so the score of the winner on that data is too optimistic. This is why we need two held-out sets and not one.
Three sets, three jobs
The protocol
- 1Split once, before you look. Shuffle (or order by time, or group by entity) and cut the data into train / validation / test. Put the test set away.
- 2Fit on train. For every candidate hyperparameter setting , fit the parameters on the training set only.
- 3Choose on validation. Score every candidate on the validation set and keep . Iterate on features and model families here, as often as you like.
- 4Optionally refit the chosen configuration on train + validation, which gives more data and usually a slightly better model.
- 5Test once. Evaluate the final model on the test set and report that number with its standard error. Do not go back and change the model because of it.
Empirical risk vs true risk
Let be the loss on one example (squared error, 0–1 error, log-loss). What we care about is the expected loss on a fresh draw from the data distribution , the true risk. What we can compute is an average over a finite sample, the empirical risk.
- a fixed, already-trained model (hypothesis)
- true risk: expected loss on new data from the same distribution
- empirical risk: average loss over the sample
- the (unknown) distribution the data comes from
If is drawn from independently of , then each term is an independent draw of the loss. In that case is an unbiased estimate, , and its spread shrinks like . Everything below follows from that small qualification, independently of :
Which estimates are honest
- Training error
- where was fitted to minimize it. It is biased low, often badly.
- Validation error of one model
- unbiased, as long as that single model was fixed before looking at the validation set.
- Validation error of the winner
- biased low: the winner was selected because its validation error was small.
- Test error of the final model
- unbiased, because the test set played no part in fitting or selection.
Why the best validation score is optimistic
Each validation score is the true risk plus noise: . Taking the minimum over candidates favours whichever model drew the most negative noise. The minimum is a concave function, so Jensen's inequality says that on average it lands below the minimum of the true risks:
- the candidate models (one per hyperparameter setting)
- validation error of candidate k
- its true risk
How big is the gap? Take the worst case, where all candidates are equally good, , and their validation noise is independent with standard deviation . Then the expected minimum is minus times the expected maximum of standard normals:
- expected maximum of independent standard normal variables
- standard deviation of the per-example loss
- number of validation examples
The bias grows with the number of things you try (slowly, roughly like ) and shrinks with the size of the validation set. Real candidates are correlated and not all equally good, so the actual gap is smaller than this worst case. The direction is always the same, though: the validation score of the winner overstates how good the winner is. A test set that took no part in the choice removes the bias.
Worked example: choosing a polynomial degree
Sixty points are collected over time, , from with noise . No model can get below the noise floor, so every true risk is at least . We shuffle and split 60/20/20, which gives 36 train, 12 validation and 12 test points. Then we fit polynomials of degree 1–10 by least squares on the 36 training points. Because the data is synthetic we can also compute each fit's true risk exactly, by integrating its squared error against and adding . Real projects never get this column. It is here so we can check the estimates.
| Degree | Train MSE | Validation MSE | True risk (unknowable in practice) |
|---|---|---|---|
| 1 | 0.397 | 0.353 | 0.303 |
| 2 | 0.396 | 0.355 | 0.304 |
| 3 | 0.133 | 0.080 | 0.101 |
| 4 | 0.127 | 0.105 | 0.105 |
| 5 | 0.100 | 0.155 | 0.109 |
| 6 | 0.090 | 0.131 | 0.114 |
| 7 | 0.090 | 0.131 | 0.114 |
| 8 | 0.088 | 0.130 | 0.114 |
| 9 | 0.087 | 0.122 | 0.113 |
| 10 | 0.087 | 0.122 | 0.113 |
Reading the table
- 1Training error is useless for choosing. It falls monotonically from 0.397 to 0.087, so it would always pick the largest degree. A sine needs an odd polynomial of degree at least 3, which is why degrees 1 and 2 are stuck near 0.4.
- 2Validation picks degree 3, with . That also happens to be the degree with the lowest true risk (0.101), so the choice is correct.
- 3But the winning validation score is optimistic. 0.080 is below the true risk of 0.101, and even below the noise floor of 0.09 that no model can beat. Degree 3 won partly because its 12 validation points happened to be easy.
- 4Test once. The degree-3 model on the 12 untouched test points gives squared errors with mean 0.077 and sample standard deviation 0.083, so . We report test MSE = 0.077 ± 0.024. The true value of 0.101 lies about one standard error away. With only 12 test points the estimate is honest (unbiased) but noisy, and here it was lucky too.
The bottom strip of the figure shows selection bias directly. Across 40 reshuffles at 36/12/12, the average gap between the winner's validation score and its true risk is only 0.003. Shrink validation to 3 points and the gap grows to 0.104. The winning validation score becomes optimistic in 31 of the 40 runs, and every one of the 10 degrees wins at least once. A tiny validation set lets you pick a model, but it does not reliably pick a good one, and the score it reports for that model can't be trusted.
How big must the test set be?
For a classifier the per-example loss is 0 or 1, so the test error is a binomial proportion and its standard error has a closed form:
- true error rate (plug in the estimate )
- number of test examples
- desired half-width of the confidence interval
- normal quantile: 1.96 for 95% confidence
| True error p | Half-width h | Test examples needed |
|---|---|---|
| 10% | ±2 points | 865 |
| 10% | ±1 point | 3,458 |
| 5% | ±1 point | 1,825 |
So 1,000 test examples at an 8% error rate give , which means "8% ± 1.7%" at 95%. To halve the interval you need four times as many examples. For losses that are not 0/1, such as the MSE above, use the same idea with the sample standard deviation of the per-example losses: .
Splitting correctly
A random split is only right if the rows are independent and identically distributed and the model will see the same distribution in deployment. When either assumption fails, the split has to copy the conditions the model will face in deployment:
Choose the split to match deployment
- Stratified
- For classification, split within each class so that every set keeps the overall class proportions. With 5% positives and a 100-row validation set, a plain shuffle gives anywhere from about 1 to 9 positives (mean 5, SD 2.2), and some metrics are then meaningless. Stratifying fixes the count. For regression you can stratify on quantile bins of , which is what the figure's stratified mode does with terciles.
- Time-ordered
- If the model will predict the future, train on the past and validate and test on later periods, without shuffling. A shuffled split lets the model learn from the future, for example by interpolating between neighbouring days. It reports an error you will never achieve. Leave a gap between the periods when features use look-back windows. For several folds, see
TimeSeriesSplit. - Grouped
- If rows come in groups (several scans per patient, many sessions per user, frames of one video), every group must fall entirely within one split. Otherwise the model can recognise the patient instead of learning the disease. Use
GroupShuffleSplit(orGroupKFold) with the entity id. - Deduplicated
- Exact and near duplicates (re-uploaded images, copied text, the same customer twice) that land in different splits turn test examples into training examples. Deduplicate before splitting, or group the duplicates together.
In the figure, switching to time-ordered puts the last 12 time points in the test set. Every polynomial now has to extrapolate. Validation chooses degree 1, the only degree that doesn't diverge, and its test MSE is 0.432, more than five times the shuffled estimate. That number is the honest one if the task is forecasting.
Data leakage
Definition
Data leakage is any path by which information from the validation or test set (or from the future, or from the target itself) reaches the model during training or selection. A leaked evaluation looks better than the model will ever perform in deployment.
The most common form is innocent preprocessing. You compute a scaler's mean and standard deviation, a PCA basis, an imputation value, a vocabulary or a target encoding on all the data and then split. The training features now contain statistics of the test rows. The rule is that every fitted step is fitted on the training set only and then applied unchanged to validation and test:
- standardization statistics, computed from training rows only
- the transformed feature fed to the model
Wrap the preprocessing and the model in one scikit-learn Pipeline. Calling fit on the training data then fits every step on that data alone, and the same pipeline object can be passed to cross-validation without leaking across folds.
In code
1import numpy as np2from sklearn.model_selection import train_test_split3from sklearn.pipeline import make_pipeline4from sklearn.preprocessing import PolynomialFeatures, StandardScaler5from sklearn.linear_model import LinearRegression67rng = np.random.default_rng(7)8x = np.sort(rng.uniform(0, 1, 60))9y = np.sin(2 * np.pi * x) + rng.normal(0, 0.3, 60)10X = x[:, None]1112# 60 / 20 / 20: carve off the test set FIRST, then split the remainder 75 / 25.13X_rest, X_test, y_rest, y_test = train_test_split(X, y, test_size=0.20, random_state=3)14X_tr, X_val, y_tr, y_val = train_test_split(X_rest, y_rest, test_size=0.25, random_state=3)15print(len(X_tr), len(X_val), len(X_test)) # 36 12 121617def model(d):18 # The scaler lives inside the pipeline, so its mean/sd are learned from X_tr only.19 return make_pipeline(PolynomialFeatures(d, include_bias=False), StandardScaler(), LinearRegression())2021val_mse = {}22for d in range(1, 11):23 m = model(d).fit(X_tr, y_tr)24 val_mse[d] = np.mean((m.predict(X_val) - y_val) ** 2)25best = min(val_mse, key=val_mse.get)2627# Touch the test set exactly once, with the chosen degree.28final = model(best).fit(X_tr, y_tr)29sq = (final.predict(X_test) - y_test) ** 230print(f"degree {best}: test MSE {sq.mean():.3f} ± {sq.std(ddof=1) / np.sqrt(len(sq)):.3f}")1from sklearn.model_selection import train_test_split, GroupShuffleSplit, TimeSeriesSplit2from sklearn.pipeline import make_pipeline3from sklearn.preprocessing import StandardScaler4from sklearn.linear_model import LogisticRegression56# Stratified 60/20/20: every split keeps the class proportions of y.7X_rest, X_test, y_rest, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=0)8X_tr, X_val, y_tr, y_val = train_test_split(X_rest, y_rest, test_size=0.25, stratify=y_rest, random_state=0)910# Grouped: no patient appears on both sides.11tr_idx, te_idx = next(GroupShuffleSplit(n_splits=1, test_size=0.2, random_state=0)12 .split(X, y, groups=patient_id))1314# Time-ordered: rows sorted by time, no shuffling; the future is held out.15cut1, cut2 = int(0.6 * len(X)), int(0.8 * len(X))16X_past, X_recent, X_future = X[:cut1], X[cut1:cut2], X[cut2:] # train, val, test17# For several expanding-window folds: TimeSeriesSplit(n_splits=5, gap=lookback)1819# LEAKY: the scaler saw the validation and test rows.20X_all = StandardScaler().fit_transform(X) # then split X_all ... (don't)2122# CORRECT: preprocessing is part of the model and is fitted on train only.23clf = make_pipeline(StandardScaler(), LogisticRegression())24clf.fit(X_tr, y_tr)25print(clf.score(X_val, y_val))Use when
- +You have plenty of data: one fixed split is simple, fast, and each model is trained only once per candidate.
- +Training is expensive (deep networks, large ensembles), so k-fold retraining is not affordable.
- +You need an auditable final number: a held-out test set that nobody touched is the clearest evidence you can give.
- +Deployment has a clear structure (a future period, new customers or new sites) that a single split can copy exactly.
Avoid when
- −Data is scarce: a 20% validation set gives a noisy choice and a 20% test set a wide interval. Use cross-validation, or nested cross-validation for the final estimate.
- −One random split is one sample. The selected model and its score can change with the seed, as the reshuffle strip shows.
- −Holding out 40% of the data means the deployed model was trained on less than it could have been. Refit on train + validation after choosing.
- −Any reuse of the test set, even only to check, slowly turns it into a validation set.
Where it's used
Model and hyperparameter selection
Degree, depth, regularization strength, learning rate, architecture: choose on validation, never on test.
Early stopping
Watch validation loss during training and keep the epoch where it was lowest. That epoch is a hyperparameter chosen on validation, so the final number still comes from test.
Benchmarks and competitions
Public leaderboards act as validation sets. Hidden private test sets exist because repeatedly submitting to a public board overfits it.
Forecasting and risk models
Backtesting on later periods is the time-ordered version of a test set, and is often a regulatory requirement.
Check yourself
You try 50 hyperparameter settings and report the best validation accuracy, 91.3%. What does that number most likely do?
Check yourself
A classifier has a 10% error rate. About how many test examples do you need for a 95% interval of ±1 percentage point?
Check yourself
You standardize all 10,000 rows with StandardScaler, then split 80/20 and train. What is wrong?
Check yourself
You predict next month's sales from daily data for 2019–2024. Which split gives an honest estimate?
Remember
- ✓Train fits parameters, validation chooses hyperparameters and models, and test gives one final estimate. Each is honest only for decisions it took no part in.
- ✓Empirical risk on a sample independent of the model is unbiased for the true risk, with standard error . For error rates this is .
- ✓The minimum of many noisy validation scores is optimistically biased. The bias grows with the number of candidates and shrinks with the size of the validation set.
- ✓Test-set size: , about 3,500 examples for ±1 point at a 10% error rate.
- ✓Match the split to deployment: stratify for rare classes, keep time order for forecasting, and split by group when rows share an entity. Remove duplicates before splitting.
- ✓Fit every preprocessing step on training data only, ideally inside a Pipeline, and never tune against the test set.