# Train / validation / test

> The test set estimates generalization only if it never influenced any modeling decision.

Source: https://www.mlfoundry.in/learn/train-validation-test  
Section: ML Algorithms › Evaluation & Generalization  
Level: core · 17 min read · updated 2026-10-08

**A train / validation / test split:** divides the labelled data into three disjoint parts with three different jobs. The **training set** fits the model's parameters. The **validation set** compares candidate models and chooses hyperparameters. The **test set** is used **once**, at the very end, to estimate how the chosen model will do on new data. Each part can only answer its own question honestly if no later decision was based on it.

- **Purpose:** Estimate generalization error, and choose a model without fooling yourself
- **Typical sizes:** 60/20/20 or 70/15/15 for thousands of rows; 98/1/1 when you have millions, since a 1% test set is still large in absolute terms
- **Train set:** fits parameters $\theta$ (weights, split thresholds, centroids)
- **Validation set:** chooses hyperparameters $\lambda$ (degree, depth, regularization, early-stopping epoch, features)
- **Test set:** one final, unbiased estimate. It is never used for a decision
- **Variants:** stratified (preserve class mix), grouped (no entity in two splits), time-ordered (train on the past, test on the future)
- **Related:** **Cross-validation** rotates the validation role when data is scarce; **Bias–variance trade-off** explains the U-shaped validation curve

## The problem

A model's error on the data it was trained on is almost useless as a forecast. A degree-10 polynomial or an unpruned decision tree can push training error close to zero while doing badly on the next data point. What we want is the error on **data the model has not seen**. The only way to measure that is to hide some data from the model.

A second, subtler problem is that we rarely train one model. We try ten degrees, fifty learning rates, three feature sets, and keep the best. Choosing is also a kind of fitting. Whatever data we used to choose is no longer unseen, so the score of the winner on that data is too optimistic. This is why we need **two** held-out sets and not one.

## Three sets, three jobs

**The protocol**

1. **Split once, before you look.** Shuffle (or order by time, or group by entity) and cut the data into train / validation / test. Put the test set away.
2. **Fit on train.** For every candidate hyperparameter setting $\lambda$, fit the parameters $\hat\theta_\lambda$ on the training set only.
3. **Choose on validation.** Score every candidate on the validation set and keep $\hat\lambda=\arg\min_\lambda \hat R_{\text{val}}(\hat\theta_\lambda)$. Iterate on features and model families here, as often as you like.
4. **Optionally refit** the chosen configuration on train + validation, which gives more data and usually a slightly better model.
5. **Test once.** Evaluate the final model on the test set and report that number with its standard error. Do not go back and change the model because of it.

> **Why the test set must stay untouched:** If you look at the test score, decide that it is disappointing, and change something, the test set has become a second validation set. Its score is then optimistically biased in exactly the way the validation score is. There is nothing statistically special about "test" data. A set is a test set only as long as you have **never acted on it**.

## Empirical risk vs true risk

Let $\ell(\hat y, y)$ be the loss on one example (squared error, 0–1 error, log-loss). What we care about is the expected loss on a fresh draw from the data distribution $\mathcal D$, the **true risk**. What we can compute is an average over a finite sample, the **empirical risk**.

$$
R(h)=\mathbb{E}_{(x,y)\sim\mathcal D}\big[\ell(h(x),y)\big]\qquad \hat R_S(h)=\frac{1}{|S|}\sum_{(x_i,y_i)\in S}\ell(h(x_i),y_i)
$$

- $h$: a fixed, already-trained model (hypothesis)
- $R(h)$: true risk: expected loss on new data from the same distribution
- $\hat R_S(h)$: empirical risk: average loss over the sample $S$
- $\mathcal D$: the (unknown) distribution the data comes from

If $S$ is drawn from $\mathcal D$ **independently of $h$**, then each term is an independent draw of the loss. In that case $\hat R_S(h)$ is an unbiased estimate, $\mathbb E[\hat R_S(h)]=R(h)$, and its spread shrinks like $1/\sqrt{|S|}$. Everything below follows from that small qualification, *independently of $h$*:

**Which estimates are honest**

- **Training error:** $\hat R_{\text{train}}(\hat h)$ where $\hat h$ was **fitted** to minimize it. It is biased low, often badly.
- **Validation error of one model:** unbiased, as long as that single model was fixed before looking at the validation set.
- **Validation error of the winner:** biased low: the winner was **selected** because its validation error was small.
- **Test error of the final model:** unbiased, because the test set played no part in fitting or selection.

## Why the best validation score is optimistic

Each validation score is the true risk plus noise: $\hat R_{\text{val}}(h_k)=R(h_k)+\varepsilon_k$. Taking the minimum over $K$ candidates favours whichever model drew the most negative noise. The minimum is a concave function, so Jensen's inequality says that on average it lands below the minimum of the true risks:

$$
\mathbb{E}\Big[\min_{k}\hat R_{\text{val}}(h_k)\Big]\;\le\;\min_k \mathbb{E}\big[\hat R_{\text{val}}(h_k)\big]=\min_k R(h_k)
$$

- $h_1,\dots,h_K$: the candidate models (one per hyperparameter setting)
- $\hat R_{\text{val}}(h_k)$: validation error of candidate k
- $R(h_k)$: its true risk

How big is the gap? Take the worst case, where all $K$ candidates are equally good, $R(h_k)=R$, and their validation noise is independent with standard deviation $s$. Then the expected minimum is $R$ minus $s$ times the expected maximum of $K$ standard normals:

$$
\mathbb{E}\Big[\min_k \hat R_{\text{val}}(h_k)\Big]\approx R - c_K\,s,\qquad s=\frac{\sigma_\ell}{\sqrt{n_{\text{val}}}},\qquad c_2=0.56,\;c_{10}=1.54,\;c_{100}=2.51
$$

- $c_K$: expected maximum of $K$ independent standard normal variables
- $\sigma_\ell$: standard deviation of the per-example loss
- $n_{\text{val}}$: number of validation examples

The bias grows with the number of things you try (slowly, roughly like $\sqrt{2\ln K}$) and shrinks with the size of the validation set. Real candidates are correlated and not all equally good, so the actual gap is smaller than this worst case. The direction is always the same, though: **the validation score of the winner overstates how good the winner is**. A test set that took no part in the choice removes the bias.

## Worked example: choosing a polynomial degree

Sixty points are collected over time, $x\in[0,1]$, from $y=\sin(2\pi x)+\varepsilon$ with noise $\sigma=0.3$. No model can get below the noise floor, so every true risk is at least $\sigma^2=0.09$. We shuffle and split 60/20/20, which gives **36 train, 12 validation and 12 test** points. Then we fit polynomials of degree 1–10 by least squares on the 36 training points. Because the data is synthetic we can also compute each fit's **true risk** exactly, by integrating its squared error against $\sin(2\pi x)$ and adding $\sigma^2$. Real projects never get this column. It is here so we can check the estimates.

Mean squared error of each degree, all fitted on the same 36 training points. Highlighted: the validation minimum, degree 3. Training error keeps falling as the degree rises; validation error does not.

| Degree | Train MSE | Validation MSE | True risk (unknowable in practice) |
| --- | --- | --- | --- |
| 1 | 0.397 | 0.353 | 0.303 |
| 2 | 0.396 | 0.355 | 0.304 |
| **3** | **0.133** | **0.080** | **0.101** |
| 4 | 0.127 | 0.105 | 0.105 |
| 5 | 0.100 | 0.155 | 0.109 |
| 6 | 0.090 | 0.131 | 0.114 |
| 7 | 0.090 | 0.131 | 0.114 |
| 8 | 0.088 | 0.130 | 0.114 |
| 9 | 0.087 | 0.122 | 0.113 |
| 10 | 0.087 | 0.122 | 0.113 |

**Reading the table**

1. **Training error is useless for choosing.** It falls monotonically from 0.397 to 0.087, so it would always pick the largest degree. A sine needs an odd polynomial of degree at least 3, which is why degrees 1 and 2 are stuck near 0.4.
2. **Validation picks degree 3**, with $\hat R_{\text{val}}=0.080$. That also happens to be the degree with the lowest true risk (0.101), so the choice is correct.
3. **But the winning validation score is optimistic.** 0.080 is below the true risk of 0.101, and even below the noise floor of 0.09 that no model can beat. Degree 3 won partly because its 12 validation points happened to be easy.
4. **Test once.** The degree-3 model on the 12 untouched test points gives squared errors with mean **0.077** and sample standard deviation 0.083, so $\text{SE}=0.083/\sqrt{12}=0.024$. We report **test MSE = 0.077 ± 0.024**. The true value of 0.101 lies about one standard error away. With only 12 test points the estimate is honest (unbiased) but noisy, and here it was lucky too.

> Figure: Sixty samples as bars (height = y), coloured by split. Drag the two handles or use the sliders to resize the splits. The middle-right panel fits degrees 1–10 on the train bars; the ring marks the validation minimum and the green diamond is the single test evaluation with ±1 SE. The bottom strip repeats the whole procedure on 40 reshuffles of the same sizes. With 12 validation points the winner's validation MSE averages 0.113 against a true risk of 0.116. Click “tiny val (3)” and the winner's validation MSE drops to an average of 0.085, below the 0.09 noise floor, while its true risk rises to 0.188. Switch to time-ordered and the polynomials have to extrapolate into the future, so the validation and test errors explode.

The bottom strip of the figure shows selection bias directly. Across 40 reshuffles at 36/12/12, the average gap between the winner's validation score and its true risk is only 0.003. Shrink validation to **3 points** and the gap grows to **0.104**. The winning validation score becomes optimistic in 31 of the 40 runs, and every one of the 10 degrees wins at least once. A tiny validation set lets you pick a model, but it does not reliably pick a good one, and the score it reports for that model can't be trusted.

## How big must the test set be?

For a classifier the per-example loss is 0 or 1, so the test error $\hat p$ is a binomial proportion and its standard error has a closed form:

$$
\text{SE}(\hat p)=\sqrt{\frac{p(1-p)}{n}}\qquad\Longrightarrow\qquad n\;\ge\;\frac{z^2\,p(1-p)}{h^2}
$$

- $p$: true error rate (plug in the estimate $\hat p$)
- $n$: number of test examples
- $h$: desired half-width of the confidence interval $\hat p\pm h$
- $z$: normal quantile: 1.96 for 95% confidence

Test-set size needed for a 95% interval of half-width h.

| True error p | Half-width h | Test examples needed |
| --- | --- | --- |
| 10% | ±2 points | 865 |
| 10% | ±1 point | 3,458 |
| 5% | ±1 point | 1,825 |

So 1,000 test examples at an 8% error rate give $\text{SE}=\sqrt{0.08\cdot0.92/1000}=0.0086$, which means "8% ± 1.7%" at 95%. To halve the interval you need **four times** as many examples. For losses that are not 0/1, such as the MSE above, use the same idea with the sample standard deviation of the per-example losses: $\text{SE}=s_\ell/\sqrt{n}$.

> **Comparing two models on one test set:** If two models score 8.0% and 8.6% on 1,000 test examples, the two intervals overlap heavily, but that is the wrong test. Both models were evaluated on the **same** examples, so compare them with a **paired** analysis: look only at the examples where they disagree (McNemar's test), or bootstrap the per-example difference in loss. Pairing removes the shared noise and is much more sensitive.

## Splitting correctly

A random split is only right if the rows are independent and identically distributed and the model will see the same distribution in deployment. When either assumption fails, the split has to copy the conditions the model will face in deployment:

**Choose the split to match deployment**

- **Stratified:** For classification, split **within each class** so that every set keeps the overall class proportions. With 5% positives and a 100-row validation set, a plain shuffle gives anywhere from about 1 to 9 positives (mean 5, SD 2.2), and some metrics are then meaningless. Stratifying fixes the count. For regression you can stratify on quantile bins of $y$, which is what the figure's *stratified* mode does with terciles.
- **Time-ordered:** If the model will predict the future, train on the past and validate and test on later periods, **without shuffling**. A shuffled split lets the model learn from the future, for example by interpolating between neighbouring days. It reports an error you will never achieve. Leave a **gap** between the periods when features use look-back windows. For several folds, see `TimeSeriesSplit`.
- **Grouped:** If rows come in groups (several scans per patient, many sessions per user, frames of one video), every group must fall entirely within one split. Otherwise the model can recognise the *patient* instead of learning the *disease*. Use `GroupShuffleSplit` (or `GroupKFold`) with the entity id.
- **Deduplicated:** Exact and near duplicates (re-uploaded images, copied text, the same customer twice) that land in different splits turn test examples into training examples. Deduplicate **before** splitting, or group the duplicates together.

In the figure, switching to *time-ordered* puts the last 12 time points in the test set. Every polynomial now has to extrapolate. Validation chooses degree 1, the only degree that doesn't diverge, and its test MSE is 0.432, more than five times the shuffled estimate. That number is the honest one if the task is forecasting.

## Data leakage

**Data leakage:** is any path by which information from the validation or test set (or from the future, or from the target itself) reaches the model during training or selection. A leaked evaluation looks better than the model will ever perform in deployment.

The most common form is innocent preprocessing. You compute a scaler's mean and standard deviation, a PCA basis, an imputation value, a vocabulary or a target encoding on **all** the data and then split. The training features now contain statistics of the test rows. The rule is that **every fitted step is fitted on the training set only** and then *applied* unchanged to validation and test:

$$
\mu=\frac{1}{n_{\text{tr}}}\sum_{i\in\text{train}}x_i,\qquad \sigma^2=\frac{1}{n_{\text{tr}}}\sum_{i\in\text{train}}(x_i-\mu)^2,\qquad z=\frac{x-\mu}{\sigma}\ \text{for train, val and test alike}
$$

- $\mu,\ \sigma$: standardization statistics, computed from training rows only
- $z$: the transformed feature fed to the model

Wrap the preprocessing and the model in one scikit-learn `Pipeline`. Calling `fit` on the training data then fits every step on that data alone, and the same pipeline object can be passed to cross-validation without leaking across folds.

## In code

```python
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures, StandardScaler
from sklearn.linear_model import LinearRegression

rng = np.random.default_rng(7)
x = np.sort(rng.uniform(0, 1, 60))
y = np.sin(2 * np.pi * x) + rng.normal(0, 0.3, 60)
X = x[:, None]

# 60 / 20 / 20: carve off the test set FIRST, then split the remainder 75 / 25.
X_rest, X_test, y_rest, y_test = train_test_split(X, y, test_size=0.20, random_state=3)
X_tr, X_val, y_tr, y_val = train_test_split(X_rest, y_rest, test_size=0.25, random_state=3)
print(len(X_tr), len(X_val), len(X_test))           # 36 12 12

def model(d):
  # The scaler lives inside the pipeline, so its mean/sd are learned from X_tr only.
  return make_pipeline(PolynomialFeatures(d, include_bias=False), StandardScaler(), LinearRegression())

val_mse = {}
for d in range(1, 11):
  m = model(d).fit(X_tr, y_tr)
  val_mse[d] = np.mean((m.predict(X_val) - y_val) ** 2)
best = min(val_mse, key=val_mse.get)

# Touch the test set exactly once, with the chosen degree.
final = model(best).fit(X_tr, y_tr)
sq = (final.predict(X_test) - y_test) ** 2
print(f"degree {best}: test MSE {sq.mean():.3f} ± {sq.std(ddof=1) / np.sqrt(len(sq)):.3f}")
```

The worked example's procedure with scikit-learn. This uses NumPy's random generator instead of the site's, so the data and numbers differ slightly: this run picks degree 4 with test MSE 0.061 ± 0.033.

```python
from sklearn.model_selection import train_test_split, GroupShuffleSplit, TimeSeriesSplit
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

# Stratified 60/20/20: every split keeps the class proportions of y.
X_rest, X_test, y_rest, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=0)
X_tr, X_val, y_tr, y_val = train_test_split(X_rest, y_rest, test_size=0.25, stratify=y_rest, random_state=0)

# Grouped: no patient appears on both sides.
tr_idx, te_idx = next(GroupShuffleSplit(n_splits=1, test_size=0.2, random_state=0)
                    .split(X, y, groups=patient_id))

# Time-ordered: rows sorted by time, no shuffling; the future is held out.
cut1, cut2 = int(0.6 * len(X)), int(0.8 * len(X))
X_past, X_recent, X_future = X[:cut1], X[cut1:cut2], X[cut2:]   # train, val, test
# For several expanding-window folds: TimeSeriesSplit(n_splits=5, gap=lookback)

# LEAKY: the scaler saw the validation and test rows.
X_all = StandardScaler().fit_transform(X)            # then split X_all ...  (don't)

# CORRECT: preprocessing is part of the model and is fitted on train only.
clf = make_pipeline(StandardScaler(), LogisticRegression())
clf.fit(X_tr, y_tr)
print(clf.score(X_val, y_val))
```

Stratified, grouped and time-ordered splits, and the leakage bug next to its fix.

> **When the data is small: cross-validation:** With 60 points, one 12-point validation set is very noisy, as the table shows. **k-fold cross-validation** rotates the validation role across $k$ folds and averages the scores, so every training point is also used for validation once. It is covered in its own article, **Cross-validation**. The test set still stays outside the loop: cross-validate on train + validation, then test once.

**Strengths**

- You have plenty of data: one fixed split is simple, fast, and each model is trained only once per candidate.
- Training is expensive (deep networks, large ensembles), so k-fold retraining is not affordable.
- You need an auditable final number: a held-out test set that nobody touched is the clearest evidence you can give.
- Deployment has a clear structure (a future period, new customers or new sites) that a single split can copy exactly.

**Limitations**

- Data is scarce: a 20% validation set gives a noisy choice and a 20% test set a wide interval. Use cross-validation, or nested cross-validation for the final estimate.
- One random split is one sample. The selected model and its score can change with the seed, as the reshuffle strip shows.
- Holding out 40% of the data means the deployed model was trained on less than it could have been. Refit on train + validation after choosing.
- Any reuse of the test set, even only to check, slowly turns it into a validation set.

- **Model and hyperparameter selection:** Degree, depth, regularization strength, learning rate, architecture: choose on validation, never on test.
- **Early stopping:** Watch validation loss during training and keep the epoch where it was lowest. That epoch is a hyperparameter chosen on validation, so the final number still comes from test.
- **Benchmarks and competitions:** Public leaderboards act as validation sets. Hidden private test sets exist because repeatedly submitting to a public board overfits it.
- **Forecasting and risk models:** Backtesting on later periods is the time-ordered version of a test set, and is often a regulatory requirement.

> **Leakage through preprocessing:** Scaling, imputing, PCA, feature selection, target encoding or oversampling (SMOTE) done **before** the split puts information from held-out rows into training. Do every fitted step inside a `Pipeline` that is fitted on train only. Oversample only the training set.

> **Peeking at the test set:** Checking test accuracy *just to see*, then tweaking a feature and checking again, makes the test set a validation set. With enough peeks you will find a configuration that looks great on test by chance. Decide everything on validation; open the test set once, and if you must iterate afterwards, collect a fresh test set.

> **Duplicates and entities across splits:** The same image under two file names, the same patient in train and test, or the same sentence scraped twice makes test examples into memorized training examples. Deduplicate before splitting, and split by group id whenever rows share an entity.

**Question:** You try 50 hyperparameter settings and report the best validation accuracy, 91.3%. What does that number most likely do?

- A. It is an unbiased estimate of the deployed accuracy
- B. It overstates the deployed accuracy, because the winner was selected for doing well on these exact validation rows
- C. It understates the deployed accuracy, because the model was not trained on the validation rows
- D. Nothing can be said without the training accuracy

**Answer:** B. It overstates the deployed accuracy, because the winner was selected for doing well on these exact validation rows — Each validation score is the true score plus noise. The maximum of 50 noisy scores favours a candidate whose noise was positive, so $\mathbb E[\max]$ is above the true accuracy of the winner. A test set that took no part in the choice removes this selection bias.

**Question:** A classifier has a 10% error rate. About how many test examples do you need for a 95% interval of ±1 percentage point?

- A. About 100
- B. About 900
- C. About 3,500
- D. About 35,000

**Answer:** C. About 3,500 — $n\ge 1.96^2\cdot0.1\cdot0.9/0.01^2\approx 3{,}458$. For ±2 points about 865 would do. Halving the interval width costs four times the data.

**Question:** You standardize all 10,000 rows with StandardScaler, then split 80/20 and train. What is wrong?

- A. Nothing, because standardization does not use the labels
- B. The scaler's mean and standard deviation were computed using the test rows, so test information leaked into training
- C. StandardScaler must be applied after the model is trained
- D. Only the test set should be standardized

**Answer:** B. The scaler's mean and standard deviation were computed using the test rows, so test information leaked into training — Any statistic fitted on data includes information about that data, even without labels. Fit the scaler on the training rows only, which a Pipeline does automatically, and apply it unchanged to validation and test. With a plain scaler the leak is usually small, but with imputation, feature selection or target encoding it can be large.

**Question:** You predict next month's sales from daily data for 2019–2024. Which split gives an honest estimate?

- A. A shuffled 60/20/20 split, stratified by month
- B. Train on 2019–2022, validate on 2023, test on 2024, without shuffling
- C. A random 80/20 split repeated with five seeds
- D. Test on 2019, the oldest year, so the model has the most recent data for training

**Answer:** B. Train on 2019–2022, validate on 2023, test on 2024, without shuffling — Deployment means predicting the future from the past, so the split should do the same. A shuffled split lets the model interpolate between neighbouring days in the test set. That is a much easier task than forecasting, and the reported error will be optimistic, as the figure's time-ordered mode shows.

**Key points**

- Train fits parameters, validation chooses hyperparameters and models, and test gives one final estimate. Each is honest only for decisions it took no part in.
- Empirical risk on a sample independent of the model is unbiased for the true risk, with standard error $s_\ell/\sqrt n$. For error rates this is $\sqrt{p(1-p)/n}$.
- The minimum of many noisy validation scores is optimistically biased. The bias grows with the number of candidates and shrinks with the size of the validation set.
- Test-set size: $n\ge z^2p(1-p)/h^2$, about 3,500 examples for ±1 point at a 10% error rate.
- Match the split to deployment: stratify for rare classes, keep time order for forecasting, and split by group when rows share an entity. Remove duplicates before splitting.
- Fit every preprocessing step on training data only, ideally inside a Pipeline, and never tune against the test set.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
