# Training, validation & test sets

> Fit on the training set, make choices on the validation set, and touch the test set once to report how well the final model generalizes.

Source: https://www.mlfoundry.in/learn/training-validation-test-sets  
Section: Deep Learning › Deep Learning Fundamentals  
Level: core · 7 min read · updated 2026-10-09

**Training, validation and test sets:** are three disjoint parts of the labelled data with three different jobs. Gradient descent fits the weights on the **training set**. Every decision a human or a script makes (which epoch to keep, which learning rate, which architecture) is made on the **validation set**. The **test set** is evaluated **once**, on the final model, to estimate how it will do on new data.

## Why three, not two

Training loss says almost nothing about new data: a large network can push it towards zero by memorising. So we hold data out. But we also *choose* things using the held-out data, and choosing is a form of fitting. Pick the best of 50 checkpoints on a set, and that set's score for the winner is optimistic.

The fix is a second held-out set that no decision ever touched. In deep learning this matters more than usual, because every epoch produces a new candidate model. A 60-epoch run is already 60 models to choose between before you change any hyperparameter.

## What a held-out loss can tell you

The loss on a split is an average over its examples. If the split was never used to pick the model, that average is an unbiased estimate of the loss on fresh data, and its noise shrinks like $1/\sqrt n$.

$$
\hat L_S=\frac{1}{|S|}\sum_{i\in S}\ell\big(f_\theta(x_i),y_i\big),\qquad \mathrm{SE}(\hat L_S)=\frac{s}{\sqrt{|S|}},\qquad \mathrm{SE}(\widehat{\mathrm{acc}})=\sqrt{\frac{p(1-p)}{|S|}}
$$

- $S$: one split (train, val or test) and its examples
- $\ell$: per-example loss, e.g. cross-entropy
- $f_\theta$: the network with weights $\theta$ (one checkpoint)
- $s$: standard deviation of the per-example loss
- $p$: true accuracy, for the special case of 0–1 loss

> Figure: Synthetic but deterministic learning curves. The ringed point is the best validation epoch; the shaded region is where early stopping (patience 5) has already halted. Drag the checkpoint slider, then press "evaluate on test (once)". Change anything afterwards and the readout warns that the test set has become a validation set. Try 98/1/1 at 10k: the 100-example val curve gets noisy, and its minimum sits well below the dashed fresh-data loss. Then try 98/1/1 at 1M.

How precise is a split? Standard error of a 90% accuracy estimate, from the formula above

| dataset | split | val / test size | SE | ≈ 95% interval |
| --- | --- | --- | --- | --- |
| 10k | 70/15/15 | 1,500 | 0.77 pp | ± 1.5 pp |
| 10k | 98/1/1 | 100 | 3.0 pp | ± 5.9 pp |
| 1M | 98/1/1 | 10,000 | 0.30 pp | ± 0.59 pp |
| 10M | 98/1/1 | 100,000 | 0.095 pp | ± 0.19 pp |

The table explains the "big data" split. What matters is the **absolute** size of the val and test sets, not their percentage. At a million examples, 1% is 10,000 examples, enough to resolve differences of about half a point, so the other 98% can go to training. At ten thousand examples, 1% is 100 examples and the estimate is noise.

## Validation in the training loop

**One run, one decision per epoch**

1. **Split once**, with a fixed seed, before looking at any results. Put the test indices away.
2. **Every epoch:** train on the training set, then compute the loss (and metric) on the full validation set with the model in eval mode.
3. **Checkpoint selection:** whenever validation loss improves, save the weights. The saved checkpoint is the run's model, not the last epoch.
4. **Early stopping:** if validation has not improved for $P$ epochs (the patience), stop and restore the best checkpoint. This is "choosing the epoch" automated, and it is a decision made on validation.
5. **Across runs**, compare learning rates, widths and augmentations by their best validation score. Iterate as often as you like.
6. **Test once:** evaluate the final checkpoint on the test set and report that number. Do not tune anything because of it.

> **The test set is defined by what you never did with it:** Nothing about the data makes it "test" data. Look at the test score, find it disappointing and change the learning rate, and the test set has become a second validation set with the same optimism. Benchmarks reused by a whole field for years drift the same way.

## Why k-fold is rare in deep learning

k-fold cross-validation trains $k$ models, so it multiplies the cost by $k$. Five folds of a run that takes 3 GPU-days cost 15 GPU-days for every hyperparameter setting. With large datasets you also do not need it, since a single 10,000-example validation set already has a standard error of about 0.3 points. Cross-validation remains common when data is scarce and models are small, for example a few thousand medical images fine-tuned from a pretrained backbone.

## Stratification and leakage

**Stratify** classification splits so each one keeps the overall class mix. With 2% positives, a random 500-example val set has about 10 positives, and the count swings a lot between splits. A stratified split holds it at 10.

Common leakage traps: each makes validation and test look better than deployment

| trap | what leaks | fix |
| --- | --- | --- |
| Augmenting before splitting | Crops or flips of one image land in both train and val, so val tests near-copies of training images | Split raw examples first; augment only inside the training data loader |
| Near-duplicates | Re-uploads, resized copies and video frames of the same scene end up on both sides | Deduplicate (e.g. perceptual hashes, embedding similarity) before splitting, or group duplicates |
| Time series shuffled randomly | The model sees the future and is validated on the past | Split by time: train on earlier data, validate later, test latest |
| Same patient / user / speaker in several splits | The model learns the person, not the task | Group split: all records of one entity go to one split |
| Normalisation statistics from all data | Mean and std (or a vocabulary) computed using val and test | Compute them on the training set only and reuse them |

```python
import torch
from torch.utils.data import DataLoader, random_split

g = torch.Generator().manual_seed(0)                 # same split every run
train_ds, val_ds, test_ds = random_split(full_ds, [0.8, 0.1, 0.1], generator=g)
# random_split neither stratifies nor groups. For those, split indices with
# sklearn's train_test_split(stratify=y) or GroupShuffleSplit and wrap them in Subset.

@torch.no_grad()                                     # no graph, less memory
def evaluate(model, loader):
    model.eval()                                     # dropout off, BatchNorm uses running stats
    total, n = 0.0, 0
    for x, y in loader:
        total += loss_fn(model(x), y).item() * len(y)
        n += len(y)
    return total / n

best, bad, patience = float("inf"), 0, 5
for epoch in range(100):
    model.train()
    for x, y in train_loader:
        opt.zero_grad(); loss_fn(model(x), y).backward(); opt.step()
    val_loss = evaluate(model, val_loader)
    if val_loss < best:
        best, bad = val_loss, 0
        torch.save(model.state_dict(), "best.pt")    # checkpoint selection
    else:
        bad += 1
        if bad >= patience:                          # early stopping
            break

model.load_state_dict(torch.load("best.pt"))
test_loss = evaluate(model, DataLoader(test_ds, batch_size=256))  # once, at the very end
```

A seeded split, plus a validation pass that selects the checkpoint

> **Validating in train mode:** Without `model.eval()`, dropout stays on, so the validation loss is noisy and too high. BatchNorm also normalises with the val batch's own statistics and updates its running averages with validation data, a small leak. `torch.no_grad()` does not switch either of these off. It only stops gradient tracking. You need both.

**Question:** You train for 80 epochs, keep the checkpoint with the lowest validation loss (epoch 34), then try three learning rates and keep the one with the best validation loss. Which number is an unbiased estimate of performance on new data?

- A. The best validation loss
- B. The training loss at epoch 34
- C. The loss of the final checkpoint on the test set, computed once
- D. The average validation loss over all 80 epochs

**Answer:** C. The loss of the final checkpoint on the test set, computed once — Both the epoch and the learning rate were chosen *by* validation loss, so the winning validation score is optimistically biased. Training loss is biased even more. Only a set that influenced no decision, the test set evaluated once, gives an unbiased estimate.

**Question:** You have 2 million labelled images. Which split is most sensible?

- A. 60/20/20, the classic default
- B. 98/1/1, since 1% is still 20,000 images
- C. 10-fold cross-validation for every hyperparameter setting
- D. No test set, since validation is large enough

**Answer:** B. 98/1/1, since 1% is still 20,000 images — Precision depends on absolute size. 20,000 examples give a standard error of about 0.2 points for 90% accuracy ($\sqrt{0.09/20000}\approx0.0021$). Moving 38% more of the data into training is worth more than shrinking that further. 10-fold multiplies compute by 10. Dropping the test set leaves you with only optimistic numbers.

**Key points**

- Train fits weights, validation makes every decision (including which epoch to keep), and test is evaluated once.
- Each epoch is a candidate model: monitor validation loss every epoch, save the best checkpoint, stop early on it.
- Size held-out sets in absolute terms: SE $\propto 1/\sqrt n$. 98/1/1 is fine at millions of examples and useless at thousands.
- k-fold costs k training runs, which is why deep learning usually uses one fixed validation set.
- Split before augmenting, deduplicate, split time series by time, group by patient or user, and fit normalisation on train only.
- Evaluate with `model.eval()` *and* `torch.no_grad()`.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
