Training, validation & test sets

Fit on the training set, make choices on the validation set, and touch the test set once to report how well the final model generalizes.

7 min read703 wordscore0 viewsupdated 2026-10-09

Definition

Training, validation and test sets are three disjoint parts of the labelled data with three different jobs. Gradient descent fits the weights on the training set. Every decision a human or a script makes (which epoch to keep, which learning rate, which architecture) is made on the validation set. The test set is evaluated once, on the final model, to estimate how it will do on new data.

Why three, not two

Training loss says almost nothing about new data: a large network can push it towards zero by memorising. So we hold data out. But we also choose things using the held-out data, and choosing is a form of fitting. Pick the best of 50 checkpoints on a set, and that set's score for the winner is optimistic.

The fix is a second held-out set that no decision ever touched. In deep learning this matters more than usual, because every epoch produces a new candidate model. A 60-epoch run is already 60 models to choose between before you change any hyperparameter.

What a held-out loss can tell you

The loss on a split is an average over its examples. If the split was never used to pick the model, that average is an unbiased estimate of the loss on fresh data, and its noise shrinks like 1/n1/\sqrt n.

L^S=1∣S∣∑i∈Sℓ(fθ(xi),yi),SE(L^S)=s∣S∣,SE(acc^)=p(1−p)∣S∣\hat L_S=\frac{1}{|S|}\sum_{i\in S}\ell\big(f_\theta(x_i),y_i\big),\qquad \mathrm{SE}(\hat L_S)=\frac{s}{\sqrt{|S|}},\qquad \mathrm{SE}(\widehat{\mathrm{acc}})=\sqrt{\frac{p(1-p)}{|S|}}
SS
one split (train, val or test) and its examples
ℓ\ell
per-example loss, e.g. cross-entropy
fθf_\theta
the network with weights θ\theta (one checkpoint)
ss
standard deviation of the per-example loss
pp
true accuracy, for the special case of 0–1 loss
10k labelled examples, shuffled once, then cut · drag the handlestrain 70%7kval 15%1.5ktest 15%1.5kloss per epoch: train · val (measured) · fresh-data loss (unknowable, dashed)00.40.81.21102030405060epoch →early stop (patience 5)best val
||
epoch 31: train 0.142 · val 0.495 ± 0.026
best val at epoch 31 · early stop halts at 36, restores 31
test set: sealed. Choose your checkpoint on val, then evaluate once.
fig.Synthetic but deterministic learning curves. The ringed point is the best validation epoch; the shaded region is where early stopping (patience 5) has already halted. Drag the checkpoint slider, then press "evaluate on test (once)". Change anything afterwards and the readout warns that the test set has become a validation set. Try 98/1/1 at 10k: the 100-example val curve gets noisy, and its minimum sits well below the dashed fresh-data loss. Then try 98/1/1 at 1M.
datasetsplitval / test sizeSE≈ 95% interval
10k70/15/151,5000.77 pp± 1.5 pp
10k98/1/11003.0 pp± 5.9 pp
1M98/1/110,0000.30 pp± 0.59 pp
10M98/1/1100,0000.095 pp± 0.19 pp
How precise is a split? Standard error of a 90% accuracy estimate, from the formula above

The table explains the "big data" split. What matters is the absolute size of the val and test sets, not their percentage. At a million examples, 1% is 10,000 examples, enough to resolve differences of about half a point, so the other 98% can go to training. At ten thousand examples, 1% is 100 examples and the estimate is noise.

Validation in the training loop

One run, one decision per epoch

  1. 1Split once, with a fixed seed, before looking at any results. Put the test indices away.
  2. 2Every epoch: train on the training set, then compute the loss (and metric) on the full validation set with the model in eval mode.
  3. 3Checkpoint selection: whenever validation loss improves, save the weights. The saved checkpoint is the run's model, not the last epoch.
  4. 4Early stopping: if validation has not improved for PP epochs (the patience), stop and restore the best checkpoint. This is "choosing the epoch" automated, and it is a decision made on validation.
  5. 5Across runs, compare learning rates, widths and augmentations by their best validation score. Iterate as often as you like.
  6. 6Test once: evaluate the final checkpoint on the test set and report that number. Do not tune anything because of it.

Why k-fold is rare in deep learning

k-fold cross-validation trains kk models, so it multiplies the cost by kk. Five folds of a run that takes 3 GPU-days cost 15 GPU-days for every hyperparameter setting. With large datasets you also do not need it, since a single 10,000-example validation set already has a standard error of about 0.3 points. Cross-validation remains common when data is scarce and models are small, for example a few thousand medical images fine-tuned from a pretrained backbone.

Stratification and leakage

Stratify classification splits so each one keeps the overall class mix. With 2% positives, a random 500-example val set has about 10 positives, and the count swings a lot between splits. A stratified split holds it at 10.

trapwhat leaksfix
Augmenting before splittingCrops or flips of one image land in both train and val, so val tests near-copies of training imagesSplit raw examples first; augment only inside the training data loader
Near-duplicatesRe-uploads, resized copies and video frames of the same scene end up on both sidesDeduplicate (e.g. perceptual hashes, embedding similarity) before splitting, or group duplicates
Time series shuffled randomlyThe model sees the future and is validated on the pastSplit by time: train on earlier data, validate later, test latest
Same patient / user / speaker in several splitsThe model learns the person, not the taskGroup split: all records of one entity go to one split
Normalisation statistics from all dataMean and std (or a vocabulary) computed using val and testCompute them on the training set only and reuse them
Common leakage traps: each makes validation and test look better than deployment
python
1import torch
2from torch.utils.data import DataLoader, random_split
3
4g = torch.Generator().manual_seed(0) # same split every run
5train_ds, val_ds, test_ds = random_split(full_ds, [0.8, 0.1, 0.1], generator=g)
6# random_split neither stratifies nor groups. For those, split indices with
7# sklearn's train_test_split(stratify=y) or GroupShuffleSplit and wrap them in Subset.
8
9@torch.no_grad() # no graph, less memory
10def evaluate(model, loader):
11 model.eval() # dropout off, BatchNorm uses running stats
12 total, n = 0.0, 0
13 for x, y in loader:
14 total += loss_fn(model(x), y).item() * len(y)
15 n += len(y)
16 return total / n
17
18best, bad, patience = float("inf"), 0, 5
19for epoch in range(100):
20 model.train()
21 for x, y in train_loader:
22 opt.zero_grad(); loss_fn(model(x), y).backward(); opt.step()
23 val_loss = evaluate(model, val_loader)
24 if val_loss < best:
25 best, bad = val_loss, 0
26 torch.save(model.state_dict(), "best.pt") # checkpoint selection
27 else:
28 bad += 1
29 if bad >= patience: # early stopping
30 break
31
32model.load_state_dict(torch.load("best.pt"))
33test_loss = evaluate(model, DataLoader(test_ds, batch_size=256)) # once, at the very end
A seeded split, plus a validation pass that selects the checkpoint

Check yourself

You train for 80 epochs, keep the checkpoint with the lowest validation loss (epoch 34), then try three learning rates and keep the one with the best validation loss. Which number is an unbiased estimate of performance on new data?

Check yourself

You have 2 million labelled images. Which split is most sensible?

Remember

  • ✓Train fits weights, validation makes every decision (including which epoch to keep), and test is evaluated once.
  • ✓Each epoch is a candidate model: monitor validation loss every epoch, save the best checkpoint, stop early on it.
  • ✓Size held-out sets in absolute terms: SE ∝1/n\propto 1/\sqrt n. 98/1/1 is fine at millions of examples and useless at thousands.
  • ✓k-fold costs k training runs, which is why deep learning usually uses one fixed validation set.
  • ✓Split before augmenting, deduplicate, split time series by time, group by patient or user, and fit normalisation on train only.
  • ✓Evaluate with model.eval() and torch.no_grad().
← back to Deep Learning roadmap