Definition
Training, validation and test sets are three disjoint parts of the labelled data with three different jobs. Gradient descent fits the weights on the training set. Every decision a human or a script makes (which epoch to keep, which learning rate, which architecture) is made on the validation set. The test set is evaluated once, on the final model, to estimate how it will do on new data.
Why three, not two
Training loss says almost nothing about new data: a large network can push it towards zero by memorising. So we hold data out. But we also choose things using the held-out data, and choosing is a form of fitting. Pick the best of 50 checkpoints on a set, and that set's score for the winner is optimistic.
The fix is a second held-out set that no decision ever touched. In deep learning this matters more than usual, because every epoch produces a new candidate model. A 60-epoch run is already 60 models to choose between before you change any hyperparameter.
What a held-out loss can tell you
The loss on a split is an average over its examples. If the split was never used to pick the model, that average is an unbiased estimate of the loss on fresh data, and its noise shrinks like .
- one split (train, val or test) and its examples
- per-example loss, e.g. cross-entropy
- the network with weights (one checkpoint)
- standard deviation of the per-example loss
- true accuracy, for the special case of 0–1 loss
| dataset | split | val / test size | SE | ≈ 95% interval |
|---|---|---|---|---|
| 10k | 70/15/15 | 1,500 | 0.77 pp | ± 1.5 pp |
| 10k | 98/1/1 | 100 | 3.0 pp | ± 5.9 pp |
| 1M | 98/1/1 | 10,000 | 0.30 pp | ± 0.59 pp |
| 10M | 98/1/1 | 100,000 | 0.095 pp | ± 0.19 pp |
The table explains the "big data" split. What matters is the absolute size of the val and test sets, not their percentage. At a million examples, 1% is 10,000 examples, enough to resolve differences of about half a point, so the other 98% can go to training. At ten thousand examples, 1% is 100 examples and the estimate is noise.
Validation in the training loop
One run, one decision per epoch
- 1Split once, with a fixed seed, before looking at any results. Put the test indices away.
- 2Every epoch: train on the training set, then compute the loss (and metric) on the full validation set with the model in eval mode.
- 3Checkpoint selection: whenever validation loss improves, save the weights. The saved checkpoint is the run's model, not the last epoch.
- 4Early stopping: if validation has not improved for epochs (the patience), stop and restore the best checkpoint. This is "choosing the epoch" automated, and it is a decision made on validation.
- 5Across runs, compare learning rates, widths and augmentations by their best validation score. Iterate as often as you like.
- 6Test once: evaluate the final checkpoint on the test set and report that number. Do not tune anything because of it.
Why k-fold is rare in deep learning
k-fold cross-validation trains models, so it multiplies the cost by . Five folds of a run that takes 3 GPU-days cost 15 GPU-days for every hyperparameter setting. With large datasets you also do not need it, since a single 10,000-example validation set already has a standard error of about 0.3 points. Cross-validation remains common when data is scarce and models are small, for example a few thousand medical images fine-tuned from a pretrained backbone.
Stratification and leakage
Stratify classification splits so each one keeps the overall class mix. With 2% positives, a random 500-example val set has about 10 positives, and the count swings a lot between splits. A stratified split holds it at 10.
| trap | what leaks | fix |
|---|---|---|
| Augmenting before splitting | Crops or flips of one image land in both train and val, so val tests near-copies of training images | Split raw examples first; augment only inside the training data loader |
| Near-duplicates | Re-uploads, resized copies and video frames of the same scene end up on both sides | Deduplicate (e.g. perceptual hashes, embedding similarity) before splitting, or group duplicates |
| Time series shuffled randomly | The model sees the future and is validated on the past | Split by time: train on earlier data, validate later, test latest |
| Same patient / user / speaker in several splits | The model learns the person, not the task | Group split: all records of one entity go to one split |
| Normalisation statistics from all data | Mean and std (or a vocabulary) computed using val and test | Compute them on the training set only and reuse them |
1import torch2from torch.utils.data import DataLoader, random_split34g = torch.Generator().manual_seed(0) # same split every run5train_ds, val_ds, test_ds = random_split(full_ds, [0.8, 0.1, 0.1], generator=g)6# random_split neither stratifies nor groups. For those, split indices with7# sklearn's train_test_split(stratify=y) or GroupShuffleSplit and wrap them in Subset.89@torch.no_grad() # no graph, less memory10def evaluate(model, loader):11 model.eval() # dropout off, BatchNorm uses running stats12 total, n = 0.0, 013 for x, y in loader:14 total += loss_fn(model(x), y).item() * len(y)15 n += len(y)16 return total / n1718best, bad, patience = float("inf"), 0, 519for epoch in range(100):20 model.train()21 for x, y in train_loader:22 opt.zero_grad(); loss_fn(model(x), y).backward(); opt.step()23 val_loss = evaluate(model, val_loader)24 if val_loss < best:25 best, bad = val_loss, 026 torch.save(model.state_dict(), "best.pt") # checkpoint selection27 else:28 bad += 129 if bad >= patience: # early stopping30 break3132model.load_state_dict(torch.load("best.pt"))33test_loss = evaluate(model, DataLoader(test_ds, batch_size=256)) # once, at the very endCheck yourself
You train for 80 epochs, keep the checkpoint with the lowest validation loss (epoch 34), then try three learning rates and keep the one with the best validation loss. Which number is an unbiased estimate of performance on new data?
Check yourself
You have 2 million labelled images. Which split is most sensible?
Remember
- ✓Train fits weights, validation makes every decision (including which epoch to keep), and test is evaluated once.
- ✓Each epoch is a candidate model: monitor validation loss every epoch, save the best checkpoint, stop early on it.
- ✓Size held-out sets in absolute terms: SE . 98/1/1 is fine at millions of examples and useless at thousands.
- ✓k-fold costs k training runs, which is why deep learning usually uses one fixed validation set.
- ✓Split before augmenting, deduplicate, split time series by time, group by patient or user, and fit normalisation on train only.
- ✓Evaluate with
model.eval()andtorch.no_grad().