Data augmentation

Train on randomly transformed copies of each example (flips, crops, colour jitter, masking) whose label you know is unchanged. It encodes an invariance as data, and it is often the strongest regularizer available, but only if the transform really preserves the label.

12 min read1208 wordscore0 viewsupdated 2026-10-11

Definition

Data augmentation replaces each training example (x,y)(x,y) by a randomly transformed version (T(x), y)(T(x),\,y), drawn fresh every time the example is used, where TT comes from a family of transformations that are believed not to change the label. The model never sees exactly the same input twice, and it is pushed to give the same answer for an input and its transformed versions.

Most regularizers limit the model: they shrink its weights, drop its units or stop it early. Augmentation instead adds information. Saying "a horizontally flipped cat is still a cat" is knowledge about the task that the training set alone does not contain. Written as transformations of the data, that knowledge is easy to state and works with any architecture and loss. In computer vision it is usually the single most effective way to reduce overfitting, and modern image classifiers are trained with heavy augmentation as a matter of course.

It is also one of the oldest tricks. Simard, Steinkraus and Platt (2003) reached state-of-the-art MNIST accuracy with affine and elastic distortions of the digits. AlexNet (Krizhevsky, Sutskever and Hinton, 2012) trained on random 224×224224\times224 crops of 256×256256\times256 images and their horizontal mirrors, which they noted enlarges the training set by a factor of 2048, plus a PCA-based colour perturbation. At test time they averaged the predictions of ten patches. Later work automated the choice of transformations, and extended the idea to audio, text and mixed examples.

What it optimizes

min⁡θ 1n∑i=1n ET∼T[ℓ(fθ(T(xi)), yi)]\min_\theta\ \frac1n\sum_{i=1}^n\ \mathbb E_{T\sim\mathcal T}\Big[\ell\big(f_\theta(T(x_i)),\,y_i\big)\Big]
T\mathcal T
the augmentation distribution: which transforms, with what probability and magnitude
T(xi)T(x_i)
a fresh random transform of example ii, drawn each time it appears in a mini-batch
yiy_i
the label is kept: the assumption is p(y∣T(x))=p(y∣x)p(y\mid T(x))=p(y\mid x)
ℓ\ell
any loss; the gradient is the ordinary gradient, evaluated at the transformed input

Each mini-batch draws one transform per example, so SGD sees an unbiased sample of the expected loss. Nothing changes in the backward pass. The gradient is just ∇θℓ(fθ(T(xi)),yi)\nabla_\theta\ell(f_\theta(T(x_i)),y_i) for the sampled TT. The cost is in the data pipeline, not in the model.

Chapelle, Weston, Bottou and Vapnik (2000) called this vicinal risk minimization: replace each training point by a small distribution (a "vicinity") around it, instead of the point mass that empirical risk minimization uses. A Gaussian vicinity, adding noise of variance σ2\sigma^2 to the inputs, is approximately a smoothness penalty. Bishop (1995) showed that, for squared error and small noise, it is equivalent to Tikhonov regularization:

Eϵ∼N(0,σ2I)[(f(x+ϵ)−y)2]  ≈  (f(x)−y)2+σ2 ∥∇xf(x)∥2(+ a term that averages out near the optimum)\mathbb E_{\epsilon\sim\mathcal N(0,\sigma^2I)}\big[(f(x+\epsilon)-y)^2\big]\;\approx\;(f(x)-y)^2+\sigma^2\,\lVert\nabla_x f(x)\rVert^2\quad(+\ \text{a term that averages out near the optimum})
σ2∥∇xf∥2\sigma^2\lVert\nabla_x f\rVert^2
penalizes functions that change quickly with the input: a smoothness prior

Structured transforms (flips, rotations, crops) are a more specific prior: they say that ff should be invariant along particular directions in input space, the orbits of the transformation group. Averaging the loss over those orbits removes variance the model would otherwise spend fitting accidental orientations and positions (Chen, Dobriban and Lee, 2020, make this precise). Mixed-example methods (Mixup, CutMix) define the vicinity differently, by blending two examples and their labels; they have their own articles.

At a glance

Changes
the training inputs only; the model, loss and optimizer stay the same
Assumption
label-preserving transforms: p(y∣T(x))=p(y∣x)p(y\mid T(x))=p(y\mid x)
Hyperparameters
which transforms, their probabilities and magnitudes (or N and M for RandAugment)
Cost
CPU/GPU time in the data loader; often slower convergence, so train for more epochs
At evaluation
off (deterministic resize and centre crop), or test-time augmentation by averaging
PyTorch
torchvision.transforms.v2, torchaudio.transforms
class 1 iff r < 0.6 (dashed circle)test accuracy 94.3%no augmentation 66.3% · 800 test ptstrain set 14 real + 112 copiesaccuracy on the real 14 85.7%copies now mislabeled 0.0%test accuracy vs copies per point50751000124816● class 1 ● class 0 · big = real, small = copy · ○ copy with wrong label
transform
fig.A rotation-invariant task: class 1 (blue) inside the dashed circle r < 0.6, class 0 (orange) outside. 14 training points, a classifier on 49 Gaussian bump features fit by ridge least squares, evaluated on 800 test points. With no augmentation it reaches 66.3%. Rotating each point by a random angle in ±180°, 8 copies per point, gives 94.3% (the curve on the right: 82.8% with 1 copy, 93.9% with 2, 94.9% with 16). Rotation by only ±30° helps less (81.6%). Jitter with σ = 0.1 is a vicinal prior that is only roughly right: 77.8%, with 10.7% of the copies crossing the boundary. Scaling by up to ×1.5 is not label-preserving: 21.4% of the copies end up with the wrong label and accuracy drops below baseline, to 60.3%.

The standard toolbox

domaintransforminvariance assumedwhere it breaks
imagesrandom resized cropposition and scale of the objectsmall objects cropped out; counting tasks
imageshorizontal flipleft–right mirror symmetrytext, digits, left/right labels, some medical images
imagescolour jitter, grayscalelighting and white balancetasks where colour is the label (ripe vs unripe, traffic lights)
imagesCutout / random erasingrobustness to occlusionerasing the only discriminative region
audioSpecAugment: time warp, time and frequency masksmissing bits of the spectrogramvery short clips, single-frequency cues
textback-translation, synonym swapparaphrase keeps the meaningnegation, entity swaps, tasks sensitive to wording
anyGaussian input noiselocal smoothness (vicinal risk)noise larger than the class margin
Common augmentations and the invariance each one asserts

Choosing and tuning such a list by hand is tedious, so several methods automate it. AutoAugment (Cubuk et al., 2019) used reinforcement learning to search for a policy of sub-policies, each a pair of operations with a probability and magnitude, on a proxy task, and transferred it to the real one. The search was very expensive. RandAugment (Cubuk et al., 2020) replaced it with two numbers: apply NN operations chosen uniformly from a list of 14, all at one shared magnitude MM, and tune NN and MM by grid search. TrivialAugment (Müller and Hutter, 2021) goes further: one random operation per image at a uniformly random magnitude, with no hyperparameters at all, and it matches or beats the searched policies on standard benchmarks.

Outside vision the transforms are domain-specific. SpecAugment (Park et al., 2019) warps the time axis and masks random blocks of frequency channels and time steps in a log-mel spectrogram; it improved end-to-end speech recognition on LibriSpeech substantially without any change to the model. In text, label-preserving transforms are harder to guarantee. Back-translation (Sennrich, Haddow and Birch, 2016) translates monolingual target-language text into the source language to create synthetic training pairs for machine translation, and simple edits such as synonym replacement, insertion, swap and deletion (EDA, Wei and Zou, 2019) help small text classifiers.

Worked example: is the label preserved?

Use the figure's task (class 1 iff r=x12+x22<0.6r=\sqrt{x_1^2+x_2^2}<0.6) and the training point x=(0.3, 0.4)x=(0.3,\,0.4), which has r=0.5r=0.5 and class 1:

transformnew pointnew rtrue classstored label correct?
rotate by 90°(−0.4, 0.3)0.51yes: rotation never changes r
rotate by 37°(−0.0011, 0.5)0.51yes
scale × 1/1.5(0.2, 0.267)0.3331yes, here
scale × 1.5(0.45, 0.6)0.750no: the copy is mislabeled
Three transforms applied to one class-1 point (computed exactly)

Rotation is an exact symmetry of this task, so every copy is a correct, free training example. Scaling is not, and it fails exactly for the points near the decision boundary, which are the ones that matter most for where the boundary goes. That is why the figure's scale mode does worse than no augmentation at all. Real transforms are rarely perfect symmetries: a crop can cut out the object, and colour jitter can turn a ripe banana into an unripe one. Their strength is a trade-off between the variance they remove and the label noise they add.

One more effect is visible in the figure. With 8 rotated copies per point, the classifier gets only 85.7% of the 14 original training points right, against 100% without augmentation, while test accuracy jumps from 66.3% to 94.3%. Augmentation makes the training problem harder on purpose. Expect higher training loss, slower convergence and, with strong augmentation, a validation loss below the training loss.

python
1import torch
2from torchvision import tv_tensors
3from torchvision.transforms import v2
4
5MEAN, STD = [0.485, 0.456, 0.406], [0.229, 0.224, 0.225]
6
7train_tf = v2.Compose([
8 v2.ToImage(), # PIL / ndarray -> uint8 tv_tensors.Image
9 v2.RandomResizedCrop(224, scale=(0.08, 1.0), antialias=True),
10 v2.RandomHorizontalFlip(p=0.5),
11 v2.TrivialAugmentWide(), # or v2.RandAugment(num_ops=2, magnitude=9)
12 v2.ToDtype(torch.float32, scale=True), # uint8 [0, 255] -> float [0, 1]
13 v2.Normalize(MEAN, STD),
14 v2.RandomErasing(p=0.25), # Cutout-style occlusion
15])
16eval_tf = v2.Compose([ # no randomness at evaluation
17 v2.ToImage(), v2.Resize(256, antialias=True), v2.CenterCrop(224),
18 v2.ToDtype(torch.float32, scale=True), v2.Normalize(MEAN, STD),
19])
20
21# Detection: v2 transforms boxes (and masks) together with the image
22img = tv_tensors.Image(torch.randint(0, 256, (3, 256, 256), dtype=torch.uint8))
23boxes = tv_tensors.BoundingBoxes([[10, 20, 110, 220]], format="XYXY", canvas_size=(256, 256))
24img2, boxes2 = v2.RandomHorizontalFlip(p=1.0)(img, boxes)
25print(boxes2) # [[146, 20, 246, 220]]: x -> 256 - x
26
27# Audio: SpecAugment-style masking on a (channel, freq, time) spectrogram
28import torchaudio.transforms as AT
29spec_aug = torch.nn.Sequential(AT.FrequencyMasking(freq_mask_param=27), AT.TimeMasking(time_mask_param=100))
30
31# Test-time augmentation: average the softmax over the image and its mirror
32@torch.no_grad()
33def predict_tta(model, x):
34 model.eval()
35 p = model(x).softmax(-1) + model(torch.flip(x, dims=[-1])).softmax(-1)
36 return p / 2
An ImageNet-style pipeline with torchvision.transforms.v2, boxes that follow the image, SpecAugment masks and test-time augmentation

Test-time augmentation

Test-time augmentation (TTA) applies a few deterministic or random transforms to each test input and averages the model's predicted probabilities. It is the same expectation as in training, now taken over the prediction instead of the loss: p^(y∣x)=1m∑j=1mpθ(y∣Tj(x))\hat p(y\mid x)=\frac1m\sum_{j=1}^m p_\theta(y\mid T_j(x)). It typically gives a small accuracy gain and better-calibrated probabilities, at mm times the inference cost, and it only helps with transforms the model was trained to be invariant to. AlexNet's ten-crop evaluation is the classic example.

Use when

  • +Adds real prior knowledge (invariances) rather than just constraining the model
  • +Works with any architecture, loss and optimizer; nothing changes in the backward pass
  • +Often the biggest single generalization gain in vision and speech
  • +Automated policies (RandAugment, TrivialAugment) need little or no tuning

Avoid when

  • −Only as good as the invariance: label-changing transforms add label noise
  • −Slows convergence and needs more epochs; small models may underfit
  • −Data-loading cost can bottleneck training
  • −Hard to design label-preserving transforms for text, tabular and structured data

Check yourself

You train a digit classifier (0–9) with random rotations in ±180°. Test accuracy drops compared with ±15°. What is the most likely reason?

Check yourself

After adding strong augmentation, your training loss is higher than your validation loss. What should you conclude?

Remember

  • ✓Augmentation minimizes ET[ℓ(f(T(x)),y)]\mathbb E_T[\ell(f(T(x)),y)]: fresh random label-preserving transforms every time an example is used.
  • ✓It encodes invariances as data. Gaussian input noise is approximately a smoothness (Tikhonov) penalty; structured transforms are a stronger, task-specific prior.
  • ✓Standard tools: random resized crop, flips, colour jitter, random erasing; RandAugment (N, M) and TrivialAugment (no hyperparameters) automate the choice; SpecAugment for speech; back-translation for text.
  • ✓The transform must preserve the label. Otherwise it adds label noise, and the points it corrupts are the ones near the decision boundary.
  • ✓Augment training only; evaluate deterministically. Test-time augmentation averages predictions over a fixed set of transforms.
← back to Deep Learning roadmap