Definition
Data augmentation replaces each training example by a randomly transformed version , drawn fresh every time the example is used, where comes from a family of transformations that are believed not to change the label. The model never sees exactly the same input twice, and it is pushed to give the same answer for an input and its transformed versions.
Most regularizers limit the model: they shrink its weights, drop its units or stop it early. Augmentation instead adds information. Saying "a horizontally flipped cat is still a cat" is knowledge about the task that the training set alone does not contain. Written as transformations of the data, that knowledge is easy to state and works with any architecture and loss. In computer vision it is usually the single most effective way to reduce overfitting, and modern image classifiers are trained with heavy augmentation as a matter of course.
It is also one of the oldest tricks. Simard, Steinkraus and Platt (2003) reached state-of-the-art MNIST accuracy with affine and elastic distortions of the digits. AlexNet (Krizhevsky, Sutskever and Hinton, 2012) trained on random crops of images and their horizontal mirrors, which they noted enlarges the training set by a factor of 2048, plus a PCA-based colour perturbation. At test time they averaged the predictions of ten patches. Later work automated the choice of transformations, and extended the idea to audio, text and mixed examples.
What it optimizes
- the augmentation distribution: which transforms, with what probability and magnitude
- a fresh random transform of example , drawn each time it appears in a mini-batch
- the label is kept: the assumption is
- any loss; the gradient is the ordinary gradient, evaluated at the transformed input
Each mini-batch draws one transform per example, so SGD sees an unbiased sample of the expected loss. Nothing changes in the backward pass. The gradient is just for the sampled . The cost is in the data pipeline, not in the model.
Chapelle, Weston, Bottou and Vapnik (2000) called this vicinal risk minimization: replace each training point by a small distribution (a "vicinity") around it, instead of the point mass that empirical risk minimization uses. A Gaussian vicinity, adding noise of variance to the inputs, is approximately a smoothness penalty. Bishop (1995) showed that, for squared error and small noise, it is equivalent to Tikhonov regularization:
- penalizes functions that change quickly with the input: a smoothness prior
Structured transforms (flips, rotations, crops) are a more specific prior: they say that should be invariant along particular directions in input space, the orbits of the transformation group. Averaging the loss over those orbits removes variance the model would otherwise spend fitting accidental orientations and positions (Chen, Dobriban and Lee, 2020, make this precise). Mixed-example methods (Mixup, CutMix) define the vicinity differently, by blending two examples and their labels; they have their own articles.
At a glance
- Changes
- the training inputs only; the model, loss and optimizer stay the same
- Assumption
- label-preserving transforms:
- Hyperparameters
- which transforms, their probabilities and magnitudes (or N and M for RandAugment)
- Cost
- CPU/GPU time in the data loader; often slower convergence, so train for more epochs
- At evaluation
- off (deterministic resize and centre crop), or test-time augmentation by averaging
- PyTorch
torchvision.transforms.v2,torchaudio.transforms
The standard toolbox
| domain | transform | invariance assumed | where it breaks |
|---|---|---|---|
| images | random resized crop | position and scale of the object | small objects cropped out; counting tasks |
| images | horizontal flip | left–right mirror symmetry | text, digits, left/right labels, some medical images |
| images | colour jitter, grayscale | lighting and white balance | tasks where colour is the label (ripe vs unripe, traffic lights) |
| images | Cutout / random erasing | robustness to occlusion | erasing the only discriminative region |
| audio | SpecAugment: time warp, time and frequency masks | missing bits of the spectrogram | very short clips, single-frequency cues |
| text | back-translation, synonym swap | paraphrase keeps the meaning | negation, entity swaps, tasks sensitive to wording |
| any | Gaussian input noise | local smoothness (vicinal risk) | noise larger than the class margin |
Choosing and tuning such a list by hand is tedious, so several methods automate it. AutoAugment (Cubuk et al., 2019) used reinforcement learning to search for a policy of sub-policies, each a pair of operations with a probability and magnitude, on a proxy task, and transferred it to the real one. The search was very expensive. RandAugment (Cubuk et al., 2020) replaced it with two numbers: apply operations chosen uniformly from a list of 14, all at one shared magnitude , and tune and by grid search. TrivialAugment (Müller and Hutter, 2021) goes further: one random operation per image at a uniformly random magnitude, with no hyperparameters at all, and it matches or beats the searched policies on standard benchmarks.
Outside vision the transforms are domain-specific. SpecAugment (Park et al., 2019) warps the time axis and masks random blocks of frequency channels and time steps in a log-mel spectrogram; it improved end-to-end speech recognition on LibriSpeech substantially without any change to the model. In text, label-preserving transforms are harder to guarantee. Back-translation (Sennrich, Haddow and Birch, 2016) translates monolingual target-language text into the source language to create synthetic training pairs for machine translation, and simple edits such as synonym replacement, insertion, swap and deletion (EDA, Wei and Zou, 2019) help small text classifiers.
Worked example: is the label preserved?
Use the figure's task (class 1 iff ) and the training point , which has and class 1:
| transform | new point | new r | true class | stored label correct? |
|---|---|---|---|---|
| rotate by 90° | (−0.4, 0.3) | 0.5 | 1 | yes: rotation never changes r |
| rotate by 37° | (−0.0011, 0.5) | 0.5 | 1 | yes |
| scale × 1/1.5 | (0.2, 0.267) | 0.333 | 1 | yes, here |
| scale × 1.5 | (0.45, 0.6) | 0.75 | 0 | no: the copy is mislabeled |
Rotation is an exact symmetry of this task, so every copy is a correct, free training example. Scaling is not, and it fails exactly for the points near the decision boundary, which are the ones that matter most for where the boundary goes. That is why the figure's scale mode does worse than no augmentation at all. Real transforms are rarely perfect symmetries: a crop can cut out the object, and colour jitter can turn a ripe banana into an unripe one. Their strength is a trade-off between the variance they remove and the label noise they add.
One more effect is visible in the figure. With 8 rotated copies per point, the classifier gets only 85.7% of the 14 original training points right, against 100% without augmentation, while test accuracy jumps from 66.3% to 94.3%. Augmentation makes the training problem harder on purpose. Expect higher training loss, slower convergence and, with strong augmentation, a validation loss below the training loss.
1import torch2from torchvision import tv_tensors3from torchvision.transforms import v245MEAN, STD = [0.485, 0.456, 0.406], [0.229, 0.224, 0.225]67train_tf = v2.Compose([8 v2.ToImage(), # PIL / ndarray -> uint8 tv_tensors.Image9 v2.RandomResizedCrop(224, scale=(0.08, 1.0), antialias=True),10 v2.RandomHorizontalFlip(p=0.5),11 v2.TrivialAugmentWide(), # or v2.RandAugment(num_ops=2, magnitude=9)12 v2.ToDtype(torch.float32, scale=True), # uint8 [0, 255] -> float [0, 1]13 v2.Normalize(MEAN, STD),14 v2.RandomErasing(p=0.25), # Cutout-style occlusion15])16eval_tf = v2.Compose([ # no randomness at evaluation17 v2.ToImage(), v2.Resize(256, antialias=True), v2.CenterCrop(224),18 v2.ToDtype(torch.float32, scale=True), v2.Normalize(MEAN, STD),19])2021# Detection: v2 transforms boxes (and masks) together with the image22img = tv_tensors.Image(torch.randint(0, 256, (3, 256, 256), dtype=torch.uint8))23boxes = tv_tensors.BoundingBoxes([[10, 20, 110, 220]], format="XYXY", canvas_size=(256, 256))24img2, boxes2 = v2.RandomHorizontalFlip(p=1.0)(img, boxes)25print(boxes2) # [[146, 20, 246, 220]]: x -> 256 - x2627# Audio: SpecAugment-style masking on a (channel, freq, time) spectrogram28import torchaudio.transforms as AT29spec_aug = torch.nn.Sequential(AT.FrequencyMasking(freq_mask_param=27), AT.TimeMasking(time_mask_param=100))3031# Test-time augmentation: average the softmax over the image and its mirror32@torch.no_grad()33def predict_tta(model, x):34 model.eval()35 p = model(x).softmax(-1) + model(torch.flip(x, dims=[-1])).softmax(-1)36 return p / 2Test-time augmentation
Test-time augmentation (TTA) applies a few deterministic or random transforms to each test input and averages the model's predicted probabilities. It is the same expectation as in training, now taken over the prediction instead of the loss: . It typically gives a small accuracy gain and better-calibrated probabilities, at times the inference cost, and it only helps with transforms the model was trained to be invariant to. AlexNet's ten-crop evaluation is the classic example.
Use when
- +Adds real prior knowledge (invariances) rather than just constraining the model
- +Works with any architecture, loss and optimizer; nothing changes in the backward pass
- +Often the biggest single generalization gain in vision and speech
- +Automated policies (RandAugment, TrivialAugment) need little or no tuning
Avoid when
- −Only as good as the invariance: label-changing transforms add label noise
- −Slows convergence and needs more epochs; small models may underfit
- −Data-loading cost can bottleneck training
- −Hard to design label-preserving transforms for text, tabular and structured data
Check yourself
You train a digit classifier (0–9) with random rotations in ±180°. Test accuracy drops compared with ±15°. What is the most likely reason?
Check yourself
After adding strong augmentation, your training loss is higher than your validation loss. What should you conclude?
Remember
- ✓Augmentation minimizes : fresh random label-preserving transforms every time an example is used.
- ✓It encodes invariances as data. Gaussian input noise is approximately a smoothness (Tikhonov) penalty; structured transforms are a stronger, task-specific prior.
- ✓Standard tools: random resized crop, flips, colour jitter, random erasing; RandAugment (N, M) and TrivialAugment (no hyperparameters) automate the choice; SpecAugment for speech; back-translation for text.
- ✓The transform must preserve the label. Otherwise it adds label noise, and the points it corrupts are the ones near the decision boundary.
- ✓Augment training only; evaluate deterministically. Test-time augmentation averages predictions over a fixed set of transforms.