Definition
Mixup is a data augmentation that replaces each training pair by a blend with another random pair : the input becomes and the target becomes the same blend of the one-hot labels. The mixing weight is drawn from for every batch.
An ordinary network trained to zero loss is only told what to output at the training points. Between them it is free to do anything, and in practice it extrapolates with high confidence and switches class abruptly, which shows up as over-confident predictions, sensitivity to small perturbations and memorization of noisy labels. Mixup gives the network targets for the space between examples: halfway between a cat and a dog, the answer should be half cat, half dog.
Zhang, Cisse, Dauphin and Lopez-Paz (2018), mixup: Beyond Empirical Risk Minimization (ICLR), introduced it. They reported lower error on ImageNet, CIFAR-10/100 and speech data, better robustness to corrupted labels and adversarial examples, and more stable GAN training. Thulasidasan et al. (2019) showed that mixup-trained networks are also markedly better calibrated. Verma et al. (2019) proposed Manifold Mixup, which mixes hidden representations at a randomly chosen layer instead of raw inputs. CutMix (Yun et al., 2019) mixes by pasting a patch instead of blending pixels.
Formula
- two training examples; in practice is a shuffled (or rolled) copy of the same batch
- the one-hot label vector
- mixing strength. is plain training; makes uniform; ImageNet recipes use 0.1 to 0.4, CIFAR uses 1
The loss is cross-entropy against the soft target . Because cross-entropy is linear in the target, this is the same as mixing two ordinary losses, which is how most code writes it:
- the prediction for the mixed input
- the logit gradient, as in Categorical cross-entropy: zero only if the network outputs exactly the blend
At a glance
- Kind
- data augmentation on (input, label) pairs; label-aware
- Hyperparameter
- (one per run); one per batch
- Cost
- one weighted sum of two batches; no extra forward pass
- Needs
- a loss that accepts soft targets (CE with probabilities, BCE, KL)
- Helps
- generalization, calibration, robustness to label noise; strongest with long training schedules
- Variants
- Manifold Mixup (hidden layers), CutMix (patches), mixing in embedding space for text and audio
Why it works: vicinal risk minimization
Ordinary training minimizes the empirical risk: the average loss over a distribution that puts a point mass on each training pair. Vicinal risk minimization (Chapelle, Weston, Bottou and Vapnik, 2000) replaces each point mass by a small neighbourhood, a vicinity, and trains on samples from it. Classic augmentation is a vicinity built from human knowledge: crops, flips and colour jitter of the same image with the same label.
- empirical risk: only the training points themselves
- vicinal risk with the mixup vicinity: every segment between two training examples, labels interpolated along it
Mixup's vicinity is generic: it needs no knowledge of the data, and it links examples of different classes. It encodes one prior: along the segment between two examples, the prediction should move linearly. A network that satisfies it cannot jump from 99% class A to 99% class B in a tiny gap, so its boundaries are smoother and its confidence falls off between classes. That is why mixup improves calibration and resists noisy labels: a memorized wrong label is contradicted by all the mixed points around it.
The Beta distribution decides how far into the segment the samples go. For small it is U-shaped, so most are near 0 or 1 and the mixed image is mostly one example with a faint trace of the other. At about two thirds of draws fall outside ; at only 20% do. Larger regularizes more and can underfit.
Worked example: one mixed sample
Three classes. Example is class 0 with input , example is class 2 with input , and the batch draws . Numbers checked in Node:
λ = 0.7, logits for the mixed input z = (2.0, 1.0, 0.1)
- 1Mixed input . Soft target .
- 2Prediction .
- 3Per-label losses: and .
- 4Mixed loss , identical to with the soft target.
- 5Logit gradient : class 0 is already about right (0.66 vs 0.7), class 1 must drop, class 2 must rise from 0.10 toward 0.30.
With a hard label the gradient on class 0 would be , pushing toward certainty. With the mixed target the network is nearly done on class 0. That is the calibration effect in miniature: confidence is rewarded only in proportion to how much of the input actually is class 0.
1import torch, torch.nn as nn, torch.nn.functional as F2from torchvision.transforms import v234NUM_CLASSES = 105model = nn.Sequential(nn.Flatten(), nn.Linear(3 * 32 * 32, NUM_CLASSES))6opt = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9)78# --- by hand: one lambda per batch, partner = a shuffled copy of the batch ---9beta = torch.distributions.Beta(torch.tensor(0.2), torch.tensor(0.2))1011def mixup_step(x, y):12 lam = beta.sample().item()13 idx = torch.randperm(x.size(0))14 x_mix = lam * x + (1 - lam) * x[idx]15 logits = model(x_mix)16 loss = lam * F.cross_entropy(logits, y) + (1 - lam) * F.cross_entropy(logits, y[idx])17 opt.zero_grad(); loss.backward(); opt.step()18 return loss.item()1920# --- torchvision: operates on a whole batch (after the DataLoader), returns soft labels ---21mixup = v2.MixUp(alpha=0.2, num_classes=NUM_CLASSES)22x, y = torch.rand(8, 3, 32, 32), torch.randint(0, NUM_CLASSES, (8,))23x_mix, y_soft = mixup(x, y) # y_soft: (8, 10) probabilities; partner is the batch rolled by one24loss = F.cross_entropy(model(x_mix), y_soft) # cross_entropy accepts probability targets2526# Evaluate on clean inputs: no mixup at test time.Use when
- +Two lines of code, almost free compute, one hyperparameter
- +Smoother boundaries and better calibration (lower NLL and ECE)
- +More robust to noisy labels and to small input perturbations
- +Domain-agnostic: works on images, audio, tabular data and embeddings
Avoid when
- −Blended inputs are unrealistic, which can hurt tasks that rely on local detail (detection, localization)
- −Slows fitting of the training set; benefits need longer schedules, and large underfits
- −Can make models under-confident when combined with other softening methods
- −Needs soft-target losses; not directly applicable to tasks without mixable labels
Check yourself
With λ = 0.7, a mixed image of a cat (a) and a dog (b) is classified with p(cat) = 0.5, p(dog) = 0.5. What is the mixup loss?
Check yourself
You set α = 0.1 for mixup. What do most mixed training images look like?
Remember
- ✓, , , one per batch (Zhang et al., 2018).
- ✓Loss , equal to CE with the soft target; logit gradient .
- ✓It is vicinal risk minimization with a generic vicinity: segments between examples, labels interpolated linearly.
- ✓Effect: smoother boundaries, lower confidence between classes, better calibration and robustness to noisy labels.
- ✓Small (0.1 to 0.4) mixes mildly; torchvision.transforms.v2.MixUp works on batches and returns soft labels. Manifold Mixup mixes hidden layers.