Mixup

Train on convex combinations of random pairs of examples and of their one-hot labels, with λ ~ Beta(α, α). The network is asked to behave linearly between examples, which smooths decision boundaries, improves calibration and costs two lines of code.

11 min read851 wordscore0 viewsupdated 2026-10-11

Definition

Mixup is a data augmentation that replaces each training pair (xa,ya)(x_a,y_a) by a blend with another random pair (xb,yb)(x_b,y_b): the input becomes λxa+(1−λ)xb\lambda x_a+(1-\lambda)x_b and the target becomes the same blend of the one-hot labels. The mixing weight λ\lambda is drawn from Beta(α,α)\mathrm{Beta}(\alpha,\alpha) for every batch.

An ordinary network trained to zero loss is only told what to output at the training points. Between them it is free to do anything, and in practice it extrapolates with high confidence and switches class abruptly, which shows up as over-confident predictions, sensitivity to small perturbations and memorization of noisy labels. Mixup gives the network targets for the space between examples: halfway between a cat and a dog, the answer should be half cat, half dog.

Zhang, Cisse, Dauphin and Lopez-Paz (2018), mixup: Beyond Empirical Risk Minimization (ICLR), introduced it. They reported lower error on ImageNet, CIFAR-10/100 and speech data, better robustness to corrupted labels and adversarial examples, and more stable GAN training. Thulasidasan et al. (2019) showed that mixup-trained networks are also markedly better calibrated. Verma et al. (2019) proposed Manifold Mixup, which mixes hidden representations at a randomly chosen layer instead of raw inputs. CutMix (Yun et al., 2019) mixes by pasting a patch instead of blending pixels.

Formula

λ∼Beta(α,α),x~=λxa+(1−λ) xb,y~=λya+(1−λ) yb\lambda\sim\mathrm{Beta}(\alpha,\alpha),\qquad \tilde x=\lambda x_a+(1-\lambda)\,x_b,\qquad \tilde y=\lambda y_a+(1-\lambda)\,y_b
(xa,ya),(xb,yb)(x_a,y_a),(x_b,y_b)
two training examples; in practice bb is a shuffled (or rolled) copy of the same batch
yy
the one-hot label vector
α\alpha
mixing strength. α→0\alpha\to0 is plain training; α=1\alpha=1 makes λ\lambda uniform; ImageNet recipes use 0.1 to 0.4, CIFAR uses 1

The loss is cross-entropy against the soft target y~\tilde y. Because cross-entropy is linear in the target, this is the same as mixing two ordinary losses, which is how most code writes it:

ℓ(f(x~),y~)=−∑ky~klog⁡pk=λ CE(f(x~),ya)+(1−λ) CE(f(x~),yb),∂ℓ∂z=p−y~\ell\big(f(\tilde x),\tilde y\big)=-\sum_k\tilde y_k\log p_k=\lambda\,\mathrm{CE}\big(f(\tilde x),y_a\big)+(1-\lambda)\,\mathrm{CE}\big(f(\tilde x),y_b\big),\qquad \frac{\partial\ell}{\partial z}=p-\tilde y
p=softmax⁡(z)p=\operatorname{softmax}(z)
the prediction for the mixed input
p−y~p-\tilde y
the logit gradient, as in Categorical cross-entropy: zero only if the network outputs exactly the blend

At a glance

Kind
data augmentation on (input, label) pairs; label-aware
Hyperparameter
α\alpha (one per run); one λ\lambda per batch
Cost
one weighted sum of two batches; no extra forward pass
Needs
a loss that accepts soft targets (CE with probabilities, BCE, KL)
Helps
generalization, calibration, robustness to label noise; strongest with long training schedules
Variants
Manifold Mixup (hidden layers), CutMix (patches), mixing in embedding space for text and audio

Why it works: vicinal risk minimization

Ordinary training minimizes the empirical risk: the average loss over a distribution that puts a point mass on each training pair. Vicinal risk minimization (Chapelle, Weston, Bottou and Vapnik, 2000) replaces each point mass by a small neighbourhood, a vicinity, and trains on samples from it. Classic augmentation is a vicinity built from human knowledge: crops, flips and colour jitter of the same image with the same label.

Remp=1n∑i=1nℓ(f(xi),yi)⟶Rmixup=1n2∑i,j Eλ ℓ(f(λxi+(1−λ)xj), λyi+(1−λ)yj)R_{\text{emp}}=\frac1n\sum_{i=1}^n\ell\big(f(x_i),y_i\big)\qquad\longrightarrow\qquad R_{\text{mixup}}=\frac1{n^2}\sum_{i,j}\ \mathbb E_{\lambda}\,\ell\big(f(\lambda x_i+(1-\lambda)x_j),\ \lambda y_i+(1-\lambda)y_j\big)
RempR_{\text{emp}}
empirical risk: only the training points themselves
RmixupR_{\text{mixup}}
vicinal risk with the mixup vicinity: every segment between two training examples, labels interpolated along it

Mixup's vicinity is generic: it needs no knowledge of the data, and it links examples of different classes. It encodes one prior: along the segment between two examples, the prediction should move linearly. A network that satisfies it cannot jump from 99% class A to 99% class B in a tiny gap, so its boundaries are smoother and its confidence falls off between classes. That is why mixup improves calibration and resists noisy labels: a memorized wrong label is contradicted by all the mixed points around it.

The Beta distribution decides how far into the segment the samples go. For small α\alpha it is U-shaped, so most λ\lambda are near 0 or 1 and the mixed image is mostly one example with a faint trace of the other. At α=0.2\alpha=0.2 about two thirds of draws fall outside [0.1, 0.9][0.1,\,0.9]; at α=1\alpha=1 only 20% do. Larger α\alpha regularizes more and can underfit.

ERM (no mixup)mixup, α = 0.201500 Adam steps, 40 train / 400 test pointsERMmixuptrain accuracy1.0000.975test accuracy0.8550.897test NLL0.9110.300mean confidence0.9790.919ECE (10 bins)0.1240.048λ ~ Beta(0.20, 0.20) · this batch λ = 1.00000.51shade = confidence max(p, 1−p) · ━ p = 0.5 boundary · ◆ mixed inputs, colour = soft label
x̃ = λxₐ + (1−λ)x_b, ỹ = λyₐ + (1−λ)y_b
fig.The same 16-unit tanh MLP trained for 1500 full-batch Adam steps on 40 noisy two-moons points, without (left) and with mixup (right). Shading is the confidence max(p, 1 − p), the black line is p = 0.5, diamonds are one mixup batch coloured by its soft label. On data set 1, plain ERM fits all 40 points (train NLL 0.001) with a wiggly, near-certain boundary: test accuracy 0.855, test NLL 0.911, ECE 0.124. Mixup with α = 0.2 gives 0.897, 0.300 and 0.048, with train accuracy 0.975. Across all 9 data sets mixup lowers test NLL and ECE every time and raises accuracy in 8 of 9 (means 0.835 → 0.873 accuracy, 1.02 → 0.31 NLL). Push α to 2 and the mean test confidence keeps falling, from 0.919 to 0.875.

Worked example: one mixed sample

Three classes. Example aa is class 0 with input (0.2, 0.9)(0.2,\,0.9), example bb is class 2 with input (0.8, 0.1)(0.8,\,0.1), and the batch draws λ=0.7\lambda=0.7. Numbers checked in Node:

λ = 0.7, logits for the mixed input z = (2.0, 1.0, 0.1)

  1. 1Mixed input x~=0.7(0.2,0.9)+0.3(0.8,0.1)=(0.38, 0.66)\tilde x=0.7(0.2,0.9)+0.3(0.8,0.1)=(0.38,\,0.66). Soft target y~=0.7(1,0,0)+0.3(0,0,1)=(0.7, 0, 0.3)\tilde y=0.7(1,0,0)+0.3(0,0,1)=(0.7,\,0,\,0.3).
  2. 2Prediction p=softmax⁡(z)=(0.6590, 0.2424, 0.0986)p=\operatorname{softmax}(z)=(0.6590,\,0.2424,\,0.0986).
  3. 3Per-label losses: CE(ya)=−ln⁡0.6590=0.4170\mathrm{CE}(y_a)=-\ln0.6590=0.4170 and CE(yb)=−ln⁡0.0986=2.3170\mathrm{CE}(y_b)=-\ln0.0986=2.3170.
  4. 4Mixed loss 0.7⋅0.4170+0.3⋅2.3170=0.98700.7\cdot0.4170+0.3\cdot2.3170=0.9870, identical to −∑ky~kln⁡pk=0.9870-\sum_k\tilde y_k\ln p_k=0.9870 with the soft target.
  5. 5Logit gradient p−y~=(−0.0410, 0.2424, −0.2014)p-\tilde y=(-0.0410,\,0.2424,\,-0.2014): class 0 is already about right (0.66 vs 0.7), class 1 must drop, class 2 must rise from 0.10 toward 0.30.

With a hard label the gradient on class 0 would be 0.659−1=−0.3410.659-1=-0.341, pushing toward certainty. With the mixed target the network is nearly done on class 0. That is the calibration effect in miniature: confidence is rewarded only in proportion to how much of the input actually is class 0.

python
1import torch, torch.nn as nn, torch.nn.functional as F
2from torchvision.transforms import v2
3
4NUM_CLASSES = 10
5model = nn.Sequential(nn.Flatten(), nn.Linear(3 * 32 * 32, NUM_CLASSES))
6opt = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9)
7
8# --- by hand: one lambda per batch, partner = a shuffled copy of the batch ---
9beta = torch.distributions.Beta(torch.tensor(0.2), torch.tensor(0.2))
10
11def mixup_step(x, y):
12 lam = beta.sample().item()
13 idx = torch.randperm(x.size(0))
14 x_mix = lam * x + (1 - lam) * x[idx]
15 logits = model(x_mix)
16 loss = lam * F.cross_entropy(logits, y) + (1 - lam) * F.cross_entropy(logits, y[idx])
17 opt.zero_grad(); loss.backward(); opt.step()
18 return loss.item()
19
20# --- torchvision: operates on a whole batch (after the DataLoader), returns soft labels ---
21mixup = v2.MixUp(alpha=0.2, num_classes=NUM_CLASSES)
22x, y = torch.rand(8, 3, 32, 32), torch.randint(0, NUM_CLASSES, (8,))
23x_mix, y_soft = mixup(x, y) # y_soft: (8, 10) probabilities; partner is the batch rolled by one
24loss = F.cross_entropy(model(x_mix), y_soft) # cross_entropy accepts probability targets
25
26# Evaluate on clean inputs: no mixup at test time.
Mixup by hand and with torchvision.transforms.v2.MixUp

Use when

  • +Two lines of code, almost free compute, one hyperparameter
  • +Smoother boundaries and better calibration (lower NLL and ECE)
  • +More robust to noisy labels and to small input perturbations
  • +Domain-agnostic: works on images, audio, tabular data and embeddings

Avoid when

  • −Blended inputs are unrealistic, which can hurt tasks that rely on local detail (detection, localization)
  • −Slows fitting of the training set; benefits need longer schedules, and large α\alpha underfits
  • −Can make models under-confident when combined with other softening methods
  • −Needs soft-target losses; not directly applicable to tasks without mixable labels

Check yourself

With λ = 0.7, a mixed image of a cat (a) and a dog (b) is classified with p(cat) = 0.5, p(dog) = 0.5. What is the mixup loss?

Check yourself

You set α = 0.1 for mixup. What do most mixed training images look like?

Remember

  • ✓x~=λxa+(1−λ)xb\tilde x=\lambda x_a+(1-\lambda)x_b, y~=λya+(1−λ)yb\tilde y=\lambda y_a+(1-\lambda)y_b, λ∼Beta(α,α)\lambda\sim\mathrm{Beta}(\alpha,\alpha), one λ\lambda per batch (Zhang et al., 2018).
  • ✓Loss =λ CE(ya)+(1−λ) CE(yb)=\lambda\,\mathrm{CE}(y_a)+(1-\lambda)\,\mathrm{CE}(y_b), equal to CE with the soft target; logit gradient p−y~p-\tilde y.
  • ✓It is vicinal risk minimization with a generic vicinity: segments between examples, labels interpolated linearly.
  • ✓Effect: smoother boundaries, lower confidence between classes, better calibration and robustness to noisy labels.
  • ✓Small α\alpha (0.1 to 0.4) mixes mildly; torchvision.transforms.v2.MixUp works on batches and returns soft labels. Manifold Mixup mixes hidden layers.
← back to Deep Learning roadmap