# Mixup

> Train on convex combinations of random pairs of examples and of their one-hot labels, with λ ~ Beta(α, α). The network is asked to behave linearly between examples, which smooths decision boundaries, improves calibration and costs two lines of code.

Source: https://www.mlfoundry.in/learn/mixup  
Section: Deep Learning › Regularization & Normalization  
Level: core · 11 min read · updated 2026-10-11

**Mixup:** is a data augmentation that replaces each training pair $(x_a,y_a)$ by a blend with another random pair $(x_b,y_b)$: the input becomes $\lambda x_a+(1-\lambda)x_b$ and the target becomes the **same** blend of the one-hot labels. The mixing weight $\lambda$ is drawn from $\mathrm{Beta}(\alpha,\alpha)$ for every batch.

An ordinary network trained to zero loss is only told what to output **at** the training points. Between them it is free to do anything, and in practice it extrapolates with high confidence and switches class abruptly, which shows up as over-confident predictions, sensitivity to small perturbations and memorization of noisy labels. Mixup gives the network targets for the space between examples: halfway between a cat and a dog, the answer should be half cat, half dog.

**Zhang, Cisse, Dauphin and Lopez-Paz (2018)**, *mixup: Beyond Empirical Risk Minimization* (ICLR), introduced it. They reported lower error on ImageNet, CIFAR-10/100 and speech data, better robustness to corrupted labels and adversarial examples, and more stable GAN training. Thulasidasan et al. (2019) showed that mixup-trained networks are also markedly better **calibrated**. Verma et al. (2019) proposed **Manifold Mixup**, which mixes hidden representations at a randomly chosen layer instead of raw inputs. **CutMix** (Yun et al., 2019) mixes by pasting a patch instead of blending pixels.

## Formula

$$
\lambda\sim\mathrm{Beta}(\alpha,\alpha),\qquad \tilde x=\lambda x_a+(1-\lambda)\,x_b,\qquad \tilde y=\lambda y_a+(1-\lambda)\,y_b
$$

- $(x_a,y_a),(x_b,y_b)$: two training examples; in practice $b$ is a shuffled (or rolled) copy of the same batch
- $y$: the one-hot label vector
- $\alpha$: mixing strength. $\alpha\to0$ is plain training; $\alpha=1$ makes $\lambda$ uniform; ImageNet recipes use 0.1 to 0.4, CIFAR uses 1

The loss is cross-entropy against the soft target $\tilde y$. Because cross-entropy is **linear in the target**, this is the same as mixing two ordinary losses, which is how most code writes it:

$$
\ell\big(f(\tilde x),\tilde y\big)=-\sum_k\tilde y_k\log p_k=\lambda\,\mathrm{CE}\big(f(\tilde x),y_a\big)+(1-\lambda)\,\mathrm{CE}\big(f(\tilde x),y_b\big),\qquad \frac{\partial\ell}{\partial z}=p-\tilde y
$$

- $p=\operatorname{softmax}(z)$: the prediction for the **mixed** input
- $p-\tilde y$: the logit gradient, as in **Categorical cross-entropy**: zero only if the network outputs exactly the blend

**At a glance**

- **Kind:** data augmentation on (input, label) pairs; label-aware
- **Hyperparameter:** $\alpha$ (one per run); one $\lambda$ per batch
- **Cost:** one weighted sum of two batches; no extra forward pass
- **Needs:** a loss that accepts soft targets (CE with probabilities, BCE, KL)
- **Helps:** generalization, calibration, robustness to label noise; strongest with long training schedules
- **Variants:** Manifold Mixup (hidden layers), CutMix (patches), mixing in embedding space for text and audio

## Why it works: vicinal risk minimization

Ordinary training minimizes the **empirical risk**: the average loss over a distribution that puts a point mass on each training pair. **Vicinal risk minimization** (Chapelle, Weston, Bottou and Vapnik, 2000) replaces each point mass by a small neighbourhood, a vicinity, and trains on samples from it. Classic augmentation is a vicinity built from human knowledge: crops, flips and colour jitter of the **same** image with the **same** label.

$$
R_{\text{emp}}=\frac1n\sum_{i=1}^n\ell\big(f(x_i),y_i\big)\qquad\longrightarrow\qquad R_{\text{mixup}}=\frac1{n^2}\sum_{i,j}\ \mathbb E_{\lambda}\,\ell\big(f(\lambda x_i+(1-\lambda)x_j),\ \lambda y_i+(1-\lambda)y_j\big)
$$

- $R_{\text{emp}}$: empirical risk: only the training points themselves
- $R_{\text{mixup}}$: vicinal risk with the mixup vicinity: every segment between two training examples, labels interpolated along it

Mixup's vicinity is generic: it needs no knowledge of the data, and it links examples of **different** classes. It encodes one prior: along the segment between two examples, the prediction should move linearly. A network that satisfies it cannot jump from 99% class A to 99% class B in a tiny gap, so its boundaries are smoother and its confidence falls off between classes. That is why mixup improves calibration and resists noisy labels: a memorized wrong label is contradicted by all the mixed points around it.

The Beta distribution decides **how far** into the segment the samples go. For small $\alpha$ it is U-shaped, so most $\lambda$ are near 0 or 1 and the mixed image is mostly one example with a faint trace of the other. At $\alpha=0.2$ about two thirds of draws fall outside $[0.1,\,0.9]$; at $\alpha=1$ only 20% do. Larger $\alpha$ regularizes more and can underfit.

> Figure: The same 16-unit tanh MLP trained for 1500 full-batch Adam steps on 40 noisy two-moons points, without (left) and with mixup (right). Shading is the confidence max(p, 1 − p), the black line is p = 0.5, diamonds are one mixup batch coloured by its soft label. On data set 1, plain ERM fits all 40 points (train NLL 0.001) with a wiggly, near-certain boundary: test accuracy 0.855, test NLL 0.911, ECE 0.124. Mixup with α = 0.2 gives 0.897, 0.300 and 0.048, with train accuracy 0.975. Across all 9 data sets mixup lowers test NLL and ECE every time and raises accuracy in 8 of 9 (means 0.835 → 0.873 accuracy, 1.02 → 0.31 NLL). Push α to 2 and the mean test confidence keeps falling, from 0.919 to 0.875.

## Worked example: one mixed sample

Three classes. Example $a$ is class 0 with input $(0.2,\,0.9)$, example $b$ is class 2 with input $(0.8,\,0.1)$, and the batch draws $\lambda=0.7$. Numbers checked in Node:

**λ = 0.7, logits for the mixed input z = (2.0, 1.0, 0.1)**

1. Mixed input $\tilde x=0.7(0.2,0.9)+0.3(0.8,0.1)=(0.38,\,0.66)$. Soft target $\tilde y=0.7(1,0,0)+0.3(0,0,1)=(0.7,\,0,\,0.3)$.
2. Prediction $p=\operatorname{softmax}(z)=(0.6590,\,0.2424,\,0.0986)$.
3. Per-label losses: $\mathrm{CE}(y_a)=-\ln0.6590=0.4170$ and $\mathrm{CE}(y_b)=-\ln0.0986=2.3170$.
4. Mixed loss $0.7\cdot0.4170+0.3\cdot2.3170=0.9870$, identical to $-\sum_k\tilde y_k\ln p_k=0.9870$ with the soft target.
5. Logit gradient $p-\tilde y=(-0.0410,\,0.2424,\,-0.2014)$: class 0 is already about right (0.66 vs 0.7), class 1 must drop, class 2 must rise from 0.10 toward 0.30.

With a hard label the gradient on class 0 would be $0.659-1=-0.341$, pushing toward certainty. With the mixed target the network is nearly done on class 0. That is the calibration effect in miniature: confidence is rewarded only in proportion to how much of the input actually is class 0.

```python
import torch, torch.nn as nn, torch.nn.functional as F
from torchvision.transforms import v2

NUM_CLASSES = 10
model = nn.Sequential(nn.Flatten(), nn.Linear(3 * 32 * 32, NUM_CLASSES))
opt = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9)

# --- by hand: one lambda per batch, partner = a shuffled copy of the batch ---
beta = torch.distributions.Beta(torch.tensor(0.2), torch.tensor(0.2))

def mixup_step(x, y):
    lam = beta.sample().item()
    idx = torch.randperm(x.size(0))
    x_mix = lam * x + (1 - lam) * x[idx]
    logits = model(x_mix)
    loss = lam * F.cross_entropy(logits, y) + (1 - lam) * F.cross_entropy(logits, y[idx])
    opt.zero_grad(); loss.backward(); opt.step()
    return loss.item()

# --- torchvision: operates on a whole batch (after the DataLoader), returns soft labels ---
mixup = v2.MixUp(alpha=0.2, num_classes=NUM_CLASSES)
x, y = torch.rand(8, 3, 32, 32), torch.randint(0, NUM_CLASSES, (8,))
x_mix, y_soft = mixup(x, y)       # y_soft: (8, 10) probabilities; partner is the batch rolled by one
loss = F.cross_entropy(model(x_mix), y_soft)   # cross_entropy accepts probability targets

# Evaluate on clean inputs: no mixup at test time.
```

Mixup by hand and with torchvision.transforms.v2.MixUp

> **Mix the labels too, and only in training:** Mixing inputs while keeping the label of $x_a$ is just noise injection with a wrong label 50% of the time when $\lambda\approx0.5$. The label must be mixed with the same $\lambda$, and the loss must accept soft targets (`F.cross_entropy` with a probability tensor, or the two-term form). At evaluation the inputs are clean. Also beware stacking regularizers that all soften predictions: mixup plus strong label smoothing plus ensembling can leave a model **under**-confident (Wen et al., 2021).

> **Linear between examples:** Mixup does not claim that a blended image is a realistic image. It asks for a simple behaviour off the data: linear interpolation of the output along segments between training points. That inductive bias is what removes the sharp, confident transitions visible in the left panel of the figure.

> **Manifold Mixup:** Verma et al. (2019) apply the same blend to the hidden activations $h=g_k(x)$ at a randomly chosen layer $k$, then continue the forward pass: $\tilde h=\lambda g_k(x_a)+(1-\lambda)g_k(x_b)$, target $\tilde y$ as before. Mixing in feature space, where classes are closer to linearly separable, flattens the class representations and gives smoother boundaries at several depths, at the cost of a custom forward pass.

**Strengths**

- Two lines of code, almost free compute, one hyperparameter
- Smoother boundaries and better calibration (lower NLL and ECE)
- More robust to noisy labels and to small input perturbations
- Domain-agnostic: works on images, audio, tabular data and embeddings

**Limitations**

- Blended inputs are unrealistic, which can hurt tasks that rely on local detail (detection, localization)
- Slows fitting of the training set; benefits need longer schedules, and large $\alpha$ underfits
- Can make models under-confident when combined with other softening methods
- Needs soft-target losses; not directly applicable to tasks without mixable labels

**Question:** With λ = 0.7, a mixed image of a cat (a) and a dog (b) is classified with p(cat) = 0.5, p(dog) = 0.5. What is the mixup loss?

- A. 0.7·(−ln 0.5) + 0.3·(−ln 0.5) = ln 2 ≈ 0.693
- B. −ln 0.5 only for the cat label, ≈ 0.693, the dog label is ignored
- C. 0.7·0.5 + 0.3·0.5 = 0.5
- D. Zero, because the prediction is a blend

**Answer:** A. 0.7·(−ln 0.5) + 0.3·(−ln 0.5) = ln 2 ≈ 0.693 — The loss is $\lambda\,\mathrm{CE}(y_a)+(1-\lambda)\,\mathrm{CE}(y_b)=-0.7\ln0.5-0.3\ln0.5=\ln2$. It is the same as CE against the soft target $(0.7,0.3)$. Its minimum, the target's own entropy $H(0.7,0.3)=0.611$, is reached at $p=(0.7,0.3)$, not at $(0.5,0.5)$.

**Question:** You set α = 0.1 for mixup. What do most mixed training images look like?

- A. Mostly one original image with a faint trace of the other, because Beta(0.1, 0.1) puts most λ near 0 or 1
- B. Even 50/50 blends, because Beta(α, α) is centred at 0.5
- C. Pure noise, because small α means strong mixing
- D. Unchanged images: α = 0.1 disables mixing

**Answer:** A. Mostly one original image with a faint trace of the other, because Beta(0.1, 0.1) puts most λ near 0 or 1 — $\mathrm{Beta}(\alpha,\alpha)$ is symmetric with mean 0.5, but for $\alpha<1$ it is U-shaped. Most draws are close to 0 or 1, so mixing is mild. $\alpha=1$ is uniform and $\alpha>1$ concentrates near 0.5.

**Key points**

- $\tilde x=\lambda x_a+(1-\lambda)x_b$, $\tilde y=\lambda y_a+(1-\lambda)y_b$, $\lambda\sim\mathrm{Beta}(\alpha,\alpha)$, one $\lambda$ per batch (Zhang et al., 2018).
- Loss $=\lambda\,\mathrm{CE}(y_a)+(1-\lambda)\,\mathrm{CE}(y_b)$, equal to CE with the soft target; logit gradient $p-\tilde y$.
- It is vicinal risk minimization with a generic vicinity: segments between examples, labels interpolated linearly.
- Effect: smoother boundaries, lower confidence between classes, better calibration and robustness to noisy labels.
- Small $\alpha$ (0.1 to 0.4) mixes mildly; torchvision.transforms.v2.MixUp works on batches and returns soft labels. Manifold Mixup mixes hidden layers.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
