Definition
CutMix is a data augmentation for images that replaces a rectangular region of image with the same region of image and sets the target to , where is the fraction of the image area taken from .
Two earlier ideas each fixed half of a problem. Cutout (DeVries and Taylor, 2017) erases a random square, forcing a classifier to use the whole object instead of one discriminative part. But the erased pixels carry no information, and the label still claims 100% of a class whose evidence may have been cut away. Mixup (Zhang et al., 2018) uses every pixel and mixes the labels, but a pixel-wise blend of two photos is not a locally plausible image, which confuses tasks that depend on local detail, such as localization.
Yun, Han, Oh, Chun, Choe and Yoo (2019), CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features (ICCV), combined them: cut like Cutout, but fill the hole with a patch of another training image, and mix the labels in proportion to area. On ImageNet with ResNet-50 they reported top-1 error 21.40%, against 23.68% for the baseline, 22.93% for Cutout and 22.58% for Mixup, and better weakly supervised localization and detection transfer. CutMix and Mixup are now used together in most image-classification recipes, from DeiT (Touvron et al., 2021) to the torchvision reference models.
Formula
- binary mask: 0 inside the box, 1 elsewhere (shared across channels)
- element-wise product
- the box size after clipping to the image
- the area fraction kept from ; also its label weight
Sampling the box (Yun et al., 2019; torchvision v2.CutMix)
- 1Draw ; the paper and torchvision default to , i.e. .
- 2Box size , : same aspect ratio as the image and area .
- 3Box centre uniform over the image. Clip the box to the image borders.
- 4Clipping shrinks the box, so recompute from the actual box. The label follows the pixels, not the sampled number.
- 5Paste the box from the partner image (a shuffled or rolled copy of the batch) and mix the labels with the corrected .
At a glance
- Kind
- label-aware, region-level data augmentation for images (also used on spectrograms)
- Hyperparameter
- (default 1), plus the probability of applying it
- Label
- area ratio after clipping; soft-target cross-entropy
- Pixels
- all real, from one of the two images; only the box edges are artificial
- Cost
- one slice copy per batch
- Compared with
- Cutout (box erased, label unchanged), Mixup (all pixels blended, label blended)
Why it works: local realism and honest labels
Each method changes the input and the label. The difference is whether the two stay consistent and whether the input still looks like natural images locally.
| method | input | untouched real pixels | pixels showing A | label (A, B) |
|---|---|---|---|---|
| Cutout | box filled with the mean colour | 75% | 75% | (1, 0) |
| Mixup | everywhere | 0% | blended | (0.75, 0.25) |
| CutMix | box replaced by B's pixels | 100% | 75% | (0.75, 0.25) |
Local realism. Apart from the edges of the box, every 3×3 neighbourhood in a CutMix image comes from a real image, so convolutional filters see the statistics they will see at test time. Honest labels. The label weight equals the share of visible evidence. No wasted pixels. The region Cutout spends on a grey square now teaches the second class. And because the network must recognize a class from a partial view and from a pasted patch in an unusual place, it learns to spread its attention over the whole object, which is why CutMix improved class-activation-map localization in the paper.
Worked example: a clipped box on a 32×32 image
λ = 0.75 drawn, CIFAR-sized image, checked in Node with the torchvision box rule
- 1Half sizes , so the box is .
- 2Centre : box , area 256, . Nothing changes.
- 3Centre : the box is clipped to in both axes, area , so .
- 4Label . If the model predicts , , the loss is .
- 5Using the uncorrected would claim 25% of class when only 14% of the pixels show it.
1import math, torch, torch.nn.functional as F2from torchvision.transforms import v234def cutmix(x, y, num_classes, alpha=1.0):5 """x: (N, C, H, W), y: (N,) int labels -> mixed images, soft labels."""6 N, _, H, W = x.shape7 lam = torch.distributions.Beta(alpha, alpha).sample().item()8 cx, cy = torch.randint(W, ()).item(), torch.randint(H, ()).item()9 rw, rh = int(0.5 * math.sqrt(1 - lam) * W), int(0.5 * math.sqrt(1 - lam) * H)10 x1, x2 = max(cx - rw, 0), min(cx + rw, W)11 y1, y2 = max(cy - rh, 0), min(cy + rh, H)12 idx = torch.randperm(N)13 x = x.clone()14 x[:, :, y1:y2, x1:x2] = x[idx, :, y1:y2, x1:x2]15 lam = 1 - (x2 - x1) * (y2 - y1) / (W * H) # area actually pasted16 y1h = F.one_hot(y, num_classes).float()17 return x, lam * y1h + (1 - lam) * y1h[idx]1819# torchvision: batch-level transforms (apply after the DataLoader or in collate_fn)20NUM_CLASSES = 100021cutmix_t = v2.CutMix(alpha=1.0, num_classes=NUM_CLASSES)22mixup_t = v2.MixUp(alpha=0.2, num_classes=NUM_CLASSES)23cutmix_or_mixup = v2.RandomChoice([cutmix_t, mixup_t]) # one of the two per batch2425images = torch.rand(16, 3, 224, 224)26labels = torch.randint(0, NUM_CLASSES, (16,))27images, soft = cutmix_or_mixup(images, labels) # soft: (16, 1000)28# loss = F.cross_entropy(model(images), soft)2930# timm's equivalent (DeiT-style recipe): one object, switch between the two per batch31# from timm.data import Mixup32# mix = Mixup(mixup_alpha=0.8, cutmix_alpha=1.0, switch_prob=0.5, label_smoothing=0.1, num_classes=1000)33# loss with timm.loss.SoftTargetCrossEntropy()Use when
- +Every pixel is real and informative; no grey squares as in Cutout
- +Label weights match the visible area, so targets stay honest
- +Improves classification and localization; locally realistic inputs suit CNNs
- +One hyperparameter and almost no compute
Avoid when
- −Box edges create artificial boundaries the model may learn to detect
- −The pasted patch may contain only background, so the label can still over-state what is visible
- −Image-specific: needs a spatial layout to cut
- −Needs soft-label losses and longer schedules to pay off
Check yourself
On a 224×224 image, CutMix pastes a 112×112 box from the partner image that lies fully inside the image. What is the label weight λ of the original image?
Check yourself
Which statement contrasts Cutout, Mixup and CutMix correctly?
Remember
- ✓: paste a box from into ; with the area kept (Yun et al., 2019).
- ✓Box side times the image side, centre uniform, then clip and recompute from the actual area.
- ✓Fixes Cutout's wasted pixels and dishonest label, and Mixup's unrealistic blended pixels.
- ✓ResNet-50 on ImageNet: 21.40% top-1 error versus 23.68% baseline, 22.93% Cutout and 22.58% Mixup in the paper; better localization too.
- ✓torchvision.transforms.v2.CutMix on batches; v2.RandomChoice([CutMix, MixUp]) or timm's Mixup(switch_prob=0.5) alternate the two.