Definition
Label smoothing trains a -class softmax classifier with cross-entropy against a softened target: the correct class gets probability and every class, the correct one included, gets an extra . The one-hot label is mixed with the uniform distribution, with weight (typically 0.1).
With a one-hot target, cross-entropy is never zero for finite logits. The network can always lower its training loss by making the correct logit larger relative to the others, so on a separable training set the logits grow without bound and the predicted probabilities head to 1. The model ends up extremely confident on the training set, including on mislabeled or ambiguous examples. Label smoothing puts a ceiling on how confident it is rewarded for being.
Szegedy, Vanhoucke, Ioffe, Shlens and Wojna introduced it in "Rethinking the Inception architecture for computer vision" (2016), as label-smoothing regularization, with and on ImageNet, reporting about 0.2% absolute improvement in top-1 and top-5 error. The Transformer (Vaswani et al., 2017) used as well, noting that it hurts perplexity but improves accuracy and BLEU. Müller, Kornblith and Hinton (2019), "When does label smoothing help?", explained what it does to the representation, to calibration and to distillation.
Formula and gradient
- the labelled class
- smoothing strength, ; PyTorch's
label_smoothingis exactly this - the model's predicted distribution over the K classes
- cross-entropy to the uniform distribution : equals , so it pulls toward uniform
- the usual softmax cross-entropy gradient; it vanishes only when
- the optimal logit gap: finite for every ; infinite for one-hot targets
The gradient is the same as for any target distribution (see Categorical cross-entropy, which derives the gap formula). With one-hot, the correct class always gets a push until , which needs an infinite gap. With smoothing, the push reverses once exceeds : a model that is more confident than the target is pulled back. For the optimal gap is for , for and for .
At a glance
- Changes
- the target only; the model and the logits are untouched
- Typical ε
- 0.1 (Inception-v3, Transformer, many ImageNet recipes)
- Loss floor
- , the entropy of the smoothed target: 0.2911 nats for , about 1.01 for at
- Equivalent to
- cross-entropy plus up to a constant: a confidence penalty
- PyTorch
nn.CrossEntropyLoss(label_smoothing=0.1)- Keras
CategoricalCrossentropy(label_smoothing=0.1)(one-hot targets)
Worked example: three classes, ε = 0.1
Logits with the label on class 1. Then , so . The smoothed target is .
- 1One-hot loss: . Gradient : push the correct logit up, the others down.
- 2Smoothed loss: . This is what
nn.CrossEntropyLoss(label_smoothing=0.1)returns. - 3Smoothed gradient . Still pushing up, but 31% less, because is already closer to its target 0.933 than to 1.
- 4Split: . Only the KL part can be trained away; is the floor.
| gap | p₁ | one-hot loss | smoothed loss |
|---|---|---|---|
| 2 | 0.7870 | 0.2395 | 0.3729 |
| 3.332 = ln 28 | 0.9333 | 0.0690 | 0.2911 (minimum) |
| 5 | 0.9867 | 0.0134 | 0.3467 |
| 10 | 0.9999 | 0.0001 | 0.6668 |
The one-hot column falls forever, which is why unsmoothed training keeps inflating the logits. The smoothed column has a minimum at the finite gap 3.332 and rises beyond it: at gap 10 the mass on the wrong classes costs nats. The model is penalized for being more confident than .
1import torch, torch.nn as nn, torch.nn.functional as F23z = torch.tensor([[2.0, 0.0, 0.0]], requires_grad=True)4y = torch.tensor([0])56print(nn.CrossEntropyLoss()(z, y).item()) # 0.23957loss = nn.CrossEntropyLoss(label_smoothing=0.1)(z, y)8print(loss.item()) # 0.37299loss.backward()10print(z.grad) # [[-0.1463, 0.0732, 0.0732]] = p - q1112# By hand: q = (1 - eps) * onehot + eps / K13K, eps = 3, 0.114q = F.one_hot(y, K).float() * (1 - eps) + eps / K15print(-(q * F.log_softmax(z, dim=-1)).sum(-1).mean().item()) # 0.37291617# Training: smooth the loss, but report unsmoothed metrics on validation data18train_loss = nn.CrossEntropyLoss(label_smoothing=0.1)19val_loss = nn.CrossEntropyLoss()What it does to the network
Representations. Müller et al. (2019) projected penultimate-layer activations and found that label smoothing makes each class a tight, roughly equidistant cluster, because every wrong class must be held at the same logit distance from the correct one. Without smoothing, clusters are broader, and how close an example sits to another class carries information about similarity.
Calibration. Over-parameterized networks trained with one-hot cross-entropy tend to be over-confident (Guo et al., 2017). Label smoothing reduces that: Müller et al. found it lowers the expected calibration error by an amount comparable to post-hoc temperature scaling. The figure shows the mechanism in miniature: same accuracy, much less confidence, a lower ECE and a lower held-out NLL. It is not a calibration guarantee. A large makes the model under-confident, and the best for accuracy is not always the best for calibration.
Distillation. A teacher trained with label smoothing is often more accurate but a worse teacher: students distilled from it do worse than students distilled from a teacher trained without it (Müller et al., 2019). Distillation transfers the teacher's soft probabilities over the wrong classes ("this 3 looks a bit like an 8"), and label smoothing erases exactly that information by forcing all wrong classes to the same small value. Label noise is the other common motivation: smoothing limits how hard the model is pushed to fit a wrong label, and Lukasik et al. (2020) found it competitive with dedicated noise-robust losses.
Use when
- +One argument in the loss; no change to the model or the cost of training
- +Bounds the logits: a finite optimal gap instead of an incentive to grow forever
- +Usually a small accuracy gain on large classification tasks, and better calibration than one-hot training
- +Limits the damage from mislabeled examples
Avoid when
- −Hurts knowledge distillation: erases the similarity structure among wrong classes
- −Too much smoothing makes the model under-confident
- −Training loss has a floor H(q), which confuses loss-based comparisons and monitoring
- −Treats all wrong classes as equally likely, which is rarely true
Check yourself
With K = 10 classes and ε = 0.1, what probability does the smoothed target give the correct class?
Check yourself
A team trains a large teacher with label smoothing ε = 0.1; its accuracy improves. Students distilled from it are worse than students from the unsmoothed teacher. Why?
Remember
- ✓Target ; loss ; gradient .
- ✓The optimal logit gap is finite, (3.332 for , 9.105 for at ), so logits stay bounded.
- ✓The training loss has a floor ; monitor accuracy or unsmoothed cross-entropy.
- ✓It usually improves accuracy and calibration and tightens class clusters, but too large an ε makes the model under-confident.
- ✓It makes the network a worse distillation teacher, because it erases the similarity information among wrong classes.