Label smoothing

Train against q = (1 − ε)·onehot + ε/K instead of the one-hot label. The optimal logit gap becomes finite, logits stay bounded and the model is less over-confident, which usually helps accuracy and calibration but makes it a worse teacher for distillation.

10 min read906 wordscore0 viewsupdated 2026-10-11

Definition

Label smoothing trains a KK-class softmax classifier with cross-entropy against a softened target: the correct class gets probability 1−ε+ε/K1-\varepsilon+\varepsilon/K and every class, the correct one included, gets an extra ε/K\varepsilon/K. The one-hot label is mixed with the uniform distribution, with weight ε\varepsilon (typically 0.1).

With a one-hot target, cross-entropy −ln⁡pc-\ln p_c is never zero for finite logits. The network can always lower its training loss by making the correct logit larger relative to the others, so on a separable training set the logits grow without bound and the predicted probabilities head to 1. The model ends up extremely confident on the training set, including on mislabeled or ambiguous examples. Label smoothing puts a ceiling on how confident it is rewarded for being.

Szegedy, Vanhoucke, Ioffe, Shlens and Wojna introduced it in "Rethinking the Inception architecture for computer vision" (2016), as label-smoothing regularization, with K=1000K=1000 and ε=0.1\varepsilon=0.1 on ImageNet, reporting about 0.2% absolute improvement in top-1 and top-5 error. The Transformer (Vaswani et al., 2017) used ε=0.1\varepsilon=0.1 as well, noting that it hurts perplexity but improves accuracy and BLEU. Müller, Kornblith and Hinton (2019), "When does label smoothing help?", explained what it does to the representation, to calibration and to distillation.

Formula and gradient

qk=(1−ε) 1[k=c]+εK,ℓ=−∑k=1Kqkln⁡pk=(1−ε) (−ln⁡pc)+ε⋅1K∑k=1K(−ln⁡pk)q_k=(1-\varepsilon)\,\mathbb 1[k=c]+\frac{\varepsilon}{K},\qquad \ell=-\sum_{k=1}^K q_k\ln p_k=(1-\varepsilon)\,(-\ln p_c)+\varepsilon\cdot\frac1K\sum_{k=1}^K(-\ln p_k)
cc
the labelled class
ε\varepsilon
smoothing strength, 0≤ε<10\le\varepsilon<1; PyTorch's label_smoothing is exactly this ε\varepsilon
p=softmax(z)p=\mathrm{softmax}(z)
the model's predicted distribution over the K classes
1K∑k(−ln⁡pk)\tfrac1K\sum_k(-\ln p_k)
cross-entropy to the uniform distribution uu: equals KL(u ∥ p)+ln⁡K\mathrm{KL}(u\,\|\,p)+\ln K, so it pulls pp toward uniform
∂ℓ∂zk=pk−qk⟹at the optimum p=q:zc∗−zj∗=ln⁡1−ε+ε/Kε/K  (j≠c)\frac{\partial\ell}{\partial z_k}=p_k-q_k\qquad\Longrightarrow\qquad \text{at the optimum } p=q:\quad z_c^*-z_j^*=\ln\frac{1-\varepsilon+\varepsilon/K}{\varepsilon/K}\ \ (j\ne c)
pk−qkp_k-q_k
the usual softmax cross-entropy gradient; it vanishes only when p=qp=q
zc∗−zj∗z_c^*-z_j^*
the optimal logit gap: finite for every ε>0\varepsilon>0; infinite for one-hot targets

The gradient is the same p−qp-q as for any target distribution (see Categorical cross-entropy, which derives the gap formula). With qq one-hot, the correct class always gets a push pc−1<0p_c-1<0 until pc=1p_c=1, which needs an infinite gap. With smoothing, the push reverses once pcp_c exceeds 1−ε+ε/K1-\varepsilon+\varepsilon/K: a model that is more confident than the target is pulled back. For ε=0.1\varepsilon=0.1 the optimal gap is ln⁡28=3.332\ln 28=3.332 for K=3K=3, ln⁡37=3.611\ln 37=3.611 for K=4K=4 and 9.1059.105 for K=1000K=1000.

At a glance

Changes
the target only; the model and the logits are untouched
Typical ε
0.1 (Inception-v3, Transformer, many ImageNet recipes)
Loss floor
H(q)H(q), the entropy of the smoothed target: 0.2911 nats for K=3K=3, about 1.01 for K=1000K=1000 at ε=0.1\varepsilon=0.1
Equivalent to
cross-entropy plus ε⋅KL(u ∥ p)\varepsilon\cdot\mathrm{KL}(u\,\|\,p) up to a constant: a confidence penalty
PyTorch
nn.CrossEntropyLoss(label_smoothing=0.1)
Keras
CategoricalCrossentropy(label_smoothing=0.1) (one-hot targets)
60 train pts · shade = max p (dark = 1/3)reliability · 600 test pts0confidence 1ε=0.10ε=0ECE0.0960.168mean conf0.8230.904accuracy0.7350.743test NLL0.7191.159train loss0.4130.130floor H(q)0.2910train logit gap · step 15000510optimum ln((1−ε+ε/K)/(ε/K)) = 3.33201500 GD steps█ accuracy per bin ━ mean confidence per bin ━ ε = 0.10 ━ ε = 0
full-batch GD, lr 4, K = 3
fig.Three overlapping classes, 60 training points, a softmax head on 49 Gaussian bump features, 1500 full-batch GD steps; reliability diagram on 600 held-out points (bars: accuracy per confidence bin; red ticks: mean confidence). Without smoothing (grey) the mean training logit gap is still climbing at step 1500 (9.36), held-out confidence is 0.904 against accuracy 0.743, ECE 0.168 and held-out NLL 1.159. With ε = 0.1 the gap levels off below the dashed optimum 3.332 (2.93 at step 1500), confidence drops to 0.823, ECE to 0.096 and NLL to 0.719, while accuracy is about the same (0.735). ε = 0.2 lowers ECE further to 0.067. Scrub back to step 100 with ε = 0.3: confidence 0.682 is now below accuracy 0.773, so too much smoothing makes the model under-confident.

Worked example: three classes, ε = 0.1

Logits z=(2, 0, 0)z=(2,\,0,\,0) with the label on class 1. Then ln⁡∑jezj=ln⁡(e2+2)=2.2395\ln\sum_j e^{z_j}=\ln(e^2+2)=2.2395, so p=(0.7870, 0.1065, 0.1065)p=(0.7870,\,0.1065,\,0.1065). The smoothed target is q=(0.9333, 0.0333, 0.0333)q=(0.9333,\,0.0333,\,0.0333).

  1. 1One-hot loss: −ln⁡0.7870=0.2395-\ln 0.7870=0.2395. Gradient p−onehot=(−0.2130, +0.1065, +0.1065)p-\text{onehot}=(-0.2130,\,+0.1065,\,+0.1065): push the correct logit up, the others down.
  2. 2Smoothed loss: (1−ε)(0.2395)+ε⋅13(0.2395+2.2395+2.2395)=0.9(0.2395)+0.1(1.5729)=0.3729(1-\varepsilon)(0.2395)+\varepsilon\cdot\tfrac13(0.2395+2.2395+2.2395)=0.9(0.2395)+0.1(1.5729)=0.3729. This is what nn.CrossEntropyLoss(label_smoothing=0.1) returns.
  3. 3Smoothed gradient p−q=(−0.1463, +0.0732, +0.0732)p-q=(-0.1463,\,+0.0732,\,+0.0732). Still pushing up, but 31% less, because p1=0.787p_1=0.787 is already closer to its target 0.933 than to 1.
  4. 4Split: ℓ=H(q)+KL(q ∥ p)=0.2911+0.0817\ell=H(q)+\mathrm{KL}(q\,\|\,p)=0.2911+0.0817. Only the KL part can be trained away; H(q)=0.2911H(q)=0.2911 is the floor.
gapp₁one-hot losssmoothed loss
20.78700.23950.3729
3.332 = ln 280.93330.06900.2911 (minimum)
50.98670.01340.3467
100.99990.00010.6668
Loss as a function of the gap z₁ − z₂ = z₁ − z₃ (K = 3, ε = 0.1; checked in Node)

The one-hot column falls forever, which is why unsmoothed training keeps inflating the logits. The smoothed column has a minimum at the finite gap 3.332 and rises beyond it: at gap 10 the ε/K\varepsilon/K mass on the wrong classes costs 0.13⋅2⋅10≈0.67\tfrac{0.1}{3}\cdot2\cdot10\approx0.67 nats. The model is penalized for being more confident than 1−ε+ε/K1-\varepsilon+\varepsilon/K.

python
1import torch, torch.nn as nn, torch.nn.functional as F
2
3z = torch.tensor([[2.0, 0.0, 0.0]], requires_grad=True)
4y = torch.tensor([0])
5
6print(nn.CrossEntropyLoss()(z, y).item()) # 0.2395
7loss = nn.CrossEntropyLoss(label_smoothing=0.1)(z, y)
8print(loss.item()) # 0.3729
9loss.backward()
10print(z.grad) # [[-0.1463, 0.0732, 0.0732]] = p - q
11
12# By hand: q = (1 - eps) * onehot + eps / K
13K, eps = 3, 0.1
14q = F.one_hot(y, K).float() * (1 - eps) + eps / K
15print(-(q * F.log_softmax(z, dim=-1)).sum(-1).mean().item()) # 0.3729
16
17# Training: smooth the loss, but report unsmoothed metrics on validation data
18train_loss = nn.CrossEntropyLoss(label_smoothing=0.1)
19val_loss = nn.CrossEntropyLoss()
Label smoothing in PyTorch, and the same loss by hand

What it does to the network

Representations. Müller et al. (2019) projected penultimate-layer activations and found that label smoothing makes each class a tight, roughly equidistant cluster, because every wrong class must be held at the same logit distance from the correct one. Without smoothing, clusters are broader, and how close an example sits to another class carries information about similarity.

Calibration. Over-parameterized networks trained with one-hot cross-entropy tend to be over-confident (Guo et al., 2017). Label smoothing reduces that: Müller et al. found it lowers the expected calibration error by an amount comparable to post-hoc temperature scaling. The figure shows the mechanism in miniature: same accuracy, much less confidence, a lower ECE and a lower held-out NLL. It is not a calibration guarantee. A large ε\varepsilon makes the model under-confident, and the best ε\varepsilon for accuracy is not always the best for calibration.

Distillation. A teacher trained with label smoothing is often more accurate but a worse teacher: students distilled from it do worse than students distilled from a teacher trained without it (Müller et al., 2019). Distillation transfers the teacher's soft probabilities over the wrong classes ("this 3 looks a bit like an 8"), and label smoothing erases exactly that information by forcing all wrong classes to the same small value. Label noise is the other common motivation: smoothing limits how hard the model is pushed to fit a wrong label, and Lukasik et al. (2020) found it competitive with dedicated noise-robust losses.

Use when

  • +One argument in the loss; no change to the model or the cost of training
  • +Bounds the logits: a finite optimal gap instead of an incentive to grow forever
  • +Usually a small accuracy gain on large classification tasks, and better calibration than one-hot training
  • +Limits the damage from mislabeled examples

Avoid when

  • −Hurts knowledge distillation: erases the similarity structure among wrong classes
  • −Too much smoothing makes the model under-confident
  • −Training loss has a floor H(q), which confuses loss-based comparisons and monitoring
  • −Treats all wrong classes as equally likely, which is rarely true

Check yourself

With K = 10 classes and ε = 0.1, what probability does the smoothed target give the correct class?

Check yourself

A team trains a large teacher with label smoothing ε = 0.1; its accuracy improves. Students distilled from it are worse than students from the unsmoothed teacher. Why?

Remember

  • ✓Target q=(1−ε) onehot+ε/Kq=(1-\varepsilon)\,\text{onehot}+\varepsilon/K; loss (1−ε)(−ln⁡pc)+ε⋅(−ln⁡pk)‾(1-\varepsilon)(-\ln p_c)+\varepsilon\cdot\overline{(-\ln p_k)}; gradient p−qp-q.
  • ✓The optimal logit gap is finite, ln⁡1−ε+ε/Kε/K\ln\frac{1-\varepsilon+\varepsilon/K}{\varepsilon/K} (3.332 for K=3K=3, 9.105 for K=1000K=1000 at ε=0.1\varepsilon=0.1), so logits stay bounded.
  • ✓The training loss has a floor H(q)H(q); monitor accuracy or unsmoothed cross-entropy.
  • ✓It usually improves accuracy and calibration and tightens class clusters, but too large an ε makes the model under-confident.
  • ✓It makes the network a worse distillation teacher, because it erases the similarity information among wrong classes.
← back to Deep Learning roadmap