Focal loss

Cross-entropy times (1 − pₜ)^γ: well-classified examples are down-weighted (100× at pₜ = 0.9 for γ = 2), so the thousands of easy negatives in dense detection stop drowning out the few hard examples.

10 min read756 wordscore0 viewsupdated 2026-10-11

Definition

Focal loss is cross-entropy scaled by a modulating factor that shrinks the loss of examples the model already gets right: FL(pt)=−αt(1−pt)γlog⁡pt\mathrm{FL}(p_t)=-\alpha_t(1-p_t)^\gamma\log p_t. Here ptp_t is the predicted probability of the true class, γ≥0\gamma\ge0 is the focusing parameter and αt\alpha_t is an optional class weight. With γ=0\gamma=0 and αt=1\alpha_t=1 it is exactly cross-entropy.

Focal loss was introduced by Lin, Goyal, Girshick, He and Dollár in Focal Loss for Dense Object Detection (ICCV 2017) to train RetinaNet. One-stage detectors classify every anchor box on a dense grid, about 100k per image, and almost all of them are easy background. Each easy anchor has a small cross-entropy, but there are so many that together they dominate the loss and its gradient. Two-stage detectors avoid this with a proposal stage and sampled mini-batches. Earlier one-stage methods used hard-example mining.

Focal loss fixes the imbalance inside the loss instead. Easy examples are discounted smoothly by how easy they are, so no sampling or mining is needed. With it, RetinaNet became the first one-stage detector to match the speed of earlier one-stage methods while beating the accuracy of the two-stage detectors of the time on COCO.

Formula and gradient

For a binary label y∈{0,1}y\in\{0,1\} and predicted probability p=σ(z)p=\sigma(z), define pt=pp_t=p if y=1y=1 and pt=1−pp_t=1-p if y=0y=0. Cross-entropy is then −log⁡pt-\log p_t, and focal loss multiplies it:

FL(pt)=−αt (1−pt)γ log⁡pt,∂ FL∂zt=αt (1−pt)γ (γ ptlog⁡pt−(1−pt))\mathrm{FL}(p_t)=-\alpha_t\,(1-p_t)^{\gamma}\,\log p_t,\qquad \frac{\partial\,\mathrm{FL}}{\partial z_t}=\alpha_t\,(1-p_t)^{\gamma}\,\big(\gamma\,p_t\log p_t-(1-p_t)\big)
pt=σ(zt)p_t=\sigma(z_t)
probability of the true class; zt=zz_t=z for positives and −z-z for negatives
(1−pt)γ(1-p_t)^\gamma
modulating factor: near 1 for hard examples (pt→0p_t\to0), near 0 for easy ones (pt→1p_t\to1)
γ\gamma
focusing parameter. γ=0\gamma=0 is cross-entropy; RetinaNet uses γ=2\gamma=2
αt\alpha_t
α\alpha for positives, 1−α1-\alpha for negatives; RetinaNet uses α=0.25\alpha=0.25
∂/∂zt\partial/\partial z_t
for γ=0\gamma=0 it reduces to the familiar αt(pt−1)\alpha_t(p_t-1); negative means "raise the true-class logit"

The gradient follows from ∂pt/∂zt=pt(1−pt)\partial p_t/\partial z_t=p_t(1-p_t) and the product rule. Differentiating (1−pt)γ(1-p_t)^\gamma adds the term γptlog⁡pt\gamma p_t\log p_t. It has the same sign as −(1−pt)-(1-p_t), so for hard examples the focal gradient can actually be slightly larger than cross-entropy's (−1.10-1.10 vs −0.90-0.90 at pt=0.1p_t=0.1, γ=2\gamma=2). For easy examples it is far smaller.

At a glance

Task
classification under heavy class imbalance, above all dense object detection
Hyperparameters
γ\gamma (focusing; 2 is the standard choice), α\alpha (balance; 0.25 for RetinaNet)
Down-weighting
CE / FL =1/(1−pt)γ=1/(1-p_t)^\gamma: 100×100\times at pt=0.9p_t=0.9 and about 1000×1000\times at pt=0.968p_t=0.968 for γ=2\gamma=2
Special cases
γ=0\gamma=0: (α-balanced) cross-entropy
Convex
in ztz_t for γ=0\gamma=0. For γ>0\gamma>0 it is still monotone in ptp_t, but slightly concave for very hard examples (around pt≈0.02p_t\approx0.02 when γ=2\gamma=2)
PyTorch
torchvision.ops.sigmoid_focal_loss(inputs, targets, alpha=0.25, gamma=2, reduction="none")
FL(pₜ) = −(1 − pₜ)^γ log pₜ (αₜ = 1)0123400.20.40.60.81pₜ (probability of the true class)well-classifiedγ = 0 · 0.5 · 1 · 2 · 5 (┄ = CE) · ━ γ = 2.0pₜ = 0.900 · (1 − pₜ)^γ = 0.0100CE 0.105 · FL 0.0011 · CE/FL 100×∂/∂zₜ: CE −0.1000 · FL −0.0029easy examples (pₜ → 1) are discounted by (1 − pₜ)^γone batch of a dense detector■ 10,000 easy background, pₜ 0.95–0.999■ 100 hard background, pₜ 0.3–0.8■ 20 foreground, pₜ 0.1–0.7cross-entropy · total 343.9focal γ=2.0, α=0.25 · total 16.43share of lossCEfocaleasy background75.4%1.5%hard background18.5%82.6%foreground6.0%15.8%αₜ = α on foreground, 1 − α on backgroundRetinaNet: γ = 2, α = 0.25; loss / #foregroundeasy background: 75.4% → 1.5% of the loss
left plot
fig.Left: FL against pₜ for γ = 0 (dashed, cross-entropy), 0.5, 1, 2 and 5, with the current γ in bold. Toggle to see the gradient magnitude w.r.t. the logit instead. At the default probe pₜ = 0.9, γ = 2, cross-entropy is 0.105 and focal loss 0.0011, a 100× reduction; the gradient falls from −0.100 to −0.0029. Right: a toy dense-detector batch of 10,000 easy background anchors, 100 hard background anchors and 20 foreground anchors. Under cross-entropy (total 343.9) the easy background supplies 75.4% of the loss. Under focal loss with γ = 2, α = 0.25 (total 16.43) it supplies 1.5%, and the hard background 82.6%. Raise α to 0.75 and the foreground share rises from 15.8% to 62.9%.

Intuition: why a power of (1 − pₜ)

Weighting by class frequency (the α\alpha term alone) tells the loss which class is rare. It cannot tell an easy background anchor from a hard one. The modulating factor weights by difficulty, using the model's own confidence: a background anchor at pt=0.99p_t=0.99 is multiplied by 10−410^{-4} when γ=2\gamma=2, while a confusing one at pt=0.3p_t=0.3 keeps half its loss (0.72=0.490.7^2=0.49).

Because the weight is smooth and recomputed every step, an example that becomes easy fades out of the loss gradually, and one that becomes hard again gets its full weight back. Hard-example mining does something similar by keeping only the top-loss examples, but it makes a hard cut and needs extra bookkeeping.

Worked example

pₜCEFLCE / FL∂CE/∂zₜ∂FL/∂zₜ
0.1 (hard)2.3031.8651.23−0.900−1.102
0.50.6930.1734−0.500−0.298
0.9 (easy)0.1050.00105100−0.100−0.0029
0.99 (very easy)0.01011.0 × 10⁻⁶10 000−0.0100−3.0 × 10⁻⁶
Cross-entropy vs focal loss (γ = 2, αₜ = 1) for one example, and the gradient w.r.t. the true-class logit (values from the figure's lib)

torchvision's sigmoid_focal_loss on four anchors (α = 0.25, γ = 2)

  1. 1Logits z=(2, −1, 0, 3)z=(2,\,-1,\,0,\,3) with targets (1, 0, 1, 0)(1,\,0,\,1,\,0). Then p=σ(z)=(0.881, 0.269, 0.5, 0.953)p=\sigma(z)=(0.881,\,0.269,\,0.5,\,0.953) and pt=(0.881, 0.731, 0.5, 0.047)p_t=(0.881,\,0.731,\,0.5,\,0.047). The last anchor is a confident false positive.
  2. 2Cross-entropy −log⁡pt=(0.127, 0.313, 0.693, 3.049)-\log p_t=(0.127,\,0.313,\,0.693,\,3.049). Weights αt=(0.25, 0.75, 0.25, 0.75)\alpha_t=(0.25,\,0.75,\,0.25,\,0.75) and (1−pt)2=(0.014, 0.072, 0.25, 0.907)(1-p_t)^2=(0.014,\,0.072,\,0.25,\,0.907).
  3. 3Focal losses (0.0005, 0.0170, 0.0433, 2.0747)(0.0005,\,0.0170,\,0.0433,\,2.0747), sum 2.1352.135. The false positive is 97% of it, while under plain cross-entropy it would be 73%.
  4. 4Gradients w.r.t. the logits zz: (−0.0012, +0.0394, −0.0746, +0.8451)(-0.0012,\,+0.0394,\,-0.0746,\,+0.8451). Almost all of the update goes into pushing the false positive down.
python
1import math, torch, torch.nn as nn, torch.nn.functional as F
2from torchvision.ops import sigmoid_focal_loss
3
4z = torch.tensor([2.0, -1.0, 0.0, 3.0], requires_grad=True) # raw logits
5t = torch.tensor([1.0, 0.0, 1.0, 0.0]) # float 0/1 targets, same shape
6
7loss = sigmoid_focal_loss(z, t, alpha=0.25, gamma=2.0, reduction="sum")
8print(loss) # 2.1355
9loss.backward(); print(z.grad) # [-0.0012, 0.0394, -0.0746, 0.8451]
10
11def focal(z, t, alpha=0.25, gamma=2.0):
12 ce = F.binary_cross_entropy_with_logits(z, t, reduction="none") # = -log p_t, stable
13 p = torch.sigmoid(z)
14 p_t = p * t + (1 - p) * (1 - t)
15 alpha_t = alpha * t + (1 - alpha) * (1 - t)
16 return alpha_t * (1 - p_t) ** gamma * ce
17
18# RetinaNet: sum over all anchors, divide by the number of foreground anchors
19# loss = focal(logits, targets).sum() / max(1, num_foreground)
20
21# Prior init: every anchor starts at p = 0.01, so background is "easy" from step 1
22cls_head = nn.Conv2d(256, 9 * 80, 3, padding=1) # 9 anchors x 80 classes
23nn.init.constant_(cls_head.bias, -math.log((1 - 0.01) / 0.01)) # -4.595
torchvision's built-in, a from-scratch version, and the RetinaNet prior initialization

Use when

  • +Handles extreme foreground/background imbalance with no sampling or hard-example mining
  • +Smooth, per-example weighting by difficulty; γ=0\gamma=0 recovers cross-entropy, so it is a strict generalization
  • +One-line change in code; built into torchvision
  • +Often improves calibration on balanced classification too

Avoid when

  • −Two extra hyperparameters, γ\gamma and α\alpha, which interact and must be tuned together
  • −For γ>0\gamma>0 the outputs are no longer maximum-likelihood probabilities: the loss systematically under-pushes confident predictions
  • −Up-weights hard examples, including mislabelled ones: label noise gets more influence
  • −Needs care with normalization and bias initialization, or training is unstable

Check yourself

With γ = 2 and αₜ = 1, how much smaller is the focal loss than cross-entropy for an example with pₜ = 0.8?

Check yourself

A detector trained with focal loss (γ = 2) has its α changed from 0.25 to 0.75. Which anchors gain weight?

Remember

  • ✓FL=−αt(1−pt)γlog⁡pt\mathrm{FL}=-\alpha_t(1-p_t)^\gamma\log p_t; γ=0\gamma=0 is cross-entropy.
  • ✓The factor (1−pt)γ(1-p_t)^\gamma down-weights easy examples by difficulty: 100×100\times at pt=0.9p_t=0.9 for γ=2\gamma=2.
  • ✓Introduced for RetinaNet (Lin et al., 2017): one-stage detection with ~100k mostly-easy anchors per image.
  • ✓Standard setting: γ=2\gamma=2, α=0.25\alpha=0.25, sum over anchors divided by the foreground count, bias initialized for π=0.01\pi=0.01.
  • ✓Use torchvision.ops.sigmoid_focal_loss on raw logits, with float 0/1 targets of the same shape.
← back to Deep Learning roadmap