Definition
Focal loss is cross-entropy scaled by a modulating factor that shrinks the loss of examples the model already gets right: . Here is the predicted probability of the true class, is the focusing parameter and is an optional class weight. With and it is exactly cross-entropy.
Focal loss was introduced by Lin, Goyal, Girshick, He and Dollár in Focal Loss for Dense Object Detection (ICCV 2017) to train RetinaNet. One-stage detectors classify every anchor box on a dense grid, about 100k per image, and almost all of them are easy background. Each easy anchor has a small cross-entropy, but there are so many that together they dominate the loss and its gradient. Two-stage detectors avoid this with a proposal stage and sampled mini-batches. Earlier one-stage methods used hard-example mining.
Focal loss fixes the imbalance inside the loss instead. Easy examples are discounted smoothly by how easy they are, so no sampling or mining is needed. With it, RetinaNet became the first one-stage detector to match the speed of earlier one-stage methods while beating the accuracy of the two-stage detectors of the time on COCO.
Formula and gradient
For a binary label and predicted probability , define if and if . Cross-entropy is then , and focal loss multiplies it:
- probability of the true class; for positives and for negatives
- modulating factor: near 1 for hard examples (), near 0 for easy ones ()
- focusing parameter. is cross-entropy; RetinaNet uses
- for positives, for negatives; RetinaNet uses
- for it reduces to the familiar ; negative means "raise the true-class logit"
The gradient follows from and the product rule. Differentiating adds the term . It has the same sign as , so for hard examples the focal gradient can actually be slightly larger than cross-entropy's ( vs at , ). For easy examples it is far smaller.
At a glance
- Task
- classification under heavy class imbalance, above all dense object detection
- Hyperparameters
- (focusing; 2 is the standard choice), (balance; 0.25 for RetinaNet)
- Down-weighting
- CE / FL : at and about at for
- Special cases
- : (α-balanced) cross-entropy
- Convex
- in for . For it is still monotone in , but slightly concave for very hard examples (around when )
- PyTorch
torchvision.ops.sigmoid_focal_loss(inputs, targets, alpha=0.25, gamma=2, reduction="none")
Intuition: why a power of (1 − pₜ)
Weighting by class frequency (the term alone) tells the loss which class is rare. It cannot tell an easy background anchor from a hard one. The modulating factor weights by difficulty, using the model's own confidence: a background anchor at is multiplied by when , while a confusing one at keeps half its loss ().
Because the weight is smooth and recomputed every step, an example that becomes easy fades out of the loss gradually, and one that becomes hard again gets its full weight back. Hard-example mining does something similar by keeping only the top-loss examples, but it makes a hard cut and needs extra bookkeeping.
Worked example
| pₜ | CE | FL | CE / FL | ∂CE/∂zₜ | ∂FL/∂zₜ |
|---|---|---|---|---|---|
| 0.1 (hard) | 2.303 | 1.865 | 1.23 | −0.900 | −1.102 |
| 0.5 | 0.693 | 0.173 | 4 | −0.500 | −0.298 |
| 0.9 (easy) | 0.105 | 0.00105 | 100 | −0.100 | −0.0029 |
| 0.99 (very easy) | 0.0101 | 1.0 × 10⁻⁶ | 10 000 | −0.0100 | −3.0 × 10⁻⁶ |
torchvision's sigmoid_focal_loss on four anchors (α = 0.25, γ = 2)
- 1Logits with targets . Then and . The last anchor is a confident false positive.
- 2Cross-entropy . Weights and .
- 3Focal losses , sum . The false positive is 97% of it, while under plain cross-entropy it would be 73%.
- 4Gradients w.r.t. the logits : . Almost all of the update goes into pushing the false positive down.
1import math, torch, torch.nn as nn, torch.nn.functional as F2from torchvision.ops import sigmoid_focal_loss34z = torch.tensor([2.0, -1.0, 0.0, 3.0], requires_grad=True) # raw logits5t = torch.tensor([1.0, 0.0, 1.0, 0.0]) # float 0/1 targets, same shape67loss = sigmoid_focal_loss(z, t, alpha=0.25, gamma=2.0, reduction="sum")8print(loss) # 2.13559loss.backward(); print(z.grad) # [-0.0012, 0.0394, -0.0746, 0.8451]1011def focal(z, t, alpha=0.25, gamma=2.0):12 ce = F.binary_cross_entropy_with_logits(z, t, reduction="none") # = -log p_t, stable13 p = torch.sigmoid(z)14 p_t = p * t + (1 - p) * (1 - t)15 alpha_t = alpha * t + (1 - alpha) * (1 - t)16 return alpha_t * (1 - p_t) ** gamma * ce1718# RetinaNet: sum over all anchors, divide by the number of foreground anchors19# loss = focal(logits, targets).sum() / max(1, num_foreground)2021# Prior init: every anchor starts at p = 0.01, so background is "easy" from step 122cls_head = nn.Conv2d(256, 9 * 80, 3, padding=1) # 9 anchors x 80 classes23nn.init.constant_(cls_head.bias, -math.log((1 - 0.01) / 0.01)) # -4.595Use when
- +Handles extreme foreground/background imbalance with no sampling or hard-example mining
- +Smooth, per-example weighting by difficulty; recovers cross-entropy, so it is a strict generalization
- +One-line change in code; built into torchvision
- +Often improves calibration on balanced classification too
Avoid when
- −Two extra hyperparameters, and , which interact and must be tuned together
- −For the outputs are no longer maximum-likelihood probabilities: the loss systematically under-pushes confident predictions
- −Up-weights hard examples, including mislabelled ones: label noise gets more influence
- −Needs care with normalization and bias initialization, or training is unstable
Check yourself
With γ = 2 and αₜ = 1, how much smaller is the focal loss than cross-entropy for an example with pₜ = 0.8?
Check yourself
A detector trained with focal loss (γ = 2) has its α changed from 0.25 to 0.75. Which anchors gain weight?
Remember
- ✓; is cross-entropy.
- ✓The factor down-weights easy examples by difficulty: at for .
- ✓Introduced for RetinaNet (Lin et al., 2017): one-stage detection with ~100k mostly-easy anchors per image.
- ✓Standard setting: , , sum over anchors divided by the foreground count, bias initialized for .
- ✓Use
torchvision.ops.sigmoid_focal_losson raw logits, with float 0/1 targets of the same shape.