# Focal loss

> Cross-entropy times (1 − pₜ)^γ: well-classified examples are down-weighted (100× at pₜ = 0.9 for γ = 2), so the thousands of easy negatives in dense detection stop drowning out the few hard examples.

Source: https://www.mlfoundry.in/learn/focal-loss  
Section: Deep Learning › Loss Functions  
Level: core · 10 min read · updated 2026-10-11

**Focal loss:** is cross-entropy scaled by a **modulating factor** that shrinks the loss of examples the model already gets right: $\mathrm{FL}(p_t)=-\alpha_t(1-p_t)^\gamma\log p_t$. Here $p_t$ is the predicted probability of the true class, $\gamma\ge0$ is the **focusing parameter** and $\alpha_t$ is an optional class weight. With $\gamma=0$ and $\alpha_t=1$ it is exactly cross-entropy.

Focal loss was introduced by Lin, Goyal, Girshick, He and Dollár in *Focal Loss for Dense Object Detection* (ICCV 2017) to train **RetinaNet**. One-stage detectors classify every anchor box on a dense grid, about 100k per image, and almost all of them are easy background. Each easy anchor has a small cross-entropy, but there are so many that together they dominate the loss and its gradient. Two-stage detectors avoid this with a proposal stage and sampled mini-batches. Earlier one-stage methods used hard-example mining.

Focal loss fixes the imbalance inside the loss instead. Easy examples are discounted smoothly by how easy they are, so no sampling or mining is needed. With it, RetinaNet became the first one-stage detector to match the speed of earlier one-stage methods while beating the accuracy of the two-stage detectors of the time on COCO.

## Formula and gradient

For a binary label $y\in\{0,1\}$ and predicted probability $p=\sigma(z)$, define $p_t=p$ if $y=1$ and $p_t=1-p$ if $y=0$. Cross-entropy is then $-\log p_t$, and focal loss multiplies it:

$$
\mathrm{FL}(p_t)=-\alpha_t\,(1-p_t)^{\gamma}\,\log p_t,\qquad \frac{\partial\,\mathrm{FL}}{\partial z_t}=\alpha_t\,(1-p_t)^{\gamma}\,\big(\gamma\,p_t\log p_t-(1-p_t)\big)
$$

- $p_t=\sigma(z_t)$: probability of the true class; $z_t=z$ for positives and $-z$ for negatives
- $(1-p_t)^\gamma$: modulating factor: near 1 for hard examples ($p_t\to0$), near 0 for easy ones ($p_t\to1$)
- $\gamma$: focusing parameter. $\gamma=0$ is cross-entropy; RetinaNet uses $\gamma=2$
- $\alpha_t$: $\alpha$ for positives, $1-\alpha$ for negatives; RetinaNet uses $\alpha=0.25$
- $\partial/\partial z_t$: for $\gamma=0$ it reduces to the familiar $\alpha_t(p_t-1)$; negative means "raise the true-class logit"

The gradient follows from $\partial p_t/\partial z_t=p_t(1-p_t)$ and the product rule. Differentiating $(1-p_t)^\gamma$ adds the term $\gamma p_t\log p_t$. It has the same sign as $-(1-p_t)$, so for hard examples the focal gradient can actually be slightly **larger** than cross-entropy's ($-1.10$ vs $-0.90$ at $p_t=0.1$, $\gamma=2$). For easy examples it is far smaller.

**At a glance**

- **Task:** classification under heavy class imbalance, above all dense object detection
- **Hyperparameters:** $\gamma$ (focusing; 2 is the standard choice), $\alpha$ (balance; 0.25 for RetinaNet)
- **Down-weighting:** CE / FL $=1/(1-p_t)^\gamma$: $100\times$ at $p_t=0.9$ and about $1000\times$ at $p_t=0.968$ for $\gamma=2$
- **Special cases:** $\gamma=0$: (α-balanced) cross-entropy
- **Convex:** in $z_t$ for $\gamma=0$. For $\gamma>0$ it is still monotone in $p_t$, but slightly concave for very hard examples (around $p_t\approx0.02$ when $\gamma=2$)
- **PyTorch:** `torchvision.ops.sigmoid_focal_loss(inputs, targets, alpha=0.25, gamma=2, reduction="none")`

> Figure: Left: FL against pₜ for γ = 0 (dashed, cross-entropy), 0.5, 1, 2 and 5, with the current γ in bold. Toggle to see the gradient magnitude w.r.t. the logit instead. At the default probe pₜ = 0.9, γ = 2, cross-entropy is 0.105 and focal loss 0.0011, a 100× reduction; the gradient falls from −0.100 to −0.0029. Right: a toy dense-detector batch of 10,000 easy background anchors, 100 hard background anchors and 20 foreground anchors. Under cross-entropy (total 343.9) the easy background supplies 75.4% of the loss. Under focal loss with γ = 2, α = 0.25 (total 16.43) it supplies 1.5%, and the hard background 82.6%. Raise α to 0.75 and the foreground share rises from 15.8% to 62.9%.

## Intuition: why a power of (1 − pₜ)

Weighting by class frequency (the $\alpha$ term alone) tells the loss *which class* is rare. It cannot tell an easy background anchor from a hard one. The modulating factor weights by **difficulty**, using the model's own confidence: a background anchor at $p_t=0.99$ is multiplied by $10^{-4}$ when $\gamma=2$, while a confusing one at $p_t=0.3$ keeps half its loss ($0.7^2=0.49$).

Because the weight is smooth and recomputed every step, an example that becomes easy fades out of the loss gradually, and one that becomes hard again gets its full weight back. Hard-example mining does something similar by keeping only the top-loss examples, but it makes a hard cut and needs extra bookkeeping.

> **α and γ interact:** You might expect the rare foreground class to need a large $\alpha$. Lin et al. found the opposite: with $\gamma=2$ the best value was $\alpha=0.25$, which *down*-weights positives. Once $\gamma$ has removed the easy negatives, the remaining negatives are few and hard, and the positives no longer need a boost. The two parameters must be tuned together. The figure shows this: with γ = 2 the foreground already supplies 36.1% of the loss even with α-balancing switched off.

## Worked example

Cross-entropy vs focal loss (γ = 2, αₜ = 1) for one example, and the gradient w.r.t. the true-class logit (values from the figure's lib)

| pₜ | CE | FL | CE / FL | ∂CE/∂zₜ | ∂FL/∂zₜ |
| --- | --- | --- | --- | --- | --- |
| 0.1 (hard) | 2.303 | 1.865 | 1.23 | −0.900 | −1.102 |
| 0.5 | 0.693 | 0.173 | 4 | −0.500 | −0.298 |
| 0.9 (easy) | 0.105 | 0.00105 | 100 | −0.100 | −0.0029 |
| 0.99 (very easy) | 0.0101 | 1.0 × 10⁻⁶ | 10 000 | −0.0100 | −3.0 × 10⁻⁶ |

**torchvision's sigmoid_focal_loss on four anchors (α = 0.25, γ = 2)**

1. Logits $z=(2,\,-1,\,0,\,3)$ with targets $(1,\,0,\,1,\,0)$. Then $p=\sigma(z)=(0.881,\,0.269,\,0.5,\,0.953)$ and $p_t=(0.881,\,0.731,\,0.5,\,0.047)$. The last anchor is a confident false positive.
2. Cross-entropy $-\log p_t=(0.127,\,0.313,\,0.693,\,3.049)$. Weights $\alpha_t=(0.25,\,0.75,\,0.25,\,0.75)$ and $(1-p_t)^2=(0.014,\,0.072,\,0.25,\,0.907)$.
3. Focal losses $(0.0005,\,0.0170,\,0.0433,\,2.0747)$, sum $2.135$. The false positive is 97% of it, while under plain cross-entropy it would be 73%.
4. Gradients w.r.t. the logits $z$: $(-0.0012,\,+0.0394,\,-0.0746,\,+0.8451)$. Almost all of the update goes into pushing the false positive down.

```python
import math, torch, torch.nn as nn, torch.nn.functional as F
from torchvision.ops import sigmoid_focal_loss

z = torch.tensor([2.0, -1.0, 0.0, 3.0], requires_grad=True)   # raw logits
t = torch.tensor([1.0, 0.0, 1.0, 0.0])                        # float 0/1 targets, same shape

loss = sigmoid_focal_loss(z, t, alpha=0.25, gamma=2.0, reduction="sum")
print(loss)                     # 2.1355
loss.backward(); print(z.grad)  # [-0.0012, 0.0394, -0.0746, 0.8451]

def focal(z, t, alpha=0.25, gamma=2.0):
    ce = F.binary_cross_entropy_with_logits(z, t, reduction="none")   # = -log p_t, stable
    p = torch.sigmoid(z)
    p_t = p * t + (1 - p) * (1 - t)
    alpha_t = alpha * t + (1 - alpha) * (1 - t)
    return alpha_t * (1 - p_t) ** gamma * ce

# RetinaNet: sum over all anchors, divide by the number of foreground anchors
# loss = focal(logits, targets).sum() / max(1, num_foreground)

# Prior init: every anchor starts at p = 0.01, so background is "easy" from step 1
cls_head = nn.Conv2d(256, 9 * 80, 3, padding=1)   # 9 anchors x 80 classes
nn.init.constant_(cls_head.bias, -math.log((1 - 0.01) / 0.01))   # -4.595
```

torchvision's built-in, a from-scratch version, and the RetinaNet prior initialization

> **Normalize by the foreground count, not the anchor count:** RetinaNet sums the focal loss over all ~100k anchors and divides by the number of anchors assigned to a ground-truth box. Taking the mean over all anchors divides by a number that is huge and nearly constant, which shrinks the effective learning rate by orders of magnitude. torchvision's default `reduction="none"` leaves this choice to you.

> **Initialize the bias for the rare class:** With a zero bias every anchor starts at $p=0.5$, so each of 100k background anchors costs $\ln2$ and the start of training is unstable. Lin et al. set the final classification bias to $-\log((1-\pi)/\pi)$ with $\pi=0.01$, i.e. $-4.595$. Then a background anchor starts at $p_t=0.99$ and focal loss with $\gamma=2,\alpha=0.25$ charges 100k of them only 0.075 in total. This trick helps plain cross-entropy too.

> **Beyond detection:** The multiclass version applies the same factor to the softmax probability of the true class, $-\alpha_t(1-p_t)^\gamma\log p_t$. Focal loss and variants are used in segmentation and in keypoint heatmaps (CenterNet's penalty-reduced focal loss). Mukhoti et al. (2020) showed that focal loss also gives better-calibrated classifiers than cross-entropy, because it stops pushing already-correct predictions toward probability 1.

**Strengths**

- Handles extreme foreground/background imbalance with no sampling or hard-example mining
- Smooth, per-example weighting by difficulty; $\gamma=0$ recovers cross-entropy, so it is a strict generalization
- One-line change in code; built into torchvision
- Often improves calibration on balanced classification too

**Limitations**

- Two extra hyperparameters, $\gamma$ and $\alpha$, which interact and must be tuned together
- For $\gamma>0$ the outputs are no longer maximum-likelihood probabilities: the loss systematically under-pushes confident predictions
- Up-weights hard examples, including mislabelled ones: label noise gets more influence
- Needs care with normalization and bias initialization, or training is unstable

**Question:** With γ = 2 and αₜ = 1, how much smaller is the focal loss than cross-entropy for an example with pₜ = 0.8?

- A. 25 times
- B. 5 times
- C. 4 times
- D. 1.25 times

**Answer:** A. 25 times — The ratio is $1/(1-p_t)^\gamma=1/0.2^2=25$. The log term is the same in both losses and cancels, so only the modulating factor matters.

**Question:** A detector trained with focal loss (γ = 2) has its α changed from 0.25 to 0.75. Which anchors gain weight?

- A. Foreground anchors: αₜ = α for positives, so they go from 0.25 to 0.75
- B. Background anchors, since they are the majority
- C. Only the easy anchors
- D. None: α only rescales the total loss

**Answer:** A. Foreground anchors: αₜ = α for positives, so they go from 0.25 to 0.75 — $\alpha_t=\alpha$ for positives and $1-\alpha$ for negatives, so raising $\alpha$ triples the positive weight and cuts the negative weight from $0.75$ to $0.25$. In the figure's toy batch this moves the foreground share of the loss from 15.8% to 62.9%.

**Key points**

- $\mathrm{FL}=-\alpha_t(1-p_t)^\gamma\log p_t$; $\gamma=0$ is cross-entropy.
- The factor $(1-p_t)^\gamma$ down-weights easy examples by difficulty: $100\times$ at $p_t=0.9$ for $\gamma=2$.
- Introduced for RetinaNet (Lin et al., 2017): one-stage detection with ~100k mostly-easy anchors per image.
- Standard setting: $\gamma=2$, $\alpha=0.25$, sum over anchors divided by the foreground count, bias initialized for $\pi=0.01$.
- Use `torchvision.ops.sigmoid_focal_loss` on raw logits, with float 0/1 targets of the same shape.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
