# RetinaNet

> One-stage detector (Lin et al., 2017): ResNet-FPN levels P3–P7, 9 anchors per location, shared class and box subnets, focal loss over all ~100k anchors and a π = 0.01 bias prior; 39.1 COCO AP, above two-stage detectors.

Source: https://www.mlfoundry.in/learn/retinanet  
Section: Deep Learning › Deep Learning for Computer Vision  
Level: core · 7 min read · updated 2026-10-15

**RetinaNet:** is the one-stage object detector of Lin, Goyal, Girshick, He and Dollár, "Focal Loss for Dense Object Detection" (ICCV 2017). It is a deliberately plain architecture: a ResNet backbone, a feature pyramid network (FPN) and two small convolutional subnets. The new part is the training loss. **Focal loss** lets a dense detector train on every anchor without being swamped by easy background, and with it a one-stage detector matched two-stage accuracy for the first time.

Input: an image, shorter side 800 px in the main results. Output: boxes with class scores. Two-stage detectors (**Faster R-CNN**) were the most accurate. Their first stage cuts about 100k candidate locations down to 1–2k proposals, and the second stage samples mini-batches with a fixed foreground:background ratio such as 1:3. One-stage detectors such as **SSD** and **YOLO** were faster but less accurate.

The paper's diagnosis: in a one-stage detector, about 100k anchors per image are classified at once and nearly all are easy background. Each contributes a small cross-entropy loss, but together they dominate the gradient. SSD's answer was hard negative mining at 3:1. RetinaNet instead keeps every anchor and down-weights the easy ones in the loss.

## Architecture

The FPN builds pyramid levels P3 to P7 from the ResNet stages C3–C5 (see **ResNet**). P3–P5 come from the top-down pathway with $1\times1$ lateral connections. P6 is a $3\times3$ stride-2 convolution on C5, and P7 is ReLU plus another $3\times3$ stride-2 convolution on P6. Level $P_l$ has stride $2^l$ (8 to 128) and 256 channels. P2 is left out to save computation.

Every location of every level has $A=9$ anchors: areas $32^2$ (P3) to $512^2$ (P7), each with 3 scales $\{2^0,2^{1/3},2^{2/3}\}$ and 3 aspect ratios $\{1{:}2,1{:}1,2{:}1\}$, covering 32 to 813 px. An anchor is foreground if its IoU with a ground-truth box is at least 0.5, background if it is below 0.4, and ignored in between.

Two subnets, each applied to every level with weights shared across levels, make the predictions. The **classification subnet** is four $3\times3$ convolutions with 256 filters and ReLU, then a $3\times3$ convolution with $KA$ outputs and a sigmoid: one binary score per class and anchor, no softmax and no background class. The **box subnet** has the same structure, separate weights and $4A$ class-agnostic outputs, the Faster R-CNN box offsets trained with smooth $\ell_1$.

> Figure: RetinaNet. ResNet stages C3–C5 feed an FPN with levels P3–P7 (shapes for an 800×800 input, anchor base size per level on the right). The same class subnet (K·A = 720 sigmoid outputs for COCO) and box subnet (4A = 36 offsets) run on every level. Training applies focal loss to all anchors, foreground at IoU ≥ 0.5 and background below 0.4; inference keeps the top 1000 scores above 0.05 per level, then applies NMS at IoU 0.5.

## Loss and the prior initialisation

The classification loss is the α-balanced focal loss, summed over **all** anchors except the ignored ones and over all $K$ sigmoid outputs, then divided by the number of foreground anchors:

$$
\mathcal L_{\text{cls}}=\frac{1}{N_{\text{pos}}}\sum_{a}\sum_{k=1}^{K}-\alpha_t\,(1-p_{t})^{\gamma}\log p_{t},\qquad p_t=\begin{cases}p_{a,k}&y_{a,k}=1\\1-p_{a,k}&y_{a,k}=0\end{cases}
$$

- $p_{a,k}$: sigmoid output of anchor $a$ for class $k$
- $\gamma=2,\ \alpha=0.25$: the paper's best setting; $\alpha_t=\alpha$ for $y=1$ and $1-\alpha$ for $y=0$
- $N_{\text{pos}}$: number of anchors assigned to a ground-truth box, not the number of anchors

Normalising by $N_{\text{pos}}$ works because, under focal loss, the easy negatives contribute almost nothing. The second ingredient is the initialisation. With a zero bias every output starts at $p=0.5$, and the background outputs produce a huge loss in the first iterations. The paper sets the bias of the final classification layer so that every output starts at the prior $\pi$:

$$
b=-\log\frac{1-\pi}{\pi},\qquad \pi=0.01\ \Rightarrow\ b=-4.595,\quad \sigma(b)=0.01
$$

- $\pi$: the model's initial foreground probability for every anchor and class
- $b$: bias of the last $3\times3$ conv of the class subnet; other new layers get bias 0 and Gaussian weights with $\sigma=0.01$

The paper reports that this prior is what keeps training stable at the start, for cross-entropy as well as focal loss. The loss itself, its gradient and the roles of $\gamma$ and $\alpha$ are covered in **Focal loss**; here the point is what it does to one detector image.

**Configuration and results (Lin et al., 2017)**

- **Training:** SGD on 8 GPUs, 16 images per batch, 90k iterations; images resized to a shorter side of 800 px for the main model
- **Inference:** per level, the top 1000 boxes with score > 0.05; then NMS at IoU 0.5 over all levels
- **Ablation (ResNet-50, 600 px, COCO minival):** best α-balanced cross-entropy 31.1 AP (α = 0.75); focal loss γ = 2, α = 0.25: 34.0 AP; best OHEM 32.8 AP
- **COCO test-dev:** RetinaNet ResNet-101-FPN, 800 px: 39.1 AP; ResNeXt-101-FPN: 40.8 AP. Faster R-CNN with ResNet-101-FPN: 36.2 AP

> Figure: Real anchors and matching for one image, and the classification loss before the first update (every output = π). Default (street, 800 px, π = 0.01, focal): 120,087 anchors, 118 foreground, 220 ignored, so background outnumbers foreground about 1,015 to 1. The total loss is 140.4: the foreground supplies 133.1 and the 119,749 background anchors only 7.2 (5.1%), 1.190 per positive. Switch to cross-entropy: 96,918.5, 99.4% from background. Also set π = 0.5 (zero bias): 6.65 million, 56,329 per positive. At 512 px, 6 of the 12 small objects get no positive anchor at all.

## Worked example: the loss at step 0

The street image at $800\times800$, $K=80$: 118 foreground, 119,749 background and 220 ignored anchors. A foreground anchor has 1 output with target 1 and 79 with target 0, so there are $118\cdot79+119{,}749\cdot80=9{,}589{,}242$ outputs with target 0 and 118 with target 1.

Classification loss at initialisation, every output p = π (street scene, 800 px)

| Setting | Loss per target-1 output | Loss per target-0 output | Total | Background share | ÷ N_pos = 118 |
| --- | --- | --- | --- | --- | --- |
| CE, π = 0.5 (zero bias) | −ln 0.5 = 0.693 | 0.693 | 6,646,838 | 100.0% | 56,329 |
| CE, π = 0.01 | −ln 0.01 = 4.605 | −ln 0.99 = 0.01005 | 96,918.5 | 99.4% | 821.3 |
| FL γ = 2, α = 0.25, π = 0.01 | 0.25 · 0.99² · 4.605 = 1.128 | 0.75 · 0.01² · 0.01005 = 7.54e-7 | 140.4 | 5.1% | 1.190 |

The prior cuts the background term per output by $0.693/0.01005\approx69\times$. Focal loss cuts it by another factor $0.75\cdot0.01^2$, about $13{,}000\times$. Only with both is the loss per positive of order 1, so dividing by $N_{\text{pos}}$ gives a well-scaled loss from the first iteration.

```python
import math, torch, torchvision
from torchvision.models.detection import retinanet_resnet50_fpn, RetinaNet_ResNet50_FPN_Weights

# a fresh model shows the paper's initialisation of the class subnet
fresh = retinanet_resnet50_fpn(weights=None, weights_backbone=None, num_classes=91)
b = fresh.head.classification_head.cls_logits.bias
print(b[0].item(), -math.log(0.99 / 0.01))       # both -4.595
print(fresh.anchor_generator.sizes[0])            # (32, 40, 50): 32 * 2**(0, 1/3, 2/3) on P3
print(fresh.anchor_generator.aspect_ratios[0])    # (0.5, 1.0, 2.0)

# a trained model for inference
weights = RetinaNet_ResNet50_FPN_Weights.COCO_V1
model = retinanet_resnet50_fpn(weights=weights).eval()
print(model.score_thresh, model.topk_candidates, model.nms_thresh)   # 0.05 1000 0.5
img = torchvision.io.read_image("street.jpg")
with torch.no_grad():
    out = model([weights.transforms()(img)])[0]
keep = out["scores"] > 0.5
names = [weights.meta["categories"][i] for i in out["labels"][keep]]
print(list(zip(names, out["boxes"][keep].round().tolist())))

# training uses torchvision.ops.sigmoid_focal_loss(logits, targets, alpha=0.25, gamma=2, reduction="sum") / num_foreground
```

RetinaNet in torchvision: anchors, the prior bias and inference (not run here; requires torch and torchvision with downloaded COCO weights)

> **Thin and tiny objects can get zero positives:** Matching is by IoU with fixed anchors. An object much smaller than 32 px, or with an aspect ratio far from 1:2–2:1, may have no anchor at IoU ≥ 0.5 and then trains nothing but background around it. In the figure, 6 of 12 small objects are lost this way at 512 px, and none at 800 px. Implementations such as torchvision and Detectron also assign each ground truth its highest-IoU anchors ("low-quality matches"); anchor-free heads such as FCOS avoid the problem.

> **The architecture was not the point:** RetinaNet's network is an FPN with a standard head. The ablations show that with focal loss and the prior, this one-stage design beats OHEM and α-balanced cross-entropy by 1.2 and 2.9 AP, and at ResNet-101 scale it surpasses two-stage Faster R-CNN with FPN. The lesson that carried over to later dense detectors such as FCOS and ATSS is that class imbalance, not the one-stage design itself, held dense detectors back.

**Question:** At initialisation every classification output of RetinaNet is 0.01. Why is the loss divided by the number of foreground anchors and not by all ~100k anchors?

- A. Because background anchors have no box targets
- B. Under focal loss the easy background contributes almost nothing, so the loss scales with the positives; dividing by ~100k would shrink it by orders of magnitude
- C. Because ignored anchors would otherwise be counted twice
- D. It is equivalent, since the anchor count is constant

**Answer:** B. Under focal loss the easy background contributes almost nothing, so the loss scales with the positives; dividing by ~100k would shrink it by orders of magnitude — In the worked example the focal loss is 140.4, of which 133.1 comes from the 118 positives. Dividing by 118 gives 1.19 per positive. Dividing by 120,087 anchors would give 0.0012 and a far smaller effective learning rate.

**Question:** What does the classification subnet of RetinaNet output for one level with an H×W map and COCO's K = 80 classes?

- A. H×W×81 softmax scores
- B. H×W×720 sigmoid scores
- C. H×W×36 offsets
- D. H×W×9 objectness scores

**Answer:** B. H×W×720 sigmoid scores — $K\cdot A=80\cdot9=720$ independent sigmoid outputs per location, one per class and anchor. There is no background class: background means every output is low. The box subnet outputs the $4A=36$ offsets.

**Key points**

- One-stage: ResNet + FPN levels P3–P7 (strides 8–128, 256 channels) + a class subnet and a box subnet shared across levels.
- 9 anchors per location (3 scales × 3 ratios, 32–512 px base); IoU ≥ 0.5 foreground, < 0.4 background, between ignored.
- Focal loss (γ = 2, α = 0.25) over all ~100k anchors, divided by the number of foreground anchors.
- Final class bias −log((1−π)/π) with π = 0.01 keeps the step-0 loss from being dominated by background.
- 39.1 AP on COCO test-dev with ResNet-101-FPN, above Faster R-CNN with FPN (36.2).

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
