Definition
RetinaNet is the one-stage object detector of Lin, Goyal, Girshick, He and Dollár, "Focal Loss for Dense Object Detection" (ICCV 2017). It is a deliberately plain architecture: a ResNet backbone, a feature pyramid network (FPN) and two small convolutional subnets. The new part is the training loss. Focal loss lets a dense detector train on every anchor without being swamped by easy background, and with it a one-stage detector matched two-stage accuracy for the first time.
Input: an image, shorter side 800 px in the main results. Output: boxes with class scores. Two-stage detectors (Faster R-CNN) were the most accurate. Their first stage cuts about 100k candidate locations down to 1–2k proposals, and the second stage samples mini-batches with a fixed foreground:background ratio such as 1:3. One-stage detectors such as SSD and YOLO were faster but less accurate.
The paper's diagnosis: in a one-stage detector, about 100k anchors per image are classified at once and nearly all are easy background. Each contributes a small cross-entropy loss, but together they dominate the gradient. SSD's answer was hard negative mining at 3:1. RetinaNet instead keeps every anchor and down-weights the easy ones in the loss.
Architecture
The FPN builds pyramid levels P3 to P7 from the ResNet stages C3–C5 (see ResNet). P3–P5 come from the top-down pathway with lateral connections. P6 is a stride-2 convolution on C5, and P7 is ReLU plus another stride-2 convolution on P6. Level has stride (8 to 128) and 256 channels. P2 is left out to save computation.
Every location of every level has anchors: areas (P3) to (P7), each with 3 scales and 3 aspect ratios , covering 32 to 813 px. An anchor is foreground if its IoU with a ground-truth box is at least 0.5, background if it is below 0.4, and ignored in between.
Two subnets, each applied to every level with weights shared across levels, make the predictions. The classification subnet is four convolutions with 256 filters and ReLU, then a convolution with outputs and a sigmoid: one binary score per class and anchor, no softmax and no background class. The box subnet has the same structure, separate weights and class-agnostic outputs, the Faster R-CNN box offsets trained with smooth .
Loss and the prior initialisation
The classification loss is the α-balanced focal loss, summed over all anchors except the ignored ones and over all sigmoid outputs, then divided by the number of foreground anchors:
- sigmoid output of anchor for class
- the paper's best setting; for and for
- number of anchors assigned to a ground-truth box, not the number of anchors
Normalising by works because, under focal loss, the easy negatives contribute almost nothing. The second ingredient is the initialisation. With a zero bias every output starts at , and the background outputs produce a huge loss in the first iterations. The paper sets the bias of the final classification layer so that every output starts at the prior :
- the model's initial foreground probability for every anchor and class
- bias of the last conv of the class subnet; other new layers get bias 0 and Gaussian weights with
The paper reports that this prior is what keeps training stable at the start, for cross-entropy as well as focal loss. The loss itself, its gradient and the roles of and are covered in Focal loss; here the point is what it does to one detector image.
Configuration and results (Lin et al., 2017)
- Training
- SGD on 8 GPUs, 16 images per batch, 90k iterations; images resized to a shorter side of 800 px for the main model
- Inference
- per level, the top 1000 boxes with score > 0.05; then NMS at IoU 0.5 over all levels
- Ablation (ResNet-50, 600 px, COCO minival)
- best α-balanced cross-entropy 31.1 AP (α = 0.75); focal loss γ = 2, α = 0.25: 34.0 AP; best OHEM 32.8 AP
- COCO test-dev
- RetinaNet ResNet-101-FPN, 800 px: 39.1 AP; ResNeXt-101-FPN: 40.8 AP. Faster R-CNN with ResNet-101-FPN: 36.2 AP
Worked example: the loss at step 0
The street image at , : 118 foreground, 119,749 background and 220 ignored anchors. A foreground anchor has 1 output with target 1 and 79 with target 0, so there are outputs with target 0 and 118 with target 1.
| Setting | Loss per target-1 output | Loss per target-0 output | Total | Background share | ÷ N_pos = 118 |
|---|---|---|---|---|---|
| CE, π = 0.5 (zero bias) | −ln 0.5 = 0.693 | 0.693 | 6,646,838 | 100.0% | 56,329 |
| CE, π = 0.01 | −ln 0.01 = 4.605 | −ln 0.99 = 0.01005 | 96,918.5 | 99.4% | 821.3 |
| FL γ = 2, α = 0.25, π = 0.01 | 0.25 · 0.99² · 4.605 = 1.128 | 0.75 · 0.01² · 0.01005 = 7.54e-7 | 140.4 | 5.1% | 1.190 |
The prior cuts the background term per output by . Focal loss cuts it by another factor , about . Only with both is the loss per positive of order 1, so dividing by gives a well-scaled loss from the first iteration.
1import math, torch, torchvision2from torchvision.models.detection import retinanet_resnet50_fpn, RetinaNet_ResNet50_FPN_Weights34# a fresh model shows the paper's initialisation of the class subnet5fresh = retinanet_resnet50_fpn(weights=None, weights_backbone=None, num_classes=91)6b = fresh.head.classification_head.cls_logits.bias7print(b[0].item(), -math.log(0.99 / 0.01)) # both -4.5958print(fresh.anchor_generator.sizes[0]) # (32, 40, 50): 32 * 2**(0, 1/3, 2/3) on P39print(fresh.anchor_generator.aspect_ratios[0]) # (0.5, 1.0, 2.0)1011# a trained model for inference12weights = RetinaNet_ResNet50_FPN_Weights.COCO_V113model = retinanet_resnet50_fpn(weights=weights).eval()14print(model.score_thresh, model.topk_candidates, model.nms_thresh) # 0.05 1000 0.515img = torchvision.io.read_image("street.jpg")16with torch.no_grad():17 out = model([weights.transforms()(img)])[0]18keep = out["scores"] > 0.519names = [weights.meta["categories"][i] for i in out["labels"][keep]]20print(list(zip(names, out["boxes"][keep].round().tolist())))2122# training uses torchvision.ops.sigmoid_focal_loss(logits, targets, alpha=0.25, gamma=2, reduction="sum") / num_foregroundCheck yourself
At initialisation every classification output of RetinaNet is 0.01. Why is the loss divided by the number of foreground anchors and not by all ~100k anchors?
Check yourself
What does the classification subnet of RetinaNet output for one level with an H×W map and COCO's K = 80 classes?
Remember
- ✓One-stage: ResNet + FPN levels P3–P7 (strides 8–128, 256 channels) + a class subnet and a box subnet shared across levels.
- ✓9 anchors per location (3 scales × 3 ratios, 32–512 px base); IoU ≥ 0.5 foreground, < 0.4 background, between ignored.
- ✓Focal loss (γ = 2, α = 0.25) over all ~100k anchors, divided by the number of foreground anchors.
- ✓Final class bias −log((1−π)/π) with π = 0.01 keeps the step-0 loss from being dominated by background.
- ✓39.1 AP on COCO test-dev with ResNet-101-FPN, above Faster R-CNN with FPN (36.2).