Definition
Faster R-CNN is a two-stage object detector (Shaoqing Ren, Kaiming He, Ross Girshick and Jian Sun, "Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks", NeurIPS 2015). A fully convolutional Region Proposal Network (RPN) predicts object proposals from the same feature map that the Fast R-CNN detection head uses, so the whole detector is one network and proposals are almost free.
Fast R-CNN made detection cheap once proposals exist: 0.32 s per image with VGG16 on a GPU. But selective search needs about 2 s per image on a CPU, so proposals had become the bottleneck. They were also fixed hand-crafted boxes that did not improve with training.
The observation behind Faster R-CNN: the conv feature map already used by the detector contains enough information to say where objects are. A small network slid over that map can predict proposals, and it adds about 10 ms per image.
The Region Proposal Network
The RPN runs a 3×3 conv (512 channels for VGG16) over the conv5_3 map, followed by ReLU. At each position, two sibling 1×1 convs predict, for reference boxes called anchors, 2 objectness scores ( outputs) and 4 box offsets ( outputs). The 3×3 window on conv5_3 sees a 228-pixel region of the image.
Anchors are fixed boxes centred on each feature-map position, with : three areas (128², 256², 512² pixels) times three aspect ratios (1:1, 1:2, 2:1). A 600×1000 image has about 60×40×9 ≈ 20,000 of them. They come from the same 9 shapes at every position, so the proposal layer is translation-invariant. Its output layer has only 512 × (4 + 2) × 9 = 27,648 parameters for VGG16, against 6.1 million for MultiBox's fully connected output.
The offsets use R-CNN's parameterisation, with the anchor in place of the proposal: , , , . Each of the 9 regressors is responsible for one anchor shape, so boxes of many sizes are predicted from a single feature map without an image pyramid.
Labelling anchors and the RPN loss
Anchor labels for one training image
- 1Positive: IoU > 0.7 with any ground-truth box, or the anchor (or anchors) with the highest IoU for a ground-truth box. The second rule exists because some boxes have no anchor above 0.7.
- 2Negative: IoU < 0.3 with every ground-truth box.
- 3Everything else, and every anchor that crosses the image border, is ignored during training. Of about 20,000 anchors per 1000×600 image, roughly 6,000 remain.
- 4Sample 256 anchors per image with up to a 1:1 ratio of positives to negatives. If there are fewer than 128 positives, negatives fill the batch.
- predicted objectness of anchor and its label (1 positive, 0 negative); is log loss
- smooth L1 on , counted only for positive anchors ()
- mini-batch size 256, number of anchor positions (about 2,400) and , which weights both terms about equally
Worked example: the 9 anchors at one position
Ground truth: the car from the figure, , a 360×180 box with centre (430, 420). The 9 anchors centred on feature position (424, 424):
| Area | w × h | IoU | Label |
|---|---|---|---|
| 128² | 181 × 91, 128 × 128, 91 × 181 | 0.253, 0.253, 0.245 | negative (< 0.3) |
| 256² | 362 × 181 | 0.926 | positive (> 0.7) |
| 256² | 256 × 256 | 0.547 | ignored |
| 256² | 181 × 362 | 0.333 | crosses the bottom edge: dropped |
| 512² | 724 × 362, 512 × 512, 362 × 724 | 0.247, 0.247, 0.243 | cross the border: dropped |
The positive anchor's targets are . If the RPN predicts instead, decoding gives centre and size .
Training the shared network
The RPN and the detector must share conv layers, but training them separately would change those layers twice. The paper's main procedure is 4-step alternating training: (1) train the RPN from an ImageNet model; (2) train a separate Fast R-CNN on the RPN's proposals, again from ImageNet; (3) initialise the RPN from the detector, freeze the shared conv layers and fine-tune only the RPN layers; (4) keep them frozen and fine-tune the Fast R-CNN layers. The two networks now share one trunk.
Approximate joint training merges both into one network and adds the losses. It ignores the gradient with respect to the proposal coordinates, because RoI pooling is not differentiable with respect to the box. It gives similar accuracy and cuts training time by about 25–50%, and later implementations use it.
Reported results (VGG16 unless noted)
- VOC 2007 test mAP
- 69.9% (trained on 07), 73.2% (07+12), 78.8% (COCO + 07 + 12), with 300 RPN proposals. Fast R-CNN with 2,000 selective-search boxes: 66.9% / 70.0%.
- Speed
- 198 ms per image on a K40 GPU (5 fps) including proposals, vs about 1.8 s with selective search. The ZF backbone runs at 17 fps.
- MS COCO test-dev
- 42.7% AP@0.5 and 21.9% AP@[.5:.95] trained on COCO trainval
- Feature Pyramid Network
- Lin et al. (CVPR 2017) attach anchors of one area per pyramid level (32² on P2 to 512² on P6, 3 ratios each). Faster R-CNN with FPN and ResNet-101 reaches 36.2 COCO AP and became the standard two-stage baseline.
1import torch2from torchvision.io import read_image3from torchvision.models.detection import fasterrcnn_resnet50_fpn, FasterRCNN_ResNet50_FPN_Weights45weights = FasterRCNN_ResNet50_FPN_Weights.COCO_V16model = fasterrcnn_resnet50_fpn(weights=weights, box_score_thresh=0.5).eval()7img = weights.transforms()(read_image("street.jpg")) # uint8 CHW -> float in [0, 1]89with torch.no_grad():10 out = model([img])[0] # dict: boxes (x1,y1,x2,y2), labels, scores11names = weights.meta["categories"]12for box, lab, s in zip(out["boxes"], out["labels"], out["scores"]):13 print(names[lab], round(s.item(), 3), box.round().tolist())1415# RPN settings live on the model, e.g. NMS threshold and proposals kept per image at test time16print(model.rpn.nms_thresh, model.rpn._post_nms_top_n["testing"]) # 0.7, 1000Check yourself
An image has 2,400 feature-map positions and k = 9. How many output channels do the RPN's two 1×1 sibling convs have in total?
Check yourself
Why is the anchor with the highest IoU for a ground-truth box made positive even if its IoU is below 0.7?
Remember
- ✓The RPN is a 3×3 conv plus two 1×1 convs on the shared feature map, with 2k objectness and 4k offset outputs per position.
- ✓k = 9 anchors: areas 128², 256², 512² × ratios 1:1, 1:2, 2:1, about 20,000 per 600×1000 image.
- ✓Anchors are positive if IoU > 0.7 or best for a box, negative if IoU < 0.3, ignored otherwise. 256 sampled anchors per image, λ = 10.
- ✓4-step alternating training shares the conv layers; approximate joint training is simpler and 25–50% faster.
- ✓VGG16: 73.2% VOC07 mAP at 5 fps with 300 proposals. FPN (2017) added multi-scale anchors and became the standard backbone.