Faster R-CNN

Faster R-CNN replaces selective search with a Region Proposal Network that shares the detector's conv features: 9 anchors per position are scored and regressed, so proposals cost about 10 ms and the whole two-stage detector runs at 5 fps with VGG16.

8 min read791 wordscore0 viewsupdated 2026-10-15

Definition

Faster R-CNN is a two-stage object detector (Shaoqing Ren, Kaiming He, Ross Girshick and Jian Sun, "Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks", NeurIPS 2015). A fully convolutional Region Proposal Network (RPN) predicts object proposals from the same feature map that the Fast R-CNN detection head uses, so the whole detector is one network and proposals are almost free.

Fast R-CNN made detection cheap once proposals exist: 0.32 s per image with VGG16 on a GPU. But selective search needs about 2 s per image on a CPU, so proposals had become the bottleneck. They were also fixed hand-crafted boxes that did not improve with training.

The observation behind Faster R-CNN: the conv feature map already used by the detector contains enough information to say where objects are. A small network slid over that map can predict proposals, and it adds about 10 ms per image.

The Region Proposal Network

The RPN runs a 3×3 conv (512 channels for VGG16) over the conv5_3 map, followed by ReLU. At each position, two sibling 1×1 convs predict, for kk reference boxes called anchors, 2 objectness scores (2k2k outputs) and 4 box offsets (4k4k outputs). The 3×3 window on conv5_3 sees a 228-pixel region of the image.

Faster R-CNN (Ren, He, Girshick, Sun 2015): proposals come from the same features9 anchors per position128² 256² 512² × 1:2 1:1 2:1VGG1613 convshared trunkconv5_3 map38 × 63 × 512(600 × 1000 image)Region Proposal Network3×3 conv, 512 + ReLUcls 1×12k = 18reg 1×14k = 36object? + (tx, ty, tw, th)decode ~20k anchorsNMS at IoU 0.7 (~2k left)keep top 300 at testRoI pooling 7×7same conv5_3 mapFast R-CNN headfc6, fc7 (4096)softmax K+1 · boxes 4KRPN loss = (1/N_cls) Σᵢ L_cls(pᵢ, pᵢ*) + λ (1/N_reg) Σᵢ pᵢ* smoothL1(tᵢ − tᵢ*)N_cls = 256 sampled anchors, N_reg ≈ 2,400 anchor positions, λ = 10anchor label pᵢ* = 1 if IoU > 0.7 (or best for a box), 0 if IoU < 0.3, else ignored
fig.Faster R-CNN. The VGG16 trunk runs once. The RPN slides a 3 × 3 conv over the 38 × 63 conv5_3 map and, for each of the 9 anchors at every position, outputs 2 objectness scores and 4 offsets (18 + 36 channels). The roughly 20,000 decoded boxes are cut by NMS at IoU 0.7 to about 2,000, of which the top 300 are RoI-pooled from the same map into the Fast R-CNN head.

Anchors are fixed boxes centred on each feature-map position, with k=9k=9: three areas (128², 256², 512² pixels) times three aspect ratios (1:1, 1:2, 2:1). A 600×1000 image has about 60×40×9 ≈ 20,000 of them. They come from the same 9 shapes at every position, so the proposal layer is translation-invariant. Its output layer has only 512 × (4 + 2) × 9 = 27,648 parameters for VGG16, against 6.1 million for MultiBox's fully connected output.

The offsets use R-CNN's parameterisation, with the anchor in place of the proposal: tx=(x−xa)/wat_x=(x-x_a)/w_a, ty=(y−ya)/hat_y=(y-y_a)/h_a, tw=log⁡(w/wa)t_w=\log(w/w_a), th=log⁡(h/ha)t_h=\log(h/h_a). Each of the 9 regressors is responsible for one anchor shape, so boxes of many sizes are predicted from a single feature map without an image pyramid.

Labelling anchors and the RPN loss

Anchor labels for one training image

  1. 1Positive: IoU > 0.7 with any ground-truth box, or the anchor (or anchors) with the highest IoU for a ground-truth box. The second rule exists because some boxes have no anchor above 0.7.
  2. 2Negative: IoU < 0.3 with every ground-truth box.
  3. 3Everything else, and every anchor that crosses the image border, is ignored during training. Of about 20,000 anchors per 1000×600 image, roughly 6,000 remain.
  4. 4Sample 256 anchors per image with up to a 1:1 ratio of positives to negatives. If there are fewer than 128 positives, negatives fill the batch.
L({pi},{ti})=1Ncls∑iLcls(pi,pi∗)+λ 1Nreg∑ipi∗ Lreg(ti,ti∗)L(\{p_i\},\{t_i\})=\frac{1}{N_{\text{cls}}}\sum_i L_{\text{cls}}(p_i,p_i^*)+\lambda\,\frac{1}{N_{\text{reg}}}\sum_i p_i^*\,L_{\text{reg}}(t_i,t_i^*)
pi, pi∗p_i,\ p_i^*
predicted objectness of anchor ii and its label (1 positive, 0 negative); LclsL_{\text{cls}} is log loss
LregL_{\text{reg}}
smooth L1 on ti−ti∗t_i-t_i^*, counted only for positive anchors (pi∗=1p_i^*=1)
Ncls, Nreg, λN_{\text{cls}},\ N_{\text{reg}},\ \lambda
mini-batch size 256, number of anchor positions (about 2,400) and λ=10\lambda=10, which weights both terms about equally
1000×600 image · 9 anchors per stride-16 cellground truth · positive anchors · best anchor · its 9 siblingsRPN labels for the caranchors 63×38×9 = 21,546cross-boundary dropped: 13,502positive: 18 (IoU > 0.70)negative: 7,624 (IoU < 0.30)ignored: 402best IoU per anchor scale128²0.253256²0.926512²0.247best: 256², w:h 2:1, IoU 0.926anchor 362.0×181.0 at (424, 424)tx 0.017 ty −0.022tw −0.006 th −0.006mini-batch of 256 anchors from this image: 18 positive + 238 negative (negatives fill the rest)18 anchors overlap the box by more than 0.70; each regresses its own t towards itanchor ratios h/w ∈ {0.5, 1, 2}, areas 128², 256², 512²; ignored anchors get no loss
object
rules
fig.RPN anchor labelling on a 1000 × 600 image: 63 × 38 × 9 = 21,546 idealised anchors, of which 8,044 lie inside the image (13,502 cross-boundary anchors are dropped; the paper reports about 6,000 left with its rounded anchor sizes). The default 360 × 180 car matches the 256², 2:1 anchor at (424, 424) with IoU 0.926: 18 positives, 7,624 negatives and 402 ignored, and the batch is 18 positive + 238 negative. The 120 × 250 person reaches only IoU 0.546, so its single positive comes from the max-IoU rule; turn the rule off and it has none. The 64 × 48 bird is rescued by an anchor with IoU 0.188, a poor match, because the smallest anchor is 128². Lowering the positive threshold to 0.5 gives the car 114 positives.

Worked example: the 9 anchors at one position

Ground truth: the car from the figure, (x1,y1,x2,y2)=(250,330,610,510)(x_1,y_1,x_2,y_2)=(250,330,610,510), a 360×180 box with centre (430, 420). The 9 anchors centred on feature position (424, 424):

Areaw × hIoULabel
128²181 × 91, 128 × 128, 91 × 1810.253, 0.253, 0.245negative (< 0.3)
256²362 × 1810.926positive (> 0.7)
256²256 × 2560.547ignored
256²181 × 3620.333crosses the bottom edge: dropped
512²724 × 362, 512 × 512, 362 × 7240.247, 0.247, 0.243cross the border: dropped
IoU and RPN label of each anchor (computed in Python)

The positive anchor's targets are t∗=(6/362.0, −4/181.0, ln⁡(360/362.0), ln⁡(180/181.0))=(0.017,−0.022,−0.006,−0.006)t^*=(6/362.0,\,-4/181.0,\,\ln(360/362.0),\,\ln(180/181.0))=(0.017,-0.022,-0.006,-0.006). If the RPN predicts t=(0.03,−0.01,0.05,−0.02)t=(0.03,-0.01,0.05,-0.02) instead, decoding gives centre (424+0.03⋅362.0, 424−0.01⋅181.0)=(434.9,422.2)(424+0.03\cdot362.0,\ 424-0.01\cdot181.0)=(434.9,422.2) and size 362.0e0.05×181.0e−0.02=380.6×177.4362.0e^{0.05}\times181.0e^{-0.02}=380.6\times177.4.

Training the shared network

The RPN and the detector must share conv layers, but training them separately would change those layers twice. The paper's main procedure is 4-step alternating training: (1) train the RPN from an ImageNet model; (2) train a separate Fast R-CNN on the RPN's proposals, again from ImageNet; (3) initialise the RPN from the detector, freeze the shared conv layers and fine-tune only the RPN layers; (4) keep them frozen and fine-tune the Fast R-CNN layers. The two networks now share one trunk.

Approximate joint training merges both into one network and adds the losses. It ignores the gradient with respect to the proposal coordinates, because RoI pooling is not differentiable with respect to the box. It gives similar accuracy and cuts training time by about 25–50%, and later implementations use it.

Reported results (VGG16 unless noted)

VOC 2007 test mAP
69.9% (trained on 07), 73.2% (07+12), 78.8% (COCO + 07 + 12), with 300 RPN proposals. Fast R-CNN with 2,000 selective-search boxes: 66.9% / 70.0%.
Speed
198 ms per image on a K40 GPU (5 fps) including proposals, vs about 1.8 s with selective search. The ZF backbone runs at 17 fps.
MS COCO test-dev
42.7% AP@0.5 and 21.9% AP@[.5:.95] trained on COCO trainval
Feature Pyramid Network
Lin et al. (CVPR 2017) attach anchors of one area per pyramid level (32² on P2 to 512² on P6, 3 ratios each). Faster R-CNN with FPN and ResNet-101 reaches 36.2 COCO AP and became the standard two-stage baseline.
python
1import torch
2from torchvision.io import read_image
3from torchvision.models.detection import fasterrcnn_resnet50_fpn, FasterRCNN_ResNet50_FPN_Weights
4
5weights = FasterRCNN_ResNet50_FPN_Weights.COCO_V1
6model = fasterrcnn_resnet50_fpn(weights=weights, box_score_thresh=0.5).eval()
7img = weights.transforms()(read_image("street.jpg")) # uint8 CHW -> float in [0, 1]
8
9with torch.no_grad():
10 out = model([img])[0] # dict: boxes (x1,y1,x2,y2), labels, scores
11names = weights.meta["categories"]
12for box, lab, s in zip(out["boxes"], out["labels"], out["scores"]):
13 print(names[lab], round(s.item(), 3), box.round().tolist())
14
15# RPN settings live on the model, e.g. NMS threshold and proposals kept per image at test time
16print(model.rpn.nms_thresh, model.rpn._post_nms_top_n["testing"]) # 0.7, 1000
Inference with torchvision's COCO-trained Faster R-CNN (ResNet-50-FPN); weights download on first use (not run here)

Check yourself

An image has 2,400 feature-map positions and k = 9. How many output channels do the RPN's two 1×1 sibling convs have in total?

Check yourself

Why is the anchor with the highest IoU for a ground-truth box made positive even if its IoU is below 0.7?

Remember

  • ✓The RPN is a 3×3 conv plus two 1×1 convs on the shared feature map, with 2k objectness and 4k offset outputs per position.
  • ✓k = 9 anchors: areas 128², 256², 512² × ratios 1:1, 1:2, 2:1, about 20,000 per 600×1000 image.
  • ✓Anchors are positive if IoU > 0.7 or best for a box, negative if IoU < 0.3, ignored otherwise. 256 sampled anchors per image, λ = 10.
  • ✓4-step alternating training shares the conv layers; approximate joint training is simpler and 25–50% faster.
  • ✓VGG16: 73.2% VOC07 mAP at 5 fps with 300 proposals. FPN (2017) added multi-scale anchors and became the standard backbone.
← back to Deep Learning roadmap