# Faster R-CNN

> Faster R-CNN replaces selective search with a Region Proposal Network that shares the detector's conv features: 9 anchors per position are scored and regressed, so proposals cost about 10 ms and the whole two-stage detector runs at 5 fps with VGG16.

Source: https://www.mlfoundry.in/learn/faster-r-cnn  
Section: Deep Learning › Deep Learning for Computer Vision  
Level: core · 8 min read · updated 2026-10-15

**Faster R-CNN:** is a two-stage object detector (Shaoqing Ren, Kaiming He, Ross Girshick and Jian Sun, "Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks", NeurIPS 2015). A fully convolutional **Region Proposal Network (RPN)** predicts object proposals from the same feature map that the **Fast R-CNN** detection head uses, so the whole detector is one network and proposals are almost free.

**Fast R-CNN** made detection cheap once proposals exist: 0.32 s per image with VGG16 on a GPU. But selective search needs about 2 s per image on a CPU, so proposals had become the bottleneck. They were also fixed hand-crafted boxes that did not improve with training.

The observation behind Faster R-CNN: the conv feature map already used by the detector contains enough information to say where objects are. A small network slid over that map can predict proposals, and it adds about 10 ms per image.

## The Region Proposal Network

The RPN runs a 3×3 conv (512 channels for VGG16) over the conv5_3 map, followed by ReLU. At each position, two sibling 1×1 convs predict, for $k$ reference boxes called **anchors**, 2 objectness scores ($2k$ outputs) and 4 box offsets ($4k$ outputs). The 3×3 window on conv5_3 sees a 228-pixel region of the image.

> Figure: Faster R-CNN. The VGG16 trunk runs once. The RPN slides a 3 × 3 conv over the 38 × 63 conv5_3 map and, for each of the 9 anchors at every position, outputs 2 objectness scores and 4 offsets (18 + 36 channels). The roughly 20,000 decoded boxes are cut by NMS at IoU 0.7 to about 2,000, of which the top 300 are RoI-pooled from the same map into the Fast R-CNN head.

Anchors are fixed boxes centred on each feature-map position, with $k=9$: three areas (128², 256², 512² pixels) times three aspect ratios (1:1, 1:2, 2:1). A 600×1000 image has about 60×40×9 ≈ 20,000 of them. They come from the same 9 shapes at every position, so the proposal layer is translation-invariant. Its output layer has only 512 × (4 + 2) × 9 = 27,648 parameters for VGG16, against 6.1 million for MultiBox's fully connected output.

The offsets use **R-CNN**'s parameterisation, with the anchor in place of the proposal: $t_x=(x-x_a)/w_a$, $t_y=(y-y_a)/h_a$, $t_w=\log(w/w_a)$, $t_h=\log(h/h_a)$. Each of the 9 regressors is responsible for one anchor shape, so boxes of many sizes are predicted from a single feature map without an image pyramid.

## Labelling anchors and the RPN loss

**Anchor labels for one training image**

1. Positive: IoU > 0.7 with any ground-truth box, or the anchor (or anchors) with the highest IoU for a ground-truth box. The second rule exists because some boxes have no anchor above 0.7.
2. Negative: IoU < 0.3 with every ground-truth box.
3. Everything else, and every anchor that crosses the image border, is ignored during training. Of about 20,000 anchors per 1000×600 image, roughly 6,000 remain.
4. Sample 256 anchors per image with up to a 1:1 ratio of positives to negatives. If there are fewer than 128 positives, negatives fill the batch.

$$
L(\{p_i\},\{t_i\})=\frac{1}{N_{\text{cls}}}\sum_i L_{\text{cls}}(p_i,p_i^*)+\lambda\,\frac{1}{N_{\text{reg}}}\sum_i p_i^*\,L_{\text{reg}}(t_i,t_i^*)
$$

- $p_i,\ p_i^*$: predicted objectness of anchor $i$ and its label (1 positive, 0 negative); $L_{\text{cls}}$ is log loss
- $L_{\text{reg}}$: smooth L1 on $t_i-t_i^*$, counted only for positive anchors ($p_i^*=1$)
- $N_{\text{cls}},\ N_{\text{reg}},\ \lambda$: mini-batch size 256, number of anchor positions (about 2,400) and $\lambda=10$, which weights both terms about equally

> Figure: RPN anchor labelling on a 1000 × 600 image: 63 × 38 × 9 = 21,546 idealised anchors, of which 8,044 lie inside the image (13,502 cross-boundary anchors are dropped; the paper reports about 6,000 left with its rounded anchor sizes). The default 360 × 180 car matches the 256², 2:1 anchor at (424, 424) with IoU 0.926: 18 positives, 7,624 negatives and 402 ignored, and the batch is 18 positive + 238 negative. The 120 × 250 person reaches only IoU 0.546, so its single positive comes from the max-IoU rule; turn the rule off and it has none. The 64 × 48 bird is rescued by an anchor with IoU 0.188, a poor match, because the smallest anchor is 128². Lowering the positive threshold to 0.5 gives the car 114 positives.

## Worked example: the 9 anchors at one position

Ground truth: the car from the figure, $(x_1,y_1,x_2,y_2)=(250,330,610,510)$, a 360×180 box with centre (430, 420). The 9 anchors centred on feature position (424, 424):

IoU and RPN label of each anchor (computed in Python)

| Area | w × h | IoU | Label |
| --- | --- | --- | --- |
| 128² | 181 × 91, 128 × 128, 91 × 181 | 0.253, 0.253, 0.245 | negative (< 0.3) |
| 256² | 362 × 181 | 0.926 | positive (> 0.7) |
| 256² | 256 × 256 | 0.547 | ignored |
| 256² | 181 × 362 | 0.333 | crosses the bottom edge: dropped |
| 512² | 724 × 362, 512 × 512, 362 × 724 | 0.247, 0.247, 0.243 | cross the border: dropped |

The positive anchor's targets are $t^*=(6/362.0,\,-4/181.0,\,\ln(360/362.0),\,\ln(180/181.0))=(0.017,-0.022,-0.006,-0.006)$. If the RPN predicts $t=(0.03,-0.01,0.05,-0.02)$ instead, decoding gives centre $(424+0.03\cdot362.0,\ 424-0.01\cdot181.0)=(434.9,422.2)$ and size $362.0e^{0.05}\times181.0e^{-0.02}=380.6\times177.4$.

## Training the shared network

The RPN and the detector must share conv layers, but training them separately would change those layers twice. The paper's main procedure is **4-step alternating training**: (1) train the RPN from an ImageNet model; (2) train a separate Fast R-CNN on the RPN's proposals, again from ImageNet; (3) initialise the RPN from the detector, freeze the shared conv layers and fine-tune only the RPN layers; (4) keep them frozen and fine-tune the Fast R-CNN layers. The two networks now share one trunk.

**Approximate joint training** merges both into one network and adds the losses. It ignores the gradient with respect to the proposal coordinates, because RoI pooling is not differentiable with respect to the box. It gives similar accuracy and cuts training time by about 25–50%, and later implementations use it.

**Reported results (VGG16 unless noted)**

- **VOC 2007 test mAP:** 69.9% (trained on 07), 73.2% (07+12), 78.8% (COCO + 07 + 12), with 300 RPN proposals. Fast R-CNN with 2,000 selective-search boxes: 66.9% / 70.0%.
- **Speed:** 198 ms per image on a K40 GPU (5 fps) including proposals, vs about 1.8 s with selective search. The ZF backbone runs at 17 fps.
- **MS COCO test-dev:** 42.7% AP@0.5 and 21.9% AP@[.5:.95] trained on COCO trainval
- **Feature Pyramid Network:** Lin et al. (CVPR 2017) attach anchors of one area per pyramid level (32² on P2 to 512² on P6, 3 ratios each). Faster R-CNN with FPN and ResNet-101 reaches 36.2 COCO AP and became the standard two-stage baseline.

```python
import torch
from torchvision.io import read_image
from torchvision.models.detection import fasterrcnn_resnet50_fpn, FasterRCNN_ResNet50_FPN_Weights

weights = FasterRCNN_ResNet50_FPN_Weights.COCO_V1
model = fasterrcnn_resnet50_fpn(weights=weights, box_score_thresh=0.5).eval()
img = weights.transforms()(read_image("street.jpg"))     # uint8 CHW -> float in [0, 1]

with torch.no_grad():
    out = model([img])[0]                                 # dict: boxes (x1,y1,x2,y2), labels, scores
names = weights.meta["categories"]
for box, lab, s in zip(out["boxes"], out["labels"], out["scores"]):
    print(names[lab], round(s.item(), 3), box.round().tolist())

# RPN settings live on the model, e.g. NMS threshold and proposals kept per image at test time
print(model.rpn.nms_thresh, model.rpn._post_nms_top_n["testing"])   # 0.7, 1000
```

Inference with torchvision's COCO-trained Faster R-CNN (ResNet-50-FPN); weights download on first use (not run here)

> **Anchors turn detection into classification:** Every anchor is a fixed hypothesis about where an object could be. The network only needs to say whether each one holds an object and how to nudge it. The same idea, with the second stage removed, drives one-stage detectors such as **SSD**, **YOLO** (v2 onward) and **RetinaNet**. **DETR** later dropped anchors and NMS altogether.

> **Anchor sizes must cover your objects:** With the original anchors the smallest box is 128², so a 64×48 object has a best IoU of about 0.19 and is learned only through the max-IoU rule. For small objects, add smaller anchors or use FPN, whose P2 level has 32² anchors at stride 4.

**Question:** An image has 2,400 feature-map positions and k = 9. How many output channels do the RPN's two 1×1 sibling convs have in total?

- A. 2 + 4 = 6
- B. 9 × (2 + 4) = 54
- C. 2,400 × 54 = 129,600
- D. 9 × 4 = 36

**Answer:** B. 9 × (2 + 4) = 54 — The 1×1 convs are shared across positions: 2k = 18 objectness channels plus 4k = 36 offset channels, 54 in total. The 2,400 positions only change the spatial size of the output, not the number of channels or weights.

**Question:** Why is the anchor with the highest IoU for a ground-truth box made positive even if its IoU is below 0.7?

- A. To balance the 1:1 ratio of positives to negatives
- B. Some objects have no anchor above 0.7, and without the rule they would never be learned
- C. Because NMS at test time requires one positive per object
- D. It is only a speed optimisation

**Answer:** B. Some objects have no anchor above 0.7, and without the rule they would never be learned — Unusually shaped or small objects (a 120 × 250 person reaches only 0.546 in the figure) may match no anchor above 0.7. The max-IoU rule guarantees that every ground-truth box gets at least one positive anchor.

**Key points**

- The RPN is a 3×3 conv plus two 1×1 convs on the shared feature map, with 2k objectness and 4k offset outputs per position.
- k = 9 anchors: areas 128², 256², 512² × ratios 1:1, 1:2, 2:1, about 20,000 per 600×1000 image.
- Anchors are positive if IoU > 0.7 or best for a box, negative if IoU < 0.3, ignored otherwise. 256 sampled anchors per image, λ = 10.
- 4-step alternating training shares the conv layers; approximate joint training is simpler and 25–50% faster.
- VGG16: 73.2% VOC07 mAP at 5 fps with 300 proposals. FPN (2017) added multi-scale anchors and became the standard backbone.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
