Definition
R-CNN ("Regions with CNN features") is the detector of Girshick, Donahue, Darrell and Malik, "Rich feature hierarchies for accurate object detection and semantic segmentation" (CVPR 2014). It was the first to show that CNN features pre-trained on ImageNet classification, then fine-tuned, beat hand-built features for detection by a wide margin: 53.3% mAP on PASCAL VOC 2012, over 30% relative better than the previous best.
Detection needs a class and a box for every object (see Object detection). In 2013 the strongest systems were deformable part models on HOG features and selective-search pipelines on bag-of-words SIFT features, around 33 to 35% mAP on VOC. Sliding a CNN over every position and scale was too expensive. R-CNN's idea was to run the CNN only on a few thousand likely regions and treat detection as classification of regions.
The pipeline
- 1Propose. Selective search (Uijlings et al., IJCV 2013) in "fast mode" merges superpixels hierarchically and returns about 2,000 class-agnostic boxes per image.
- 2Warp. Each box is dilated so that, after warping, 16 pixels of image context surround it, then resized to regardless of its aspect ratio.
- 3Extract. An AlexNet, pre-trained on ILSVRC-2012 and fine-tuned on warped regions, maps each crop to a 4096-dimensional fc7 feature.
- 4Score. One linear SVM per class scores every region: a feature matrix times a weight matrix.
- 5Suppress. Per class, greedy NMS rejects a region whose IoU with a higher-scoring kept region exceeds a learned threshold.
- 6Refine. A class-specific linear regressor on pool5 features predicts a correction to each box.
Training: three objectives, three label rules
Fine-tuning replaces AlexNet's 1000-way classifier with a 21-way one (20 VOC classes + background) and runs SGD at learning rate 0.001 on warped proposals. A proposal with IoU to a ground-truth box is a positive for that class; all others are background. Each mini-batch of 128 has 32 positives and 96 background windows, because positives are rare.
SVMs use stricter labels: only the ground-truth boxes are positives, proposals with IoU are negatives (0.3 was chosen by grid search over 0 to 0.5), and everything in between is ignored. Standard hard-negative mining keeps the negative set manageable. The paper's appendix reports that using the fine-tuned softmax directly instead of SVMs drops VOC 2007 mAP from 54.2% to 50.9%; Fast R-CNN later removed the SVMs once training was end to end.
Bounding-box regression
A proposal (centre, width, height) is assigned to the ground-truth box it overlaps most, and used for training only if that IoU exceeds 0.6. The regression targets are:
- centre shift measured in units of the proposal's own width and height, so the target does not depend on the box's size
- log scale change; at decoding keeps widths positive and makes ×2 and ×½ symmetric
Each target is a ridge regression on the pool5 feature ( values), and the box is decoded by inverting the equations:
- one of ; one regressor per target per class
- ridge penalty, set to 1000 in the paper
- the prediction
This parameterisation outlived R-CNN. Fast R-CNN, Faster R-CNN, SSD, RetinaNet and Mask R-CNN all regress the same four values, with anchors or proposals as ; the IoU-only-near-boxes rule survives as the IoU threshold for positive anchors.
Results (PASCAL VOC, mAP at IoU 0.5)
- VOC 2007 test, AlexNet
- 54.2% fine-tuned (fc7 features); 58.5% with box regression; fine-tuning added 8.0 points
- VOC 2007 test, VGG-16
- 66.0% with box regression (the paper's "O-Net"), at about 7× the forward-pass time
- VOC 2010 / 2012 test
- 53.7% / 53.3% with box regression
- Earlier baseline
- DPM v5 (HOG): 33.7% on VOC 2007 test
- Speed
- proposals + features: 13 s per image on a GPU, 53 s on a CPU
Worked example: encode, warp, decode
| Quantity | Computation | Value |
|---|---|---|
| IoU(P, G) | 120 × 90 = 10800 over 12960 + 12000 − 10800 = 14160 | 0.763 |
| t_x | (180 − 190) / 144 | −0.0694 |
| t_y | (110 − 105) / 90 | 0.0556 |
| t_w | ln(120 / 144) | −0.1823 |
| t_h | ln(100 / 90) | 0.1054 |
| warp scale x, y | 195 / 144 and 195 / 90 (227 minus 2 × 16 px of context) | 1.354, 2.167 |
| decode with 0.7·t | 190 − 144 × 0.0486 = 183.0; 144 × e^(−0.1276) = 126.75; … | (183.0, 108.5, 126.75, 96.89) |
| IoU after | against G | 0.919 |
Doubling every coordinate of both boxes leaves all four targets unchanged, which is why one regressor can serve objects of every size. The warp, by contrast, is anisotropic: this crop is stretched 1.6 times more vertically than horizontally, so the CNN sees a squashed dog. R-CNN accepted that distortion; later detectors pool features instead of warping pixels.
1import numpy as np23def encode(P, G):4 """R-CNN box-regression targets. Boxes are (cx, cy, w, h), shape (N, 4)."""5 return np.stack([(G[:, 0] - P[:, 0]) / P[:, 2],6 (G[:, 1] - P[:, 1]) / P[:, 3],7 np.log(G[:, 2] / P[:, 2]),8 np.log(G[:, 3] / P[:, 3])], axis=1)910def decode(P, d):11 return np.stack([P[:, 0] + P[:, 2] * d[:, 0],12 P[:, 1] + P[:, 3] * d[:, 1],13 P[:, 2] * np.exp(d[:, 2]),14 P[:, 3] * np.exp(d[:, 3])], axis=1)1516def fit_ridge(phi, T, lam=1000.0):17 """One class-specific regressor: pool5 features phi (N, 9216) -> targets T (N, 4)."""18 A = phi.T @ phi + lam * np.eye(phi.shape[1])19 return np.linalg.solve(A, phi.T @ T) # (9216, 4)2021P = np.array([[190., 105., 144., 90.]])22G = np.array([[180., 110., 120., 100.]])23t = encode(P, G)24print(t.round(4)) # [[-0.0694 0.0556 -0.1823 0.1054]]25print(decode(P, t)) # [[180. 110. 120. 100.]]26print(np.allclose(encode(2 * P, 2 * G), t)) # True: the targets are scale-invariantCheck yourself
A proposal has IoU 0.45 with a dog box. How is it labelled in R-CNN's three training stages?
Check yourself
Why are the width and height targets log ratios rather than differences G_w − P_w?
Remember
- ✓R-CNN = selective search (~2000 boxes) → warp to 227×227 with 16 px context → fine-tuned AlexNet fc7 (4096-d) → per-class linear SVMs → NMS → box regression.
- ✓Labels: fine-tuning IoU ≥ 0.5; SVM positives = ground truth only, negatives IoU < 0.3; regression on proposals with IoU > 0.6.
- ✓Box targets t_x = (G_x − P_x)/P_w, t_y = (G_y − P_y)/P_h, t_w = log(G_w/P_w), t_h = log(G_h/P_h), reused by every later R-CNN, SSD and RetinaNet.
- ✓VOC 2007: 54.2% mAP, 58.5% with box regression, 66.0% with VGG-16; VOC 2012: 53.3%.
- ✓One CNN pass per proposal (13 s/image on a GPU) and a four-stage training pipeline motivated Fast R-CNN.