R-CNN

R-CNN (Girshick et al., 2014): ~2000 selective-search proposals, each warped to 227×227 and run through a fine-tuned AlexNet, scored by per-class SVMs and refined by box regression. 53.3% mAP on VOC 2012, at 13 s per image.

7 min read764 wordscore0 viewsupdated 2026-10-15

Definition

R-CNN ("Regions with CNN features") is the detector of Girshick, Donahue, Darrell and Malik, "Rich feature hierarchies for accurate object detection and semantic segmentation" (CVPR 2014). It was the first to show that CNN features pre-trained on ImageNet classification, then fine-tuned, beat hand-built features for detection by a wide margin: 53.3% mAP on PASCAL VOC 2012, over 30% relative better than the previous best.

Detection needs a class and a box for every object (see Object detection). In 2013 the strongest systems were deformable part models on HOG features and selective-search pipelines on bag-of-words SIFT features, around 33 to 35% mAP on VOC. Sliding a CNN over every position and scale was too expensive. R-CNN's idea was to run the CNN only on a few thousand likely regions and treat detection as classification of regions.

The pipeline

  1. 1Propose. Selective search (Uijlings et al., IJCV 2013) in "fast mode" merges superpixels hierarchically and returns about 2,000 class-agnostic boxes per image.
  2. 2Warp. Each box is dilated so that, after warping, 16 pixels of image context surround it, then resized to 227×227227\times227 regardless of its aspect ratio.
  3. 3Extract. An AlexNet, pre-trained on ILSVRC-2012 and fine-tuned on warped regions, maps each crop to a 4096-dimensional fc7 feature.
  4. 4Score. One linear SVM per class scores every region: a 2000×40962000\times4096 feature matrix times a 4096×N4096\times N weight matrix.
  5. 5Suppress. Per class, greedy NMS rejects a region whose IoU with a higher-scoring kept region exceeds a learned threshold.
  6. 6Refine. A class-specific linear regressor on pool5 features predicts a correction to each box.
R-CNN at test time: one CNN forward pass per region, ~2000 per image1 selective search~2000 class-agnostic boxes2 warp each227×227, 16 pxcontext, any aspect3 CNN (AlexNet)5 conv → pool56×6×256 = 9216fc6 → fc7: 4096-dper regionfeaturesX: 2000 × 4096scores = X · W_svmW_svm: 4096 × 20→ 2000 × 204 per-class SVMslinear, one per class(20 for VOC)score every region5 greedy NMSper class, on theSVM scoresdrop IoU > learned τ6 box regressionpool5 → (tx, ty, tw, th)shift centre, rescalew, h → detectionstraining: four separate stages1 pre-trainAlexNet on ILSVRC-2012 classification (1000-way)2 fine-tune21-way softmax on warped regions: IoU ≥ 0.5 = object, else background3 SVMsper class on fc7: gt boxes = positive, IoU < 0.3 = negative, hard negatives4 box reg.ridge regression from pool5 to (tx, ty, tw, th), on proposals with IoU > 0.6fc7 features of every training region are cached on disk; test: proposals + features 13 s / image on a GPU
fig.R-CNN's test-time pipeline (top) and its four separate training stages (bottom). The CNN runs once per proposal, about 2,000 times per image, and the SVMs, regressors and CNN are trained separately, with fc7 features cached to disk.

Training: three objectives, three label rules

Fine-tuning replaces AlexNet's 1000-way classifier with a 21-way one (20 VOC classes + background) and runs SGD at learning rate 0.001 on warped proposals. A proposal with IoU ≥0.5\ge 0.5 to a ground-truth box is a positive for that class; all others are background. Each mini-batch of 128 has 32 positives and 96 background windows, because positives are rare.

SVMs use stricter labels: only the ground-truth boxes are positives, proposals with IoU <0.3<0.3 are negatives (0.3 was chosen by grid search over 0 to 0.5), and everything in between is ignored. Standard hard-negative mining keeps the negative set manageable. The paper's appendix reports that using the fine-tuned softmax directly instead of SVMs drops VOC 2007 mAP from 54.2% to 50.9%; Fast R-CNN later removed the SVMs once training was end to end.

Bounding-box regression

A proposal P=(Px,Py,Pw,Ph)P=(P_x,P_y,P_w,P_h) (centre, width, height) is assigned to the ground-truth box GG it overlaps most, and used for training only if that IoU exceeds 0.6. The regression targets are:

tx=Gx−PxPw,ty=Gy−PyPh,tw=log⁡GwPw,th=log⁡GhPht_x=\frac{G_x-P_x}{P_w},\qquad t_y=\frac{G_y-P_y}{P_h},\qquad t_w=\log\frac{G_w}{P_w},\qquad t_h=\log\frac{G_h}{P_h}
tx, tyt_x,\ t_y
centre shift measured in units of the proposal's own width and height, so the target does not depend on the box's size
tw, tht_w,\ t_h
log scale change; exp⁡\exp at decoding keeps widths positive and makes ×2 and ×½ symmetric

Each target t⋆t_\star is a ridge regression on the pool5 feature ϕ5(P)\phi_5(P) (6×6×256=92166\times6\times256=9216 values), and the box is decoded by inverting the equations:

w⋆=arg⁡min⁡w^∑i(t⋆i−w^⊤ϕ5(Pi))2+λ∥w^∥2,G^x=Px+Pw dx(P),G^w=Pw edw(P)w_\star=\arg\min_{\hat w}\sum_i\big(t_\star^i-\hat w^{\top}\phi_5(P^i)\big)^2+\lambda\lVert\hat w\rVert^2,\qquad \hat G_x=P_x+P_w\,d_x(P),\quad \hat G_w=P_w\,e^{d_w(P)}
⋆\star
one of x,y,w,hx,y,w,h; one regressor per target per class
λ\lambda
ridge penalty, set to 1000 in the paper
d⋆(P)d_\star(P)
the prediction w⋆⊤ϕ5(P)w_\star^{\top}\phi_5(P)

This parameterisation outlived R-CNN. Fast R-CNN, Faster R-CNN, SSD, RetinaNet and Mask R-CNN all regress the same four tt values, with anchors or proposals as PP; the IoU-only-near-boxes rule survives as the IoU threshold for positive anchors.

Results (PASCAL VOC, mAP at IoU 0.5)

VOC 2007 test, AlexNet
54.2% fine-tuned (fc7 features); 58.5% with box regression; fine-tuning added 8.0 points
VOC 2007 test, VGG-16
66.0% with box regression (the paper's "O-Net"), at about 7× the forward-pass time
VOC 2010 / 2012 test
53.7% / 53.3% with box regression
Earlier baseline
DPM v5 (HOG): 33.7% on VOC 2007 test
Speed
proposals + features: 13 s per image on a GPU, 53 s on a CPU
proposal P vs ground truth G: labels, regression targets and the 227 × 227 warpG (dog) · P · dashed blue = crop with context · regressedP = (190, 105, w 144, h 90) · G = (180, 110, w 120, h 100)CNN input: warp to 227 × 227P → 195²+16 px eachside×1.354 x×2.167 yaspect ×1.60→ AlexNet → fc7: 4096-d featureIoU(P, G)0.763fine-tuning (21-way)positive (dog)SVM for dogignored (0.3 ≤ IoU, not gt)box regressiontraining pair (IoU > 0.6)t = (tx, ty, tw, th)(−0.069, +0.056, −0.182, +0.105)regressed, 0.7·tIoU 0.763 → 0.919regressed = decode(P, 0.7·t), an illustrative regressor: x = P_x + P_w·d_x, w = P_w·exp(d_w)
preset
fig.Move the proposal P around the ground-truth box G. At the default P = (190, 105, 144 × 90) the IoU is 0.763: a fine-tuning positive, ignored by the SVM (not a ground-truth box) and a box-regression training pair. The targets are t = (−0.069, +0.056, −0.182, +0.105); applying 0.7·t (an illustrative regressor) lifts the IoU to 0.919. The warp stretches P by ×1.354 horizontally and ×2.167 vertically. "Shifted" (IoU 0.468) is background for fine-tuning but not an SVM negative; "background" (IoU 0.107) is a negative everywhere.

Worked example: encode, warp, decode

QuantityComputationValue
IoU(P, G)120 × 90 = 10800 over 12960 + 12000 − 10800 = 141600.763
t_x(180 − 190) / 144−0.0694
t_y(110 − 105) / 900.0556
t_wln(120 / 144)−0.1823
t_hln(100 / 90)0.1054
warp scale x, y195 / 144 and 195 / 90 (227 minus 2 × 16 px of context)1.354, 2.167
decode with 0.7·t190 − 144 × 0.0486 = 183.0; 144 × e^(−0.1276) = 126.75; …(183.0, 108.5, 126.75, 96.89)
IoU afteragainst G0.919
P = (190, 105, 144, 90) and G = (180, 110, 120, 100) as (centre x, centre y, w, h)

Doubling every coordinate of both boxes leaves all four targets unchanged, which is why one regressor can serve objects of every size. The warp, by contrast, is anisotropic: this crop is stretched 1.6 times more vertically than horizontally, so the CNN sees a squashed dog. R-CNN accepted that distortion; later detectors pool features instead of warping pixels.

python
1import numpy as np
2
3def encode(P, G):
4 """R-CNN box-regression targets. Boxes are (cx, cy, w, h), shape (N, 4)."""
5 return np.stack([(G[:, 0] - P[:, 0]) / P[:, 2],
6 (G[:, 1] - P[:, 1]) / P[:, 3],
7 np.log(G[:, 2] / P[:, 2]),
8 np.log(G[:, 3] / P[:, 3])], axis=1)
9
10def decode(P, d):
11 return np.stack([P[:, 0] + P[:, 2] * d[:, 0],
12 P[:, 1] + P[:, 3] * d[:, 1],
13 P[:, 2] * np.exp(d[:, 2]),
14 P[:, 3] * np.exp(d[:, 3])], axis=1)
15
16def fit_ridge(phi, T, lam=1000.0):
17 """One class-specific regressor: pool5 features phi (N, 9216) -> targets T (N, 4)."""
18 A = phi.T @ phi + lam * np.eye(phi.shape[1])
19 return np.linalg.solve(A, phi.T @ T) # (9216, 4)
20
21P = np.array([[190., 105., 144., 90.]])
22G = np.array([[180., 110., 120., 100.]])
23t = encode(P, G)
24print(t.round(4)) # [[-0.0694 0.0556 -0.1823 0.1054]]
25print(decode(P, t)) # [[180. 110. 120. 100.]]
26print(np.allclose(encode(2 * P, 2 * G), t)) # True: the targets are scale-invariant
Box encoding, decoding and the ridge regressor in NumPy (runs as is). Proposals themselves can come from OpenCV's selective search in opencv-contrib-python (cv2.ximgproc.segmentation, fast mode), which returns (x, y, w, h) boxes; that part is not shown or run here.

Check yourself

A proposal has IoU 0.45 with a dog box. How is it labelled in R-CNN's three training stages?

Check yourself

Why are the width and height targets log ratios rather than differences G_w − P_w?

Remember

  • ✓R-CNN = selective search (~2000 boxes) → warp to 227×227 with 16 px context → fine-tuned AlexNet fc7 (4096-d) → per-class linear SVMs → NMS → box regression.
  • ✓Labels: fine-tuning IoU ≥ 0.5; SVM positives = ground truth only, negatives IoU < 0.3; regression on proposals with IoU > 0.6.
  • ✓Box targets t_x = (G_x − P_x)/P_w, t_y = (G_y − P_y)/P_h, t_w = log(G_w/P_w), t_h = log(G_h/P_h), reused by every later R-CNN, SSD and RetinaNet.
  • ✓VOC 2007: 54.2% mAP, 58.5% with box regression, 66.0% with VGG-16; VOC 2012: 53.3%.
  • ✓One CNN pass per proposal (13 s/image on a GPU) and a four-stage training pipeline motivated Fast R-CNN.
← back to Deep Learning roadmap