# R-CNN

> R-CNN (Girshick et al., 2014): ~2000 selective-search proposals, each warped to 227×227 and run through a fine-tuned AlexNet, scored by per-class SVMs and refined by box regression. 53.3% mAP on VOC 2012, at 13 s per image.

Source: https://www.mlfoundry.in/learn/r-cnn  
Section: Deep Learning › Deep Learning for Computer Vision  
Level: core · 7 min read · updated 2026-10-15

**R-CNN:** ("Regions with CNN features") is the detector of Girshick, Donahue, Darrell and Malik, "Rich feature hierarchies for accurate object detection and semantic segmentation" (CVPR 2014). It was the first to show that CNN features pre-trained on ImageNet classification, then fine-tuned, beat hand-built features for detection by a wide margin: 53.3% mAP on PASCAL VOC 2012, over 30% relative better than the previous best.

Detection needs a class and a box for every object (see **Object detection**). In 2013 the strongest systems were deformable part models on HOG features and selective-search pipelines on bag-of-words SIFT features, around 33 to 35% mAP on VOC. Sliding a CNN over every position and scale was too expensive. R-CNN's idea was to run the CNN only on a few thousand likely regions and treat detection as **classification of regions**.

## The pipeline

1. **Propose.** Selective search (Uijlings et al., IJCV 2013) in "fast mode" merges superpixels hierarchically and returns about 2,000 class-agnostic boxes per image.
2. **Warp.** Each box is dilated so that, after warping, 16 pixels of image context surround it, then resized to $227\times227$ regardless of its aspect ratio.
3. **Extract.** An **AlexNet**, pre-trained on ILSVRC-2012 and fine-tuned on warped regions, maps each crop to a 4096-dimensional fc7 feature.
4. **Score.** One linear SVM per class scores every region: a $2000\times4096$ feature matrix times a $4096\times N$ weight matrix.
5. **Suppress.** Per class, greedy NMS rejects a region whose IoU with a higher-scoring kept region exceeds a learned threshold.
6. **Refine.** A class-specific linear regressor on pool5 features predicts a correction to each box.

> Figure: R-CNN's test-time pipeline (top) and its four separate training stages (bottom). The CNN runs once per proposal, about 2,000 times per image, and the SVMs, regressors and CNN are trained separately, with fc7 features cached to disk.

## Training: three objectives, three label rules

**Fine-tuning** replaces AlexNet's 1000-way classifier with a 21-way one (20 VOC classes + background) and runs SGD at learning rate 0.001 on warped proposals. A proposal with IoU $\ge 0.5$ to a ground-truth box is a positive for that class; all others are background. Each mini-batch of 128 has 32 positives and 96 background windows, because positives are rare.

**SVMs** use stricter labels: only the ground-truth boxes are positives, proposals with IoU $<0.3$ are negatives (0.3 was chosen by grid search over 0 to 0.5), and everything in between is ignored. Standard hard-negative mining keeps the negative set manageable. The paper's appendix reports that using the fine-tuned softmax directly instead of SVMs drops VOC 2007 mAP from 54.2% to 50.9%; Fast R-CNN later removed the SVMs once training was end to end.

## Bounding-box regression

A proposal $P=(P_x,P_y,P_w,P_h)$ (centre, width, height) is assigned to the ground-truth box $G$ it overlaps most, and used for training only if that IoU exceeds 0.6. The regression targets are:

$$
t_x=\frac{G_x-P_x}{P_w},\qquad t_y=\frac{G_y-P_y}{P_h},\qquad t_w=\log\frac{G_w}{P_w},\qquad t_h=\log\frac{G_h}{P_h}
$$

- $t_x,\ t_y$: centre shift measured in units of the proposal's own width and height, so the target does not depend on the box's size
- $t_w,\ t_h$: log scale change; $\exp$ at decoding keeps widths positive and makes ×2 and ×½ symmetric

Each target $t_\star$ is a ridge regression on the pool5 feature $\phi_5(P)$ ($6\times6\times256=9216$ values), and the box is decoded by inverting the equations:

$$
w_\star=\arg\min_{\hat w}\sum_i\big(t_\star^i-\hat w^{\top}\phi_5(P^i)\big)^2+\lambda\lVert\hat w\rVert^2,\qquad \hat G_x=P_x+P_w\,d_x(P),\quad \hat G_w=P_w\,e^{d_w(P)}
$$

- $\star$: one of $x,y,w,h$; one regressor per target per class
- $\lambda$: ridge penalty, set to 1000 in the paper
- $d_\star(P)$: the prediction $w_\star^{\top}\phi_5(P)$

This parameterisation outlived R-CNN. **Fast R-CNN**, **Faster R-CNN**, **SSD**, **RetinaNet** and **Mask R-CNN** all regress the same four $t$ values, with anchors or proposals as $P$; the IoU-only-near-boxes rule survives as the IoU threshold for positive anchors.

**Results (PASCAL VOC, mAP at IoU 0.5)**

- **VOC 2007 test, AlexNet:** 54.2% fine-tuned (fc7 features); 58.5% with box regression; fine-tuning added 8.0 points
- **VOC 2007 test, VGG-16:** 66.0% with box regression (the paper's "O-Net"), at about 7× the forward-pass time
- **VOC 2010 / 2012 test:** 53.7% / 53.3% with box regression
- **Earlier baseline:** DPM v5 (HOG): 33.7% on VOC 2007 test
- **Speed:** proposals + features: 13 s per image on a GPU, 53 s on a CPU

> Figure: Move the proposal P around the ground-truth box G. At the default P = (190, 105, 144 × 90) the IoU is 0.763: a fine-tuning positive, ignored by the SVM (not a ground-truth box) and a box-regression training pair. The targets are t = (−0.069, +0.056, −0.182, +0.105); applying 0.7·t (an illustrative regressor) lifts the IoU to 0.919. The warp stretches P by ×1.354 horizontally and ×2.167 vertically. "Shifted" (IoU 0.468) is background for fine-tuning but not an SVM negative; "background" (IoU 0.107) is a negative everywhere.

## Worked example: encode, warp, decode

P = (190, 105, 144, 90) and G = (180, 110, 120, 100) as (centre x, centre y, w, h)

| Quantity | Computation | Value |
| --- | --- | --- |
| IoU(P, G) | 120 × 90 = 10800 over 12960 + 12000 − 10800 = 14160 | 0.763 |
| t_x | (180 − 190) / 144 | −0.0694 |
| t_y | (110 − 105) / 90 | 0.0556 |
| t_w | ln(120 / 144) | −0.1823 |
| t_h | ln(100 / 90) | 0.1054 |
| warp scale x, y | 195 / 144 and 195 / 90 (227 minus 2 × 16 px of context) | 1.354, 2.167 |
| decode with 0.7·t | 190 − 144 × 0.0486 = 183.0; 144 × e^(−0.1276) = 126.75; … | (183.0, 108.5, 126.75, 96.89) |
| IoU after | against G | 0.919 |

Doubling every coordinate of both boxes leaves all four targets unchanged, which is why one regressor can serve objects of every size. The warp, by contrast, is anisotropic: this crop is stretched 1.6 times more vertically than horizontally, so the CNN sees a squashed dog. R-CNN accepted that distortion; later detectors pool features instead of warping pixels.

```python
import numpy as np

def encode(P, G):
    """R-CNN box-regression targets. Boxes are (cx, cy, w, h), shape (N, 4)."""
    return np.stack([(G[:, 0] - P[:, 0]) / P[:, 2],
                     (G[:, 1] - P[:, 1]) / P[:, 3],
                     np.log(G[:, 2] / P[:, 2]),
                     np.log(G[:, 3] / P[:, 3])], axis=1)

def decode(P, d):
    return np.stack([P[:, 0] + P[:, 2] * d[:, 0],
                     P[:, 1] + P[:, 3] * d[:, 1],
                     P[:, 2] * np.exp(d[:, 2]),
                     P[:, 3] * np.exp(d[:, 3])], axis=1)

def fit_ridge(phi, T, lam=1000.0):
    """One class-specific regressor: pool5 features phi (N, 9216) -> targets T (N, 4)."""
    A = phi.T @ phi + lam * np.eye(phi.shape[1])
    return np.linalg.solve(A, phi.T @ T)              # (9216, 4)

P = np.array([[190., 105., 144., 90.]])
G = np.array([[180., 110., 120., 100.]])
t = encode(P, G)
print(t.round(4))                                     # [[-0.0694  0.0556 -0.1823  0.1054]]
print(decode(P, t))                                   # [[180. 110. 120. 100.]]
print(np.allclose(encode(2 * P, 2 * G), t))           # True: the targets are scale-invariant
```

Box encoding, decoding and the ridge regressor in NumPy (runs as is). Proposals themselves can come from OpenCV's selective search in opencv-contrib-python (cv2.ximgproc.segmentation, fast mode), which returns (x, y, w, h) boxes; that part is not shown or run here.

> **Why R-CNN was slow:** Each of the ~2,000 proposals gets its own CNN forward pass, although they overlap heavily and share most pixels. Training is a multi-stage pipeline (fine-tune, cache features, train SVMs, train regressors), and the cached features take hundreds of gigabytes. **Fast R-CNN** (Girshick, 2015) runs the CNN once per image, pools each region from the shared feature map, and trains classification and regression jointly; with VGG-16 it trained 9× and tested 213× faster than R-CNN.

> **Pre-train, then fine-tune:** R-CNN's lasting lesson is not the pipeline but the recipe: supervised pre-training on a large auxiliary dataset (ImageNet), followed by fine-tuning on a small target dataset (VOC 2007 has 5,011 trainval images). Fine-tuning alone added 8 points of mAP. The same recipe now underlies almost all of transfer learning in vision.

**Question:** A proposal has IoU 0.45 with a dog box. How is it labelled in R-CNN's three training stages?

- A. Positive for fine-tuning, negative for the SVM, used for regression
- B. Background for fine-tuning, ignored by the SVM, not used for regression
- C. Background for fine-tuning, negative for the SVM, used for regression
- D. Positive in all three

**Answer:** B. Background for fine-tuning, ignored by the SVM, not used for regression — Fine-tuning positives need IoU $\ge 0.5$, so 0.45 is background. SVM negatives need IoU $<0.3$ and positives must be ground-truth boxes, so 0.45 is ignored. Box regression only uses proposals with IoU $>0.6$.

**Question:** Why are the width and height targets log ratios rather than differences G_w − P_w?

- A. Logs are faster to compute
- B. A log ratio is scale-invariant and its exponential keeps decoded widths positive
- C. The SVM needs positive inputs
- D. It makes the IoU loss convex

**Answer:** B. A log ratio is scale-invariant and its exponential keeps decoded widths positive — $\log(G_w/P_w)$ is the same for a 20-pixel and a 200-pixel object with the same relative error, and decoding with $P_w e^{d_w}$ can never give a negative width. A raw difference would have to be learned separately for every object size.

**Key points**

- R-CNN = selective search (~2000 boxes) → warp to 227×227 with 16 px context → fine-tuned AlexNet fc7 (4096-d) → per-class linear SVMs → NMS → box regression.
- Labels: fine-tuning IoU ≥ 0.5; SVM positives = ground truth only, negatives IoU < 0.3; regression on proposals with IoU > 0.6.
- Box targets t_x = (G_x − P_x)/P_w, t_y = (G_y − P_y)/P_h, t_w = log(G_w/P_w), t_h = log(G_h/P_h), reused by every later R-CNN, SSD and RetinaNet.
- VOC 2007: 54.2% mAP, 58.5% with box regression, 66.0% with VGG-16; VOC 2012: 53.3%.
- One CNN pass per proposal (13 s/image on a GPU) and a four-stage training pipeline motivated Fast R-CNN.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
