# Fast R-CNN

> Fast R-CNN runs the CNN once per image, max-pools each proposal to a 7×7 grid with RoI pooling, and trains classification and box regression jointly with softmax + smooth L1: 9× faster training and 213× faster testing than R-CNN.

Source: https://www.mlfoundry.in/learn/fast-r-cnn  
Section: Deep Learning › Deep Learning for Computer Vision  
Level: core · 7 min read · updated 2026-10-15

**Fast R-CNN:** is a two-stage object detector (Ross Girshick, "Fast R-CNN", ICCV 2015). It takes an image and a set of region proposals, computes one convolutional feature map for the whole image, extracts a fixed-size feature for each proposal with an **RoI pooling** layer, and feeds it to fully connected layers with two sibling outputs: a softmax over $K+1$ classes and $4K$ class-specific box offsets. The whole network after the proposals is trained end to end with one multi-task loss.

**R-CNN** warps each of about 2,000 selective-search proposals to 227×227 and runs a full CNN on every one, so overlapping regions are convolved again and again. It then trains three things separately: the fine-tuned CNN, per-class SVMs and box regressors, and it writes features to disk in between. With VGG16 this took 84 hours of training and 47 s per test image.

SPPnet (He et al., 2014) already shared the convolution: one feature map per image and spatial-pyramid pooling per region. But its training was still multi-stage, and it did not fine-tune the layers below the pooling layer. Fast R-CNN keeps the sharing and makes training a single stage that updates every layer.

## One pass, then a cheap head per RoI

The image is resized so its shorter side is 600 px (longer side capped at 1,000) and passed once through the VGG16 conv layers. conv5_3 has stride 16, so a 600×1000 image gives a 38×63×512 map. Each proposal $(x_1,y_1,x_2,y_2)$ is divided by 16 and rounded to feature-map cells.

> Figure: Fast R-CNN at test time. The conv trunk runs once; the ~2,000 selective-search proposals are only projected onto its stride-16 map. Each RoI is max-pooled to 7 × 7 × 512 = 25,088 numbers, the input size VGG16's fc6 expects. fc6 and fc7 (4,096 each) feed two sibling layers: 21 class scores for PASCAL VOC (20 classes + background) and 84 = 4 × 21 box offsets.

**RoI pooling** turns an RoI of any size $h\times w$ cells into a fixed $H\times W$ grid (7×7 for VGG16). It splits the RoI into sub-windows of about $h/H\times w/W$ cells and max-pools each one, channel by channel. It is the one-level special case of SPPnet's pyramid. Gradients flow back through it to the argmax cell of each bin, so the conv layers can be fine-tuned.

## The multi-task loss

Each RoI is labelled with a class $u$ and, if it is foreground, a ground-truth box target $v$. The network outputs class probabilities $p$ and, for every class $k$, offsets $t^k$ that use **R-CNN**'s parameterisation: scale-invariant shifts of the centre and log-space changes of width and height relative to the proposal.

$$
L(p,u,t^u,v)=-\log p_u+\lambda\,[u\ge 1]\sum_{i\in\{x,y,w,h\}}\operatorname{smooth}_{L_1}\!\big(t^u_i-v_i\big),\qquad \operatorname{smooth}_{L_1}(x)=\begin{cases}0.5x^2 & |x|<1\\ |x|-0.5 & \text{otherwise}\end{cases}
$$

- $u$: true class of the RoI; $u=0$ is background, and the Iverson bracket $[u\ge1]$ switches the box loss off for it
- $v$: regression target $\big(\tfrac{G_x-P_x}{P_w},\tfrac{G_y-P_y}{P_h},\log\tfrac{G_w}{P_w},\log\tfrac{G_h}{P_h}\big)$, normalised to zero mean and unit variance
- $\lambda$: balance between the two terms; 1 in all experiments

Smooth L1 behaves like L2 near zero and like L1 for large errors, so its gradient is capped at 1. R-CNN and SPPnet used L2 for regression, which needed careful learning-rate tuning to avoid exploding gradients from badly placed proposals.

**Training recipe**

1. Sample N = 2 images per mini-batch and R = 128 RoIs (64 per image). RoIs from the same image share the forward and backward pass of the conv trunk, which is about 64× cheaper than drawing 128 RoIs from 128 images.
2. 25% of RoIs are foreground: proposals with IoU ≥ 0.5 to a ground-truth box. The rest are background, with maximum IoU in [0.1, 0.5).
3. Flip images horizontally with probability 0.5. No other augmentation.
4. Fine-tune VGG16 from conv3_1 upward. The first layers are generic, and freezing them saves memory and time.
5. At test time, score all proposals, decode each class's box, and run per-class non-maximum suppression.

**Reported results (VGG16, PASCAL VOC 2007 test)**

- **mAP:** 66.9% trained on VOC07 trainval, vs R-CNN 66.0% and SPPnet 63.1%. 70.0% with 07+12 training data.
- **Training time:** 9.5 h vs 84 h for R-CNN (8.8×) and 25 h for SPPnet
- **Test time per image:** 0.32 s, or 0.22 s with truncated SVD, vs 47 s for R-CNN (213× with SVD) and 2.3 s for SPPnet. Proposal time is not included.
- **Truncated SVD:** cuts detection time by more than 30% for a 0.3-point mAP drop, with no extra fine-tuning
- **Fine-tuning conv layers:** training only the fc layers, as SPPnet did, drops VGG16 from 66.9% to 61.4% mAP

> Figure: RoI pooling with the integer rules of the original Caffe layer. The default proposal (76, 44, 226, 164) px becomes cells x 5–14, y 3–10 after ÷16 and rounding, so the pooled box is (80, 48, 240, 176): its edges have moved by +4, +4, +14 and +12 px. Ten columns into 7 bins is 1.43 cells per bin, so floor/ceil gives bins 2 or 3 cells wide, and neighbouring bins overlap. Shrink the RoI to 32 × 32 px at (100, 60): it covers only 3 × 3 cells, the 49 outputs take just 9 distinct values, and one edge moves by 20 px. Mask R-CNN's RoIAlign removes exactly this rounding.

## Worked example: one foreground RoI

Proposal $P$: centre (100, 80), 60×40 px. Ground truth $G$: centre (110, 76), 72×36 px. Their IoU is 1904 / 3088 = 0.617, which is at least 0.5, so the RoI is foreground. Suppose the box head predicts $t^u=(0.30,-0.05,0.10,1.60)$. The target normalisation is skipped here.

Box target, error and loss per coordinate (computed in Python)

| Coordinate | Target v | Prediction tᵘ | Error | smooth L1 | 0.5 · error² (L2) |
| --- | --- | --- | --- | --- | --- |
| x | 10 / 60 = 0.1667 | 0.30 | 0.1333 | 0.0089 | 0.0089 |
| y | −4 / 40 = −0.1000 | −0.05 | 0.0500 | 0.0013 | 0.0013 |
| w | ln(72/60) = 0.1823 | 0.10 | −0.0823 | 0.0034 | 0.0034 |
| h | ln(36/40) = −0.1054 | 1.60 | 1.7054 | 1.2054 | 1.4541 |
| Sum |  |  |  | 1.2189 | 1.4677 |

The three small errors cost the same under both losses. The outlier $h$ costs 1.21 instead of 1.45, and more importantly its gradient is 1 instead of 1.71. Under L2 the gradient grows with the error, so one bad box can dominate a mini-batch.

## Truncated SVD for the fc layers

At test time there are thousands of RoIs per image but only one conv pass, so nearly half the forward time goes to fc6 and fc7. Fast R-CNN factorises each weight matrix $W\approx U\Sigma_tV^\top$, keeping the top $t$ singular values, and replaces one layer by two with no non-linearity between them. A $u\times v$ matrix with $uv$ parameters becomes $t(u+v)$ parameters. For fc6 (25,088×4,096, 102.8M parameters) $t=1024$ leaves 29.9M. For fc7 (4,096×4,096, 16.8M) $t=256$ leaves 2.1M.

```python
import torch
from torchvision.ops import roi_pool, roi_align

feat = torch.randn(1, 512, 38, 63)                # conv5_3 of a 600x1000 image
boxes = torch.tensor([[0, 76., 44., 226., 164.],  # (batch index, x1, y1, x2, y2) in image pixels
                      [0, 300., 120., 520., 400.]])
pooled = roi_pool(feat, boxes, output_size=(7, 7), spatial_scale=1 / 16)
print(pooled.shape)                                # torch.Size([2, 512, 7, 7]) -> flatten to 25,088 for fc6

# Mask R-CNN's replacement: no rounding, bilinear sampling (2x2 points per bin)
aligned = roi_align(feat, boxes, output_size=(7, 7), spatial_scale=1 / 16,
                    sampling_ratio=2, aligned=True)
```

RoI pooling vs RoIAlign on a stride-16 map with torchvision.ops (not run here)

> **Why sampling 2 images per batch works:** R-CNN and SPPnet drew each training RoI from a different image, so every RoI paid for its own large receptive field. Fast R-CNN samples many RoIs from few images. The RoIs within an image are correlated, but in practice training still converged in fewer SGD iterations than R-CNN.

> **Fast at test time, but not end to end:** The 0.32 s per image excludes the proposals. Selective search takes about 2 s per image on a CPU, so it becomes the bottleneck. **Faster R-CNN** removes it by predicting proposals from the same feature map with a Region Proposal Network.

**Question:** An RoI covers 10 × 8 cells of the conv5 map and is pooled to 7 × 7. What does the fc6 layer receive for it, with VGG16?

- A. A 10 × 8 × 512 tensor, so fc6 must handle variable sizes
- B. 7 × 7 × 512 = 25,088 numbers, the same as for every other RoI
- C. A 227 × 227 warped image crop
- D. 7 × 7 = 49 numbers, one per bin

**Answer:** B. 7 × 7 × 512 = 25,088 numbers, the same as for every other RoI — RoI pooling max-pools each of the 512 channels separately into a fixed 7 × 7 grid, whatever the RoI size. That fixed 25,088-d vector is exactly what VGG16's pre-trained fc6 expects.

**Question:** Why is the box term multiplied by [u ≥ 1]?

- A. Background RoIs have no ground-truth box to regress to
- B. Smooth L1 is undefined for negative classes
- C. It doubles the weight of foreground RoIs
- D. So that λ can be set to zero

**Answer:** A. Background RoIs have no ground-truth box to regress to — A background RoI (u = 0) does not correspond to any object, so there is no target v. Only foreground RoIs train the regressor, and only the output tᵘ for their true class gets a loss.

**Key points**

- One conv pass per image; proposals are projected onto the stride-16 feature map instead of being cropped from the image.
- RoI pooling max-pools any RoI to a fixed 7 × 7 × 512 grid. Rounding to cells moves the box by up to about a cell, which RoIAlign later fixes.
- Multi-task loss: $-\log p_u+\lambda[u\ge1]\sum_i\operatorname{smooth}_{L_1}(t^u_i-v_i)$ with $\lambda=1$, trained in one stage that also fine-tunes the conv layers.
- Mini-batches of 2 images × 64 RoIs, 25% foreground (IoU ≥ 0.5).
- VGG16 on VOC07: 66.9% mAP, 9.5 h training, 0.22 s per image with truncated SVD. Proposals (about 2 s each image) are now the bottleneck.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
