Definition
Fast R-CNN is a two-stage object detector (Ross Girshick, "Fast R-CNN", ICCV 2015). It takes an image and a set of region proposals, computes one convolutional feature map for the whole image, extracts a fixed-size feature for each proposal with an RoI pooling layer, and feeds it to fully connected layers with two sibling outputs: a softmax over classes and class-specific box offsets. The whole network after the proposals is trained end to end with one multi-task loss.
R-CNN warps each of about 2,000 selective-search proposals to 227×227 and runs a full CNN on every one, so overlapping regions are convolved again and again. It then trains three things separately: the fine-tuned CNN, per-class SVMs and box regressors, and it writes features to disk in between. With VGG16 this took 84 hours of training and 47 s per test image.
SPPnet (He et al., 2014) already shared the convolution: one feature map per image and spatial-pyramid pooling per region. But its training was still multi-stage, and it did not fine-tune the layers below the pooling layer. Fast R-CNN keeps the sharing and makes training a single stage that updates every layer.
One pass, then a cheap head per RoI
The image is resized so its shorter side is 600 px (longer side capped at 1,000) and passed once through the VGG16 conv layers. conv5_3 has stride 16, so a 600×1000 image gives a 38×63×512 map. Each proposal is divided by 16 and rounded to feature-map cells.
RoI pooling turns an RoI of any size cells into a fixed grid (7×7 for VGG16). It splits the RoI into sub-windows of about cells and max-pools each one, channel by channel. It is the one-level special case of SPPnet's pyramid. Gradients flow back through it to the argmax cell of each bin, so the conv layers can be fine-tuned.
The multi-task loss
Each RoI is labelled with a class and, if it is foreground, a ground-truth box target . The network outputs class probabilities and, for every class , offsets that use R-CNN's parameterisation: scale-invariant shifts of the centre and log-space changes of width and height relative to the proposal.
- true class of the RoI; is background, and the Iverson bracket switches the box loss off for it
- regression target , normalised to zero mean and unit variance
- balance between the two terms; 1 in all experiments
Smooth L1 behaves like L2 near zero and like L1 for large errors, so its gradient is capped at 1. R-CNN and SPPnet used L2 for regression, which needed careful learning-rate tuning to avoid exploding gradients from badly placed proposals.
Training recipe
- 1Sample N = 2 images per mini-batch and R = 128 RoIs (64 per image). RoIs from the same image share the forward and backward pass of the conv trunk, which is about 64× cheaper than drawing 128 RoIs from 128 images.
- 225% of RoIs are foreground: proposals with IoU ≥ 0.5 to a ground-truth box. The rest are background, with maximum IoU in [0.1, 0.5).
- 3Flip images horizontally with probability 0.5. No other augmentation.
- 4Fine-tune VGG16 from conv3_1 upward. The first layers are generic, and freezing them saves memory and time.
- 5At test time, score all proposals, decode each class's box, and run per-class non-maximum suppression.
Reported results (VGG16, PASCAL VOC 2007 test)
- mAP
- 66.9% trained on VOC07 trainval, vs R-CNN 66.0% and SPPnet 63.1%. 70.0% with 07+12 training data.
- Training time
- 9.5 h vs 84 h for R-CNN (8.8×) and 25 h for SPPnet
- Test time per image
- 0.32 s, or 0.22 s with truncated SVD, vs 47 s for R-CNN (213× with SVD) and 2.3 s for SPPnet. Proposal time is not included.
- Truncated SVD
- cuts detection time by more than 30% for a 0.3-point mAP drop, with no extra fine-tuning
- Fine-tuning conv layers
- training only the fc layers, as SPPnet did, drops VGG16 from 66.9% to 61.4% mAP
Worked example: one foreground RoI
Proposal : centre (100, 80), 60×40 px. Ground truth : centre (110, 76), 72×36 px. Their IoU is 1904 / 3088 = 0.617, which is at least 0.5, so the RoI is foreground. Suppose the box head predicts . The target normalisation is skipped here.
| Coordinate | Target v | Prediction tᵘ | Error | smooth L1 | 0.5 · error² (L2) |
|---|---|---|---|---|---|
| x | 10 / 60 = 0.1667 | 0.30 | 0.1333 | 0.0089 | 0.0089 |
| y | −4 / 40 = −0.1000 | −0.05 | 0.0500 | 0.0013 | 0.0013 |
| w | ln(72/60) = 0.1823 | 0.10 | −0.0823 | 0.0034 | 0.0034 |
| h | ln(36/40) = −0.1054 | 1.60 | 1.7054 | 1.2054 | 1.4541 |
| Sum | 1.2189 | 1.4677 |
The three small errors cost the same under both losses. The outlier costs 1.21 instead of 1.45, and more importantly its gradient is 1 instead of 1.71. Under L2 the gradient grows with the error, so one bad box can dominate a mini-batch.
Truncated SVD for the fc layers
At test time there are thousands of RoIs per image but only one conv pass, so nearly half the forward time goes to fc6 and fc7. Fast R-CNN factorises each weight matrix , keeping the top singular values, and replaces one layer by two with no non-linearity between them. A matrix with parameters becomes parameters. For fc6 (25,088×4,096, 102.8M parameters) leaves 29.9M. For fc7 (4,096×4,096, 16.8M) leaves 2.1M.
1import torch2from torchvision.ops import roi_pool, roi_align34feat = torch.randn(1, 512, 38, 63) # conv5_3 of a 600x1000 image5boxes = torch.tensor([[0, 76., 44., 226., 164.], # (batch index, x1, y1, x2, y2) in image pixels6 [0, 300., 120., 520., 400.]])7pooled = roi_pool(feat, boxes, output_size=(7, 7), spatial_scale=1 / 16)8print(pooled.shape) # torch.Size([2, 512, 7, 7]) -> flatten to 25,088 for fc6910# Mask R-CNN's replacement: no rounding, bilinear sampling (2x2 points per bin)11aligned = roi_align(feat, boxes, output_size=(7, 7), spatial_scale=1 / 16,12 sampling_ratio=2, aligned=True)Check yourself
An RoI covers 10 × 8 cells of the conv5 map and is pooled to 7 × 7. What does the fc6 layer receive for it, with VGG16?
Check yourself
Why is the box term multiplied by [u ≥ 1]?
Remember
- ✓One conv pass per image; proposals are projected onto the stride-16 feature map instead of being cropped from the image.
- ✓RoI pooling max-pools any RoI to a fixed 7 × 7 × 512 grid. Rounding to cells moves the box by up to about a cell, which RoIAlign later fixes.
- ✓Multi-task loss: with , trained in one stage that also fine-tunes the conv layers.
- ✓Mini-batches of 2 images × 64 RoIs, 25% foreground (IoU ≥ 0.5).
- ✓VGG16 on VOC07: 66.9% mAP, 9.5 h training, 0.22 s per image with truncated SVD. Proposals (about 2 s each image) are now the bottleneck.