Definition
SSD (Single Shot MultiBox Detector; Liu, Anguelov, Erhan, Szegedy, Reed, Fu and Berg, ECCV 2016) is a one-stage object detector. A convolutional network predicts, at every location of several feature maps, class scores and box offsets for a fixed set of default boxes (anchors) of different scales and aspect ratios. There is no region-proposal stage and no per-region feature resampling.
Input: an image resized to (SSD300) or (SSD512). Output: a list of boxes, each with a class and a score. Faster R-CNN reached high accuracy with two stages: a region proposal network, then RoI pooling and a classifier for each of about 300 proposals. That second stage made it run at about 7 FPS. YOLO (v1) ran at 45 FPS with one pass, but it predicted from a single coarse grid and lost accuracy, especially on small objects.
SSD keeps the single pass and adds two ideas. First, it predicts from several feature maps: high-resolution early maps for small objects, coarse late maps for large ones. Second, it uses anchor boxes with several aspect ratios at every location, as the RPN does, but here each anchor directly gets the final class scores.
Architecture: six prediction maps
The base is VGG-16 (see VGGNet) pre-trained on ImageNet. fc6 and fc7 become convolutions (fc6 is a dilated convolution, see Dilated convolution), pool5 becomes with stride 1, and extra convolution layers shrink the map further. Predictions come from conv4_3 (), fc7 (), conv8_2 (), conv9_2 (), conv10_2 () and conv11_2 (). conv4_3 has a much larger feature scale than the others, so it is L2-normalised with a learned per-channel scale.
Each map with default boxes per location gets one convolution with output channels: class scores (including background) and 4 offsets per box. For PASCAL VOC, . With on conv4_3, conv10_2 and conv11_2 and elsewhere, the total is boxes.
Default boxes
Map of gets a box scale (as a fraction of the image side) that grows linearly. Each location gets boxes with aspect ratios , plus a second square of scale . Maps with omit and . Box centres sit at the cell centres.
- 0.2 and 0.9 in the paper, so for
- aspect ratio ; the box area is the same for every ratio
- extra square box for
The released VOC model deviates slightly: conv4_3 gets scale 0.1, so the box sizes are 30, 60, 111, 162, 213 and 264 px, with extra squares of 42, 82, 134, 186, 237 and 288 px. The figure below uses these sizes and clips boxes to the image.
Matching and the loss
Each ground-truth box is matched to the default box with the highest IoU. Every other default box with IoU above 0.5 to some ground truth is also positive. The rest are negatives (background). An object therefore trains several neighbouring boxes, often on two adjacent maps.
- number of positive default boxes; the loss is 0 if
- softmax cross-entropy over classes, summed over positives and the mined negatives
- localisation weight, 1 in the paper
- predicted offsets of box
- default box (centre, width, height)
- matched ground-truth box; the same parameterisation as Faster R-CNN
After matching, almost all of the 8732 boxes are negatives. SSD uses hard negative mining: it sorts the negatives by their confidence loss and keeps only the highest, so that negatives:positives is at most 3:1. The paper reports that this gives faster optimisation and more stable training.
Configuration and results (Liu et al., 2016)
- Backbone
- VGG-16 pre-trained on ImageNet, fc6/fc7 converted to convolutions, plus extra layers conv8–conv11
- Inputs and boxes
- SSD300: , 8732 default boxes. SSD512 adds a seventh map and uses 24,564 boxes.
- Augmentation
- random crops whose IoU with the objects is at least 0.1, 0.3, 0.5, 0.7 or 0.9, plus flips and photometric distortion; the paper credits the crop sampling with +8.8 mAP
- VOC2007 test (07+12 train)
- SSD300 74.3 mAP, SSD512 76.8 mAP; Faster R-CNN (VGG-16) 73.2
- Speed (Titan X)
- SSD300 59 FPS; Faster R-CNN 7 FPS; YOLO v1 45 FPS at 63.4 mAP
- COCO test-dev
- AP 23.2 (SSD300) and 26.8 (SSD512)
Worked example: one 100 px object
Ground truth in a image. For each map, take its default box with the highest IoU:
| Map | Best default box (x₁, y₁, x₂, y₂) | IoU | Label |
|---|---|---|---|
| conv4_3 | (102.8, 102.8, 145.2, 145.2), 42 px square | 0.180 | negative |
| fc7 | (111.2, 111.2, 192.8, 192.8), inside g: 81.6² / 100² = 0.666 | 0.666 | positive |
| conv8_2 | (88.5, 88.5, 199.5, 199.5): 99.5² / (111² + 100² − 99.5²) | 0.797 | positive (best) |
| conv9_2 | (83.7, 113.2, 300.0, 206.8), 2:1 box | 0.403 | negative |
The best box has centre and side 111, so its targets are and . In total 16 boxes exceed IoU 0.5, so and the classification loss uses the 48 hardest of the 8716 negatives.
1import numpy as np23fmap = [38, 19, 10, 5, 3, 1]4step = [8, 16, 32, 64, 100, 300]5smin = [30, 60, 111, 162, 213, 264]6smax = [60, 111, 162, 213, 264, 315]7ratio = [[2], [2, 3], [2, 3], [2, 3], [2], [2]]89boxes = []10for f, st, s, S, ars in zip(fmap, step, smin, smax, ratio):11 wh = [(s, s), (np.sqrt(s * S),) * 2]12 wh += [(s * np.sqrt(a), s / np.sqrt(a)) for a in ars] + [(s / np.sqrt(a), s * np.sqrt(a)) for a in ars]13 for i in range(f):14 for j in range(f):15 cx, cy = (j + .5) * st, (i + .5) * st16 boxes += [(cx - w / 2, cy - h / 2, cx + w / 2, cy + h / 2) for w, h in wh]17D = np.clip(np.array(boxes), 0, 300)18print(len(D)) # 87321920def iou(D, g):21 lt, rb = np.maximum(D[:, :2], g[:2]), np.minimum(D[:, 2:], g[2:])22 inter = np.prod(np.clip(rb - lt, 0, None), axis=1)23 area = lambda b: (b[..., 2] - b[..., 0]) * (b[..., 3] - b[..., 1])24 return inter / (area(D) + area(g) - inter)2526g = np.array([100., 100., 200., 200.])27o = iou(D, g)28pos = o > 0.529pos[o.argmax()] = True # every gt keeps its best box30n_pos = pos.sum()31print(n_pos, o.max().round(3), min(3 * n_pos, (~pos).sum())) # 16 0.797 48Use when
- +One forward pass with only convolutions: 59 FPS for SSD300, versus 7 FPS for Faster R-CNN with the same VGG-16 backbone
- +Multi-scale maps and several aspect ratios per location, so object sizes are covered without an image pyramid
- +Simple to deploy; SSDLite with a MobileNet backbone became a standard mobile detector
Avoid when
- −Weak on small objects: the finest map is stride 8 with shallow features
- −Hand-designed scales and aspect ratios must suit the dataset
- −Hard negative mining keeps only the top 3:1 negatives; RetinaNet's focal loss trains on all of them instead
Check yourself
An SSD300 image contains one object. After matching, 12 default boxes are positive. How many default boxes contribute to the classification loss?
Check yourself
Why does SSD attach predictors to conv4_3 as well as to the deep 1×1–10×10 maps?
Remember
- ✓One-stage detector: a conv head on six maps () predicts scores and 4 offsets for or 6 default boxes per cell, 8732 in total for SSD300.
- ✓Default-box scales grow linearly from 0.2 to 0.9 of the image; aspect ratios plus an extra square .
- ✓Positives: each ground truth's best box plus every box with IoU > 0.5; offsets use the Faster R-CNN (cx, cy, log w, log h) encoding.
- ✓Hard negative mining keeps negatives at most 3:1 to positives; the loss is (confidence + localisation) / N.
- ✓SSD300: 74.3 mAP on VOC2007 at 59 FPS; weak on small objects, which later feature-pyramid detectors improve.