Object detection

Detection outputs a set of (box, class, score) triples. Two-stage, one-stage and set-prediction detectors differ in how they produce that set; all are scored by IoU-matched average precision: AP50 on PASCAL VOC, AP averaged over IoU 0.50–0.95 on COCO.

8 min read748 wordscore0 viewsupdated 2026-10-15

Definition

Object detection is the task of finding every instance of a set of KK object classes in an image and returning, for each one, an axis-aligned bounding box, a class label and a confidence score. The number of objects is not known in advance. The standard benchmarks are PASCAL VOC (Everingham et al., IJCV 2010; 20 classes) and MS COCO (Lin et al., ECCV 2014; 80 classes).

Classification answers "what is in the image"; detection answers "what is where, and how many". This adds three difficulties: objects range from a few pixels to the whole frame, they overlap and occlude each other, and the output is a variable-size set. Before deep learning, detectors slid a window over an image pyramid and scored each window with hand-built features: Viola–Jones for faces (2001), HOG with a linear SVM for pedestrians (Dalal and Triggs, 2005), and deformable part models (DPM, Felzenszwalb et al., 2010).

Boxes and IoU

A box is (x1,y1,x2,y2)(x_1,y_1,x_2,y_2) or (centre, width, height) in pixels. Two boxes are compared by intersection over union, which is 1 for identical boxes and 0 for disjoint ones and does not depend on the image scale:

IoU(A,B)=∣A∩B∣∣A∪B∣=∣A∩B∣∣A∣+∣B∣−∣A∩B∣\mathrm{IoU}(A,B)=\frac{|A\cap B|}{|A\cup B|}=\frac{|A\cap B|}{|A|+|B|-|A\cap B|}
∣A∩B∣|A\cap B|
area of the overlap rectangle, max⁡(0,min⁡(x2)−max⁡(x1))⋅max⁡(0,min⁡(y2)−max⁡(y1))\max(0,\min(x_2)-\max(x_1))\cdot\max(0,\min(y_2)-\max(y_1))
∣A∣, ∣B∣|A|,\ |B|
box areas (x2−x1)(y2−y1)(x_2-x_1)(y_2-y_1)

IoU is used three times: to decide which anchors or proposals are positives in training, to remove duplicates at test time, and to decide whether a detection counts as correct. As a training loss it appears in Intersection over Union (IoU) loss.

Three families

detection = a variable-size set of (box, class, score), scored against ground truth by IoUperson 0.97car 0.91box = (x1, y1, x2, y2) in pixelsground truthpredictionIoU = |A ∩ B| / |A ∪ B|here 896 / 4032 = 0.22: not a matchTP if IoU ≥ t with an unmatched gt boxVOC: t = 0.5 · COCO: mean overt = 0.50, 0.55, …, 0.95two-stagepropose, then classifyregion proposals~2k selective search / 300 RPNcrop features per RoIRoIPool / RoIAlign → 7×7head per RoIK+1 scores + 4 box deltasper-class NMSdrop IoU > τ with a kept boxR-CNN, Fast/Faster/Mask R-CNNone-stage (dense)predict at every locationfeature pyramidstrides 8, 16, 32, … (FPN)every cell × A anchorsor anchor-free: a pointK scores + 4 box valuesdeltas, or distances l,t,r,bscore filter + NMSthousands → ≤ 100YOLO, SSD, RetinaNet, FCOSset predictiona fixed set of N outputsCNN + transformer encoderimage tokensN = 100 object queriestransformer decoderN × (K+1 classes, box)class ∅ = no objectHungarian matchingone query per object, no NMSDETR, Deformable DETR
fig.The output format, IoU, and the three ways to produce the set. Two-stage detectors propose regions and classify each one (R-CNN warps image crops; later versions crop features). One-stage detectors predict at every cell of a feature map. Set-prediction detectors decode a fixed set of queries and are trained with one-to-one Hungarian matching, so they need no NMS.

Two-stage detectors first generate class-agnostic region proposals, then classify and refine each one. R-CNN (2014) ran a CNN on about 2,000 warped crops from selective search. Fast R-CNN (2015) ran the CNN once and pooled each region from the feature map. Faster R-CNN (2015) replaced selective search with a learned region proposal network on anchors. Mask R-CNN (2017) added a mask branch and RoIAlign.

One-stage detectors skip proposals and predict class scores and a box at every feature-map location. YOLO (2016) divides the image into a grid. SSD (2016) uses default boxes on several feature maps. RetinaNet (2017) matched two-stage accuracy by using Focal loss against the extreme foreground–background imbalance. Anchor-based heads regress offsets from preset boxes; anchor-free heads such as FCOS (Tian et al., 2019) predict the distances from a point to the four box sides, or, as in CornerNet and CenterNet, keypoints.

Set prediction: DETR (Carion et al., 2020) decodes N=100N=100 learned object queries with a transformer and matches them one to one to the ground truth with the Hungarian algorithm, so duplicates are penalised in training and no NMS is needed. Deformable DETR (2021) attends to a few sampled points per query, which converges much faster and helps small objects.

Non-maximum suppression

  1. 1Drop detections below a score threshold (e.g. 0.05) and sort the rest by score.
  2. 2Take the highest-scoring box and keep it.
  3. 3Remove every remaining box of the same class whose IoU with it exceeds τ\tau (typically 0.3 to 0.7).
  4. 4Repeat on what is left; keep at most 100 boxes per image (the COCO limit).

A dense or proposal-based detector fires many times on each object, so NMS is part of the model at inference. Its threshold is a trade-off: too low and a second, overlapping object is deleted; too high and duplicates survive as false positives.

Evaluation: average precision

Detections of one class are sorted by score over the whole test set. Each is a true positive if it overlaps a not-yet-matched ground-truth box with IoU ≥t\ge t, otherwise a false positive. Sweeping down the list gives a precision–recall curve; AP is the area under its monotone envelope:

APt=1101∑r∈{0, 0.01, …, 1} max⁡r′≥rp(r′),AP=110∑t∈{0.50, 0.55, …, 0.95}APt\mathrm{AP}_t=\frac{1}{101}\sum_{r\in\{0,\,0.01,\,\dots,\,1\}}\ \max_{r'\ge r}p(r'),\qquad \mathrm{AP}=\frac{1}{10}\sum_{t\in\{0.50,\,0.55,\,\dots,\,0.95\}}\mathrm{AP}_t
p(r′)p(r')
precision at the point of the ranked list where recall reaches r′r' (0 if never reached)
tt
IoU threshold for a match; VOC uses only t=0.5t=0.5, COCO averages ten
AP\mathrm{AP}
also averaged over classes; COCO calls this mean simply "AP" and writes AP50 and AP75 for single thresholds

Benchmarks and reference results

PASCAL VOC
20 classes; VOC 2007 has 9,963 images (5,011 trainval, 4,952 test); mAP at IoU 0.5, 11-point interpolated for VOC 2007, area under the curve from VOC 2010 on
MS COCO
80 classes; train2017 about 118k images, val2017 5k; reports AP (IoU 0.50:0.95), AP50, AP75 and AP for small (area <322<32^2), medium and large (>962>96^2) objects, with up to 100 detections per image
Faster R-CNN, VGG-16
73.2% mAP on VOC 2007 test (trained on VOC 2007 + 2012), about 5 FPS
YOLO (v1)
63.4% mAP on VOC 2007 test at 45 FPS
RetinaNet, ResNet-101-FPN
39.1 AP on COCO test-dev
DETR, ResNet-50
42.0 AP on COCO val, matching a tuned Faster R-CNN with FPN, after a 500-epoch schedule
NMS at IoU > 0.50: 5 of 10 boxes kept · 3 TP at IoU ≥ 0.5person 0.70ground truth (people overlap, IoU 0.474) · TP FP · dashed = suppresseddetections by score (click one)0.95 personkept · TP0.92 car kept · TP0.85 personremoved by 0.95 (IoU 0.69)0.80 car removed by 0.92 (IoU 0.66)0.70 personkept · TP0.66 car removed by 0.92 (IoU 0.71)0.62 personremoved by 0.95 (IoU 0.72)0.50 personremoved by 0.70 (IoU 0.74)0.30 personkept · FP0.20 car kept · FPmAP over 2 classes at each match threshold (101-point interpolated)1.000.501.000.551.000.601.000.651.000.701.000.751.000.801.000.851.000.900.000.95AP50 1.000AP75 1.000AP@[.5:.95] 0.900AP@[.5:.95] averages the 10 bars; at 0.95 no box is tight enough
post-processing
fig.Ten scored boxes around two people (ground-truth IoU 0.474) and a car, with per-class greedy NMS. At the default τ = 0.50, 5 boxes survive, 3 are true positives, AP50 = 1.000 and AP@[.5:.95] = 0.900 (no box reaches IoU 0.95). At τ = 0.30 the second person's 0.70 box is suppressed by the first (IoU 0.47), person AP50 falls to 0.505 and mAP50 to 0.753. At τ = 0.70 or without NMS the 0.85 duplicate outranks a true positive: mAP50 0.918, AP 0.826.

Worked example: NMS and AP for the person class

StepComputationResult
IoU(A, B)6804 / (8024 + 8576 − 6804) = 6804 / 97960.695
IoU(A, C)5109 / (8024 + 7923 − 5109) = 5109 / 108380.471
NMS, τ = 0.5keep A (0.95); B: 0.695 > 0.5, removed; C: 0.471 ≤ 0.5, keptA, C
Match at t = 0.5A → person 1 (IoU 0.924); C → person 2 (IoU 0.930)TP, TP
AP50, τ = 0.5precision 1 at recall 0.5 and 1.01.000
NMS, τ = 0.7B survives (0.695 ≤ 0.7); ranked A, B, C = TP, FP, TPA, B, C
AP50, τ = 0.7precision 1 for r ≤ 0.5 (51 points), 2/3 for r > 0.5 (50 points): (51 + 33.33) / 1010.835
Three person detections from the figure; det A = (62, 42, 121, 178), det B = (52, 34, 116, 168), det C = (82, 47, 139, 186)

B overlaps person 1 with IoU 0.731, enough to be a correct box on its own, but person 1 is already taken by A, so B is a false positive. Detection metrics count each object once.

python
1import torch
2from torchvision.io import read_image
3from torchvision.models.detection import fasterrcnn_resnet50_fpn_v2, FasterRCNN_ResNet50_FPN_V2_Weights
4from torchvision.ops import box_iou, batched_nms
5
6weights = FasterRCNN_ResNet50_FPN_V2_Weights.COCO_V1
7model = fasterrcnn_resnet50_fpn_v2(weights=weights, box_score_thresh=0.05).eval()
8img = read_image("street.jpg")
9with torch.no_grad():
10 out = model([weights.transforms()(img)])[0] # dict: boxes (N, 4) xyxy, labels, scores
11names = weights.meta["categories"]
12for b, l, s in zip(out["boxes"], out["labels"], out["scores"]):
13 if s > 0.5:
14 print(names[l], round(s.item(), 3), b.round().tolist())
15
16# NMS is built into the model; this is the same operation done by hand
17boxes = torch.tensor([[62, 42, 121, 178], [52, 34, 116, 168], [82, 47, 139, 186.]])
18scores = torch.tensor([0.95, 0.85, 0.70])
19labels = torch.zeros(3, dtype=torch.long) # same class: suppression applies
20print(box_iou(boxes, boxes)[0]) # tensor([1.0000, 0.6946, 0.4714])
21print(batched_nms(boxes, scores, labels, iou_threshold=0.5)) # tensor([0, 2])
Pretrained COCO detector, IoU and NMS with torchvision (not run here: needs torch, torchvision and an image file)

Use when

  • +Two-stage: accurate localisation, per-region features are easy to extend (masks, keypoints)
  • +One-stage: one dense pass, simple and fast; the basis of most real-time detectors
  • +Set prediction: no anchors and no NMS; the loss directly targets a duplicate-free set

Avoid when

  • −Two-stage: two networks or heads and more latency per image
  • −One-stage: extreme class imbalance (handled by focal loss or sampling); needs NMS
  • −Set prediction (original DETR): slow convergence and weak small-object AP

Check yourself

Two people stand close together; their ground-truth boxes have IoU 0.47. What happens with NMS at τ = 0.3?

Check yourself

Which change raises COCO AP@[.5:.95] but leaves AP50 unchanged?

Remember

  • ✓Output: a variable-size set of (box, class, score); boxes are compared by IoU = intersection / union.
  • ✓Two-stage (R-CNN family): propose, then classify each region. One-stage (YOLO, SSD, RetinaNet, FCOS): predict at every location. Set prediction (DETR): N queries, Hungarian matching, no NMS.
  • ✓Greedy per-class NMS removes same-class boxes with IoU > τ; too low a τ deletes overlapping objects, too high keeps duplicates.
  • ✓AP = area under the interpolated precision–recall curve; VOC uses IoU 0.5, COCO averages AP over IoU 0.50 to 0.95 and over 80 classes.
← back to Deep Learning roadmap