# Object detection

> Detection outputs a set of (box, class, score) triples. Two-stage, one-stage and set-prediction detectors differ in how they produce that set; all are scored by IoU-matched average precision: AP50 on PASCAL VOC, AP averaged over IoU 0.50–0.95 on COCO.

Source: https://www.mlfoundry.in/learn/object-detection  
Section: Deep Learning › Deep Learning for Computer Vision  
Level: core · 8 min read · updated 2026-10-15

**Object detection:** is the task of finding every instance of a set of $K$ object classes in an image and returning, for each one, an axis-aligned bounding box, a class label and a confidence score. The number of objects is not known in advance. The standard benchmarks are PASCAL VOC (Everingham et al., IJCV 2010; 20 classes) and MS COCO (Lin et al., ECCV 2014; 80 classes).

Classification answers "what is in the image"; detection answers "what is where, and how many". This adds three difficulties: objects range from a few pixels to the whole frame, they overlap and occlude each other, and the output is a variable-size **set**. Before deep learning, detectors slid a window over an image pyramid and scored each window with hand-built features: Viola–Jones for faces (2001), HOG with a linear SVM for pedestrians (Dalal and Triggs, 2005), and deformable part models (DPM, Felzenszwalb et al., 2010).

## Boxes and IoU

A box is $(x_1,y_1,x_2,y_2)$ or (centre, width, height) in pixels. Two boxes are compared by **intersection over union**, which is 1 for identical boxes and 0 for disjoint ones and does not depend on the image scale:

$$
\mathrm{IoU}(A,B)=\frac{|A\cap B|}{|A\cup B|}=\frac{|A\cap B|}{|A|+|B|-|A\cap B|}
$$

- $|A\cap B|$: area of the overlap rectangle, $\max(0,\min(x_2)-\max(x_1))\cdot\max(0,\min(y_2)-\max(y_1))$
- $|A|,\ |B|$: box areas $(x_2-x_1)(y_2-y_1)$

IoU is used three times: to decide which anchors or proposals are positives in training, to remove duplicates at test time, and to decide whether a detection counts as correct. As a training loss it appears in **Intersection over Union (IoU) loss**.

## Three families

> Figure: The output format, IoU, and the three ways to produce the set. Two-stage detectors propose regions and classify each one (R-CNN warps image crops; later versions crop features). One-stage detectors predict at every cell of a feature map. Set-prediction detectors decode a fixed set of queries and are trained with one-to-one Hungarian matching, so they need no NMS.

**Two-stage** detectors first generate class-agnostic region proposals, then classify and refine each one. **R-CNN** (2014) ran a CNN on about 2,000 warped crops from selective search. **Fast R-CNN** (2015) ran the CNN once and pooled each region from the feature map. **Faster R-CNN** (2015) replaced selective search with a learned region proposal network on anchors. **Mask R-CNN** (2017) added a mask branch and RoIAlign.

**One-stage** detectors skip proposals and predict class scores and a box at every feature-map location. **YOLO** (2016) divides the image into a grid. **SSD** (2016) uses default boxes on several feature maps. **RetinaNet** (2017) matched two-stage accuracy by using **Focal loss** against the extreme foreground–background imbalance. Anchor-based heads regress offsets from preset boxes; anchor-free heads such as FCOS (Tian et al., 2019) predict the distances from a point to the four box sides, or, as in CornerNet and CenterNet, keypoints.

**Set prediction**: **DETR** (Carion et al., 2020) decodes $N=100$ learned object queries with a transformer and matches them one to one to the ground truth with the Hungarian algorithm, so duplicates are penalised in training and no NMS is needed. **Deformable DETR** (2021) attends to a few sampled points per query, which converges much faster and helps small objects.

## Non-maximum suppression

1. Drop detections below a score threshold (e.g. 0.05) and sort the rest by score.
2. Take the highest-scoring box and keep it.
3. Remove every remaining box **of the same class** whose IoU with it exceeds $\tau$ (typically 0.3 to 0.7).
4. Repeat on what is left; keep at most 100 boxes per image (the COCO limit).

A dense or proposal-based detector fires many times on each object, so NMS is part of the model at inference. Its threshold is a trade-off: too low and a second, overlapping object is deleted; too high and duplicates survive as false positives.

## Evaluation: average precision

Detections of one class are sorted by score over the whole test set. Each is a true positive if it overlaps a not-yet-matched ground-truth box with IoU $\ge t$, otherwise a false positive. Sweeping down the list gives a precision–recall curve; AP is the area under its monotone envelope:

$$
\mathrm{AP}_t=\frac{1}{101}\sum_{r\in\{0,\,0.01,\,\dots,\,1\}}\ \max_{r'\ge r}p(r'),\qquad \mathrm{AP}=\frac{1}{10}\sum_{t\in\{0.50,\,0.55,\,\dots,\,0.95\}}\mathrm{AP}_t
$$

- $p(r')$: precision at the point of the ranked list where recall reaches $r'$ (0 if never reached)
- $t$: IoU threshold for a match; VOC uses only $t=0.5$, COCO averages ten
- $\mathrm{AP}$: also averaged over classes; COCO calls this mean simply "AP" and writes AP50 and AP75 for single thresholds

**Benchmarks and reference results**

- **PASCAL VOC:** 20 classes; VOC 2007 has 9,963 images (5,011 trainval, 4,952 test); mAP at IoU 0.5, 11-point interpolated for VOC 2007, area under the curve from VOC 2010 on
- **MS COCO:** 80 classes; train2017 about 118k images, val2017 5k; reports AP (IoU 0.50:0.95), AP50, AP75 and AP for small (area $<32^2$), medium and large ($>96^2$) objects, with up to 100 detections per image
- **Faster R-CNN, VGG-16:** 73.2% mAP on VOC 2007 test (trained on VOC 2007 + 2012), about 5 FPS
- **YOLO (v1):** 63.4% mAP on VOC 2007 test at 45 FPS
- **RetinaNet, ResNet-101-FPN:** 39.1 AP on COCO test-dev
- **DETR, ResNet-50:** 42.0 AP on COCO val, matching a tuned Faster R-CNN with FPN, after a 500-epoch schedule

> Figure: Ten scored boxes around two people (ground-truth IoU 0.474) and a car, with per-class greedy NMS. At the default τ = 0.50, 5 boxes survive, 3 are true positives, AP50 = 1.000 and AP@[.5:.95] = 0.900 (no box reaches IoU 0.95). At τ = 0.30 the second person's 0.70 box is suppressed by the first (IoU 0.47), person AP50 falls to 0.505 and mAP50 to 0.753. At τ = 0.70 or without NMS the 0.85 duplicate outranks a true positive: mAP50 0.918, AP 0.826.

## Worked example: NMS and AP for the person class

Three person detections from the figure; det A = (62, 42, 121, 178), det B = (52, 34, 116, 168), det C = (82, 47, 139, 186)

| Step | Computation | Result |
| --- | --- | --- |
| IoU(A, B) | 6804 / (8024 + 8576 − 6804) = 6804 / 9796 | 0.695 |
| IoU(A, C) | 5109 / (8024 + 7923 − 5109) = 5109 / 10838 | 0.471 |
| NMS, τ = 0.5 | keep A (0.95); B: 0.695 > 0.5, removed; C: 0.471 ≤ 0.5, kept | A, C |
| Match at t = 0.5 | A → person 1 (IoU 0.924); C → person 2 (IoU 0.930) | TP, TP |
| AP50, τ = 0.5 | precision 1 at recall 0.5 and 1.0 | 1.000 |
| NMS, τ = 0.7 | B survives (0.695 ≤ 0.7); ranked A, B, C = TP, FP, TP | A, B, C |
| AP50, τ = 0.7 | precision 1 for r ≤ 0.5 (51 points), 2/3 for r > 0.5 (50 points): (51 + 33.33) / 101 | 0.835 |

B overlaps person 1 with IoU 0.731, enough to be a correct box on its own, but person 1 is already taken by A, so B is a false positive. Detection metrics count each object once.

```python
import torch
from torchvision.io import read_image
from torchvision.models.detection import fasterrcnn_resnet50_fpn_v2, FasterRCNN_ResNet50_FPN_V2_Weights
from torchvision.ops import box_iou, batched_nms

weights = FasterRCNN_ResNet50_FPN_V2_Weights.COCO_V1
model = fasterrcnn_resnet50_fpn_v2(weights=weights, box_score_thresh=0.05).eval()
img = read_image("street.jpg")
with torch.no_grad():
    out = model([weights.transforms()(img)])[0]          # dict: boxes (N, 4) xyxy, labels, scores
names = weights.meta["categories"]
for b, l, s in zip(out["boxes"], out["labels"], out["scores"]):
    if s > 0.5:
        print(names[l], round(s.item(), 3), b.round().tolist())

# NMS is built into the model; this is the same operation done by hand
boxes = torch.tensor([[62, 42, 121, 178], [52, 34, 116, 168], [82, 47, 139, 186.]])
scores = torch.tensor([0.95, 0.85, 0.70])
labels = torch.zeros(3, dtype=torch.long)                # same class: suppression applies
print(box_iou(boxes, boxes)[0])                          # tensor([1.0000, 0.6946, 0.4714])
print(batched_nms(boxes, scores, labels, iou_threshold=0.5))   # tensor([0, 2])
```

Pretrained COCO detector, IoU and NMS with torchvision (not run here: needs torch, torchvision and an image file)

**Strengths**

- Two-stage: accurate localisation, per-region features are easy to extend (masks, keypoints)
- One-stage: one dense pass, simple and fast; the basis of most real-time detectors
- Set prediction: no anchors and no NMS; the loss directly targets a duplicate-free set

**Limitations**

- Two-stage: two networks or heads and more latency per image
- One-stage: extreme class imbalance (handled by focal loss or sampling); needs NMS
- Set prediction (original DETR): slow convergence and weak small-object AP

> **Say which AP:** "mAP 45" means nothing without the protocol. VOC mAP is at IoU 0.5; COCO AP averages 0.50 to 0.95 and is typically 15 to 25 points lower than COCO AP50 for the same model. Also state val2017 or test-dev, the input size and whether test-time augmentation was used.

**Question:** Two people stand close together; their ground-truth boxes have IoU 0.47. What happens with NMS at τ = 0.3?

- A. Both are kept, NMS only removes boxes of different classes
- B. The lower-scoring person's box is removed, which costs recall
- C. Both are removed
- D. NMS merges them into one larger box

**Answer:** B. The lower-scoring person's box is removed, which costs recall — Greedy NMS keeps the higher-scoring box and removes every same-class box with IoU above 0.3, including the correct box on the second person. In the figure this drops person AP50 from 1.000 to 0.505. Crowded scenes need a higher threshold, Soft-NMS, or a set-prediction detector.

**Question:** Which change raises COCO AP@[.5:.95] but leaves AP50 unchanged?

- A. Finding an object that was missed
- B. Removing a high-scoring false positive
- C. Tightening boxes that already had IoU 0.6 to IoU 0.9
- D. Using a lower NMS threshold

**Answer:** C. Tightening boxes that already had IoU 0.6 to IoU 0.9 — A box with IoU 0.6 already counts at $t=0.5$, so AP50 does not move, but it fails the thresholds 0.65 to 0.95. At IoU 0.9 it passes $t=0.5,\dots,0.9$, so the COCO average rises. COCO AP rewards localisation quality.

**Key points**

- Output: a variable-size set of (box, class, score); boxes are compared by IoU = intersection / union.
- Two-stage (R-CNN family): propose, then classify each region. One-stage (YOLO, SSD, RetinaNet, FCOS): predict at every location. Set prediction (DETR): N queries, Hungarian matching, no NMS.
- Greedy per-class NMS removes same-class boxes with IoU > τ; too low a τ deletes overlapping objects, too high keeps duplicates.
- AP = area under the interpolated precision–recall curve; VOC uses IoU 0.5, COCO averages AP over IoU 0.50 to 0.95 and over 80 classes.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
