Definition
YOLO (You Only Look Once; Redmon, Divvala, Girshick and Farhadi, "You Only Look Once: Unified, Real-Time Object Detection", CVPR 2016) frames detection as a single regression problem. One convolutional network reads the whole image once and outputs all boxes and class probabilities directly. There are no region proposals, no sliding windows and no per-region classifier.
Input: an image resized to . Output: boxes with class scores. In 2015 the accurate detectors were two-stage: R-CNN-style pipelines that propose regions, then classify each one (see Fast R-CNN and Faster R-CNN). Faster R-CNN with VGG-16 ran at about 7 FPS. YOLO traded some accuracy for a simple, end-to-end network that ran at 45 FPS; the paper reports more than twice the mAP of earlier real-time detectors on PASCAL VOC.
The grid
The image is divided into an grid with . If an object's centre falls into a cell, that cell is responsible for detecting it. Each cell predicts boxes, each with 5 numbers , and one set of conditional class probabilities . The output is an tensor: 1470 numbers and 98 boxes per image.
is the box centre as an offset inside its cell, in . are relative to the whole image. The confidence target is : 0 for cells without an object, and the IoU between the predicted and the true box otherwise.
Network and loss
The network has 24 convolutional layers followed by 2 fully connected layers, using reduction layers before convolutions in the style of GoogLeNet (see GoogLeNet / Inception). Fast YOLO uses 9 convolutional layers. The first 20 layers are pre-trained on ImageNet at , then the input is raised to for detection. All layers use leaky ReLU with slope 0.1, except the last, which is linear.
Training uses sum-squared error on all outputs. Of the predictors in a responsible cell, the one whose current box has the highest IoU with the ground truth is made responsible (). This makes the two predictors specialise in different sizes and shapes.
- 1 if predictor of cell is responsible for an object
- up-weights box coordinates
- down-weights the confidence of the many empty-cell predictors
- an error of 0.05 in costs more on a box of width 0.1 than on one of width 0.8
At test time the class-specific score of each box is . Low scores are dropped and non-maximum suppression removes duplicates; see Object detection for IoU and NMS.
Results and the lineage (only the published facts)
- YOLO v1 (2016)
- VOC2007 test: 63.4 mAP at 45 FPS; Fast YOLO 52.7 mAP at 155 FPS (Titan X). Fewer background false positives than Fast R-CNN, more localisation errors.
- YOLOv2 (2017)
- Redmon and Farhadi, "YOLO9000: Better, Faster, Stronger" (CVPR 2017). Batch norm, Darknet-19, anchor boxes from k-means on training boxes (distance , ), centres , sizes , a grid at 416 px, multi-scale training. VOC2007: 76.8 mAP at 67 FPS (416 px).
- YOLOv3 (2018)
- Redmon and Farhadi, "YOLOv3: An Incremental Improvement" (tech report). Darknet-53, predictions at 3 scales (strides 32, 16, 8), 3 of 9 k-means anchors per scale, independent logistic class scores. At 416 px: boxes.
- Later
- YOLOv4 (Bochkovskiy, Wang and Liao, 2020); YOLOX (Ge et al., 2021): anchor-free with a decoupled head; Ultralytics YOLOv5 (2020) and YOLOv8 (2023, anchor-free), released as software without a paper
Worked example: two people in one cell
A image has two people with centres at and in normalised coordinates, sizes and .
| Object | S = 7: cx·S, cy·S | cell | targets x, y, √w, √h | S = 13: cell |
|---|---|---|---|---|
| person 1 | 4.62, 4.20 | (4, 4) | 0.620, 0.200, 0.316, 0.632 | (7, 8) |
| person 2 | 4.90, 4.34 | (4, 4) | 0.900, 0.340, 0.300, 0.600 | (8, 9) |
At both centres land in cell . The cell has two box predictors but only one class vector and, in practice, one target object, so one person goes unsupervised. Decoding person 1 back: px and px. At the two people fall in different cells. This is the limitation the paper names: YOLO struggles with small objects in groups, such as flocks of birds.
1import numpy as np23S, B, C, IMG = 7, 2, 20, 44845def decode_v1(out, thresh=0.2):6 """out: (S, S, B*5 + C) YOLOv1 output -> list of (x1, y1, x2, y2, score, cls) in pixels."""7 dets = []8 for row in range(S):9 for col in range(S):10 cell = out[row, col]11 cls_prob = cell[B * 5:] # Pr(class | object), shared by both boxes12 for b in range(B):13 x, y, sw, sh, conf = cell[b * 5:(b + 1) * 5]14 cx, cy = (col + x) / S * IMG, (row + y) / S * IMG # x, y are offsets inside the cell15 w, h = sw ** 2 * IMG, sh ** 2 * IMG # the net predicts sqrt(w), sqrt(h)16 scores = conf * cls_prob # class-specific confidence17 k = scores.argmax()18 if scores[k] > thresh:19 dets.append((cx - w / 2, cy - h / 2, cx + w / 2, cy + h / 2, scores[k], k))20 return dets # then per-class NMS, e.g. torchvision.ops.batched_nms2122out = np.zeros((S, S, B * 5 + C))23out[4, 4, :5] = [0.62, 0.20, 0.316, 0.632, 0.9] # the person of the worked example24out[4, 4, 5:10] = [0.50, 0.50, 0.200, 0.200, 0.1] # second predictor of the same cell, low confidence25out[4, 4, 10 + 14] = 0.95 # class 14 = person in VOC order26for d in decode_v1(out):27 print(np.round(d[:4], 1), round(d[4], 3), d[5]) # [273.3 179.3 318. 358.3] 0.855 14Where it's used
Real-time video
Surveillance, traffic and sports analytics, where 30+ FPS on one GPU matters more than the last few AP points
Edge and robotics
Small YOLO variants run on embedded GPUs and phones; the single fully convolutional head (from v2 on) exports easily to ONNX and TensorRT
Tracking front end
Per-frame YOLO detections feed trackers that associate boxes by IoU and motion (see Object tracking)
Check yourself
For S = 7, B = 2 and C = 20, what is the size of YOLO v1's output, and how many boxes does it predict?
Check yourself
Why does YOLO v1 regress √w and √h instead of w and h?
Remember
- ✓One network, one pass: the image is split into a 7×7 grid; the cell holding an object's centre is responsible for it.
- ✓Each cell predicts 2 boxes (x, y, √w, √h, confidence = Pr(object)·IoU) and 20 class probabilities, a 7×7×30 tensor and 98 boxes.
- ✓Sum-squared loss with λ_coord = 5 and λ_noobj = 0.5; score = class probability × confidence, then NMS.
- ✓63.4 mAP on VOC2007 at 45 FPS; weak on small grouped objects and on precise localisation.
- ✓v2: k-means anchors and σ(t) + c offsets; v3: 3 scales, 9 anchors, 10,647 boxes at 416 px; YOLOX and YOLOv8 are anchor-free.