Definition
Object tracking estimates where each object is in every frame of a video while keeping its identity. Single-object tracking (SOT) follows one target given only its box in the first frame. SiamFC (Bertinetto, Valmadre, Henriques, Vedaldi and Torr, ECCV 2016 workshops) is the standard deep baseline. Multi-object tracking (MOT) follows every object of a class, usually by tracking by detection: SORT (Bewley et al., ICIP 2016), DeepSORT (Wojke, Bewley and Paulus, ICIP 2017) and ByteTrack (Zhang et al., ECCV 2022).
The output is a set of trajectories : an identity and a box per frame. A detector alone gives boxes but no identities, and it misses objects that are occluded, blurred or small. A tracker must link boxes over time and bridge such gaps. It must also avoid swapping identities when two objects cross, and it has to work online, frame by frame, at video rate.
Before deep features, SOT was dominated by correlation filters (MOSSE in 2010, KCF in 2015), which learn a template online in the Fourier domain. MOT relied on hand-crafted motion and appearance models with multiple-hypothesis or network-flow data association. The deep era kept the overall structure and replaced the parts with learned detectors and embeddings.
Single object: Siamese matching
SiamFC learns a similarity function offline and does not update it while tracking. One shared fully convolutional network embeds a exemplar (the first-frame target) and a search region centred on the last position. Their cross-correlation is a score map:
- slid over as a convolution kernel, giving one similarity per position
- AlexNet-style network with total stride 8, trained with a logistic loss on frame pairs from ImageNet VID
- a learned bias; the peak, upsampled and mapped back through the stride, gives the new centre
Scale is handled by scoring the search region at a few scales. SiamRPN (Li et al., CVPR 2018) added a region-proposal head that regresses the box directly. Later trackers use deeper backbones and transformer cross-attention between template and search region (see Cross-attention).
Many objects: detect, predict, associate
SORT combines an off-the-shelf detector (Faster R-CNN in the paper, see Faster R-CNN) with two classical tools. Each track carries a Kalman filter with state : the box centre, area and aspect ratio, plus the velocities of the first three, with the aspect ratio held constant. Every frame the filter predicts each track forward, and the Hungarian algorithm matches predictions to detections by IoU.
- constant-velocity transition: position += velocity each frame
- process and measurement noise; a large initial velocity variance lets the first matches set the velocity
- Kalman gain: how far the matched detection z_t pulls the prediction
- one-to-one assignment of tracks i to detections j (Hungarian algorithm on cost 1 − IoU)
- unmatched detections start new tracks; unmatched tracks are deleted after a few missed frames (1 in SORT)
SORT ran at 260 Hz excluding detection, but motion and IoU alone fail when objects are occluded for longer than the Kalman prediction stays accurate. DeepSORT adds a 128-d appearance embedding from a re-identification CNN trained on the MARS person dataset. The cost combines a Mahalanobis distance from the Kalman filter, used as a gate at the 95% quantile (9.4877 for 4 degrees of freedom), with the smallest cosine distance to the track's last 100 embeddings. A matching cascade gives priority to recently seen tracks, and lost tracks are kept for up to 30 frames.
ByteTrack observed that discarding low-confidence detections throws away occluded objects. It splits detections at a score of 0.6. High boxes are matched first. The remaining tracks are then matched by IoU to low boxes above 0.1, which recovers partly hidden objects, and unmatched low boxes are discarded as background. New tracks start only from unmatched high boxes. With a YOLOX detector, it reached 80.3 MOTA, 77.3 IDF1 and 63.1 HOTA on the MOT17 test set at 30 FPS.
Metrics: detection versus identity
- frames where a ground-truth object's matched track ID changes (CLEAR MOT, Bernardin and Stiefelhagen, 2008)
- box matches under the single best one-to-one mapping between whole GT and predicted trajectories (Ristani et al., 2016); 2·IDTP + IDFP + IDFN = GT boxes + predicted boxes
- detection accuracy times association accuracy, square-rooted and averaged over IoU thresholds α = 0.05 … 0.95 (Luiten et al., IJCV 2021)
MOTA counts every error as one event per frame. Because missed and false boxes usually outnumber identity switches, MOTA mostly measures the detector. IDF1 asks how long each object keeps one ID, so it punishes switches and fragmentation. HOTA balances the two.
Benchmarks and reported numbers
- MOT17
- 14 pedestrian sequences (7 train, 7 test), each released with public detections from DPM, Faster R-CNN and SDP. MOT20 adds very crowded scenes.
- SOT benchmarks
- OTB, VOT and LaSOT: one target per sequence, given in the first frame; scored by overlap success and precision, or by accuracy and robustness in VOT
- SORT
- Kalman filter + Hungarian on IoU, IoU_min = 0.3; 260 Hz for the tracking step alone
- ByteTrack (YOLOX-X)
- MOT17 test: 80.3 MOTA, 77.3 IDF1, 63.1 HOTA, 30 FPS
Worked example: one switch, two metrics
| Tracker | FN | FP | IDSW | MOTA | IDTP | IDF1 | IDs |
|---|---|---|---|---|---|---|---|
| Kalman, gate 0.3, max age 3 | 3 | 0 | 0 | 1 − 3/48 = 0.938 | 45 | 90/93 = 0.968 | 2 |
| Kalman, max age 1 | 3 | 0 | 1 | 1 − 4/48 = 0.917 | 35 | 70/93 = 0.753 | 3 |
| No motion model, max age 3 | 3 | 0 | 2 | 1 − 5/48 = 0.896 | 23 | 46/93 = 0.495 | 2 |
| Kalman, gate 0.7 | 3 | 0 | 35 | 1 − 38/48 = 0.208 | 5 | 10/93 = 0.108 | 37 |
IDF1's denominator is GT + predicted boxes . With max age 1, B's trajectory is split into a 10-frame and an 11-frame piece. Only one piece can map to B, so 10 correct boxes stop counting and IDF1 drops by 0.215. MOTA sees just one extra error out of 48 and drops by 0.021.
1from collections import defaultdict2from ultralytics import YOLO34model = YOLO("yolo11n.pt") # any detection checkpoint5tracks = defaultdict(list)67# persist=True keeps tracker state between calls; botsort.yaml adds camera-motion compensation8for r in model.track("street.mp4", tracker="bytetrack.yaml", classes=[0], stream=True, persist=True):9 if r.boxes.id is None: # no confirmed tracks in this frame10 continue11 for tid, xywh in zip(r.boxes.id.int().tolist(), r.boxes.xywh.tolist()):12 tracks[tid].append(xywh) # centre x, y, width, height1314print(len(tracks), "identities;", sum(map(len, tracks.values())), "boxes")Check yourself
Why does ByteTrack match low-score detections only to tracks left unmatched after the high-score stage?
Check yourself
A tracker has the same FN and FP counts as a second tracker, but its tracks are fragmented into twice as many IDs. Which metric shows the difference most clearly?
Remember
- ✓SOT follows one target given its first box. SiamFC scores : a exemplar slid over a search region gives a map.
- ✓MOT is mostly tracking by detection: detect, predict each track with a constant-velocity Kalman filter, and match with the Hungarian algorithm on IoU (SORT, IoU_min = 0.3).
- ✓DeepSORT adds 128-d re-ID embeddings with Mahalanobis gating and a matching cascade. ByteTrack adds a second association stage for low-score boxes.
- ✓MOTA = 1 − (FN + FP + IDSW)/GT is dominated by detection. IDF1 and HOTA measure whether identities persist.
- ✓In the figure, one track deletion costs 0.021 MOTA but 0.215 IDF1.