Object tracking

Tracking keeps object identities across frames. SiamFC matches a template; SORT predicts tracks with a Kalman filter and matches by IoU, DeepSORT adds appearance, ByteTrack adds low-score boxes.

8 min read817 wordscore0 viewsupdated 2026-10-15

Definition

Object tracking estimates where each object is in every frame of a video while keeping its identity. Single-object tracking (SOT) follows one target given only its box in the first frame. SiamFC (Bertinetto, Valmadre, Henriques, Vedaldi and Torr, ECCV 2016 workshops) is the standard deep baseline. Multi-object tracking (MOT) follows every object of a class, usually by tracking by detection: SORT (Bewley et al., ICIP 2016), DeepSORT (Wojke, Bewley and Paulus, ICIP 2017) and ByteTrack (Zhang et al., ECCV 2022).

The output is a set of trajectories {(k,t,bk,t)}\{(k, t, b_{k,t})\}: an identity kk and a box bb per frame. A detector alone gives boxes but no identities, and it misses objects that are occluded, blurred or small. A tracker must link boxes over time and bridge such gaps. It must also avoid swapping identities when two objects cross, and it has to work online, frame by frame, at video rate.

Before deep features, SOT was dominated by correlation filters (MOSSE in 2010, KCF in 2015), which learn a template online in the Fourier domain. MOT relied on hand-crafted motion and appearance models with multiple-hypothesis or network-flow data association. The deep era kept the overall structure and replaced the parts with learned detectors and embeddings.

Single object: Siamese matching

SiamFC learns a similarity function offline and does not update it while tracking. One shared fully convolutional network φ\varphi embeds a 127×127127\times127 exemplar zz (the first-frame target) and a 255×255255\times255 search region xx centred on the last position. Their cross-correlation is a score map:

f(z,x)=φ(z)⋆φ(x)+b 1,6×6×128⏟φ(z) ⋆ 22×22×128⏟φ(x) → 17×17f(z,x)=\varphi(z)\star\varphi(x)+b\,\mathbb 1,\qquad \underbrace{6\times6\times128}_{\varphi(z)}\ \star\ \underbrace{22\times22\times128}_{\varphi(x)}\ \to\ 17\times17
⋆\star
φ(z)\varphi(z) slid over φ(x)\varphi(x) as a convolution kernel, giving one similarity per position
φ\varphi
AlexNet-style network with total stride 8, trained with a logistic loss on frame pairs from ImageNet VID
bb
a learned bias; the peak, upsampled and mapped back through the stride, gives the new centre

Scale is handled by scoring the search region at a few scales. SiamRPN (Li et al., CVPR 2018) added a region-proposal head that regresses the box directly. Later trackers use deeper backbones and transformer cross-attention between template and search region (see Cross-attention).

Many objects: detect, predict, associate

SORT combines an off-the-shelf detector (Faster R-CNN in the paper, see Faster R-CNN) with two classical tools. Each track carries a Kalman filter with state [u,v,s,r,u˙,v˙,s˙][u,v,s,r,\dot u,\dot v,\dot s]: the box centre, area and aspect ratio, plus the velocities of the first three, with the aspect ratio held constant. Every frame the filter predicts each track forward, and the Hungarian algorithm matches predictions to detections by IoU.

x^t∣t−1=Fx^t−1,Pt∣t−1=FPt−1F⊤+Q;K=Pt∣t−1H⊤(HPt∣t−1H⊤+R)−1,x^t=x^t∣t−1+K(zt−Hx^t∣t−1)\hat x_{t\mid t-1}=F\hat x_{t-1},\quad P_{t\mid t-1}=FP_{t-1}F^\top+Q;\qquad K=P_{t\mid t-1}H^\top\big(HP_{t\mid t-1}H^\top+R\big)^{-1},\quad \hat x_t=\hat x_{t\mid t-1}+K\big(z_t-H\hat x_{t\mid t-1}\big)
FF
constant-velocity transition: position += velocity each frame
Q, RQ,\ R
process and measurement noise; a large initial velocity variance lets the first matches set the velocity
KK
Kalman gain: how far the matched detection z_t pulls the prediction
π⋆=arg⁡max⁡π∑(i,j)∈πIoU(Hx^t∣t−1(i), dj),reject pairs with IoU<IoUmin⁡=0.3\pi^\star=\arg\max_{\pi}\sum_{(i,j)\in\pi}\mathrm{IoU}\big(H\hat x^{(i)}_{t\mid t-1},\,d_j\big),\qquad \text{reject pairs with }\mathrm{IoU}<\mathrm{IoU}_{\min}=0.3
π\pi
one-to-one assignment of tracks i to detections j (Hungarian algorithm on cost 1 − IoU)
djd_j
unmatched detections start new tracks; unmatched tracks are deleted after a few missed frames (1 in SORT)
multi-object tracking by detection, one frame (SORT / ByteTrack)T1T2T3d1 .92d2 .81d3 .35d4 .12dashed: predicted tracks · solid: detsmatch 4: IoU, high detections × tracksT1T2T3d10.72700d200.7260.146→ d1–T1, d2–T2; T3 left overmatch 5: d3–T3 0.509 ✓, d4 dropped1 detectboxes + scores (ByteTrack uses YOLOX)2 splithigh ≥ 0.6: d1, d2 · low 0.1–0.6: d3, d43 predictKalman: move every track to frame t4 match highd1, d2 ↔ all tracks, Hungarian on IoU5 match lowd3, d4 ↔ unmatched tracks only (T3)6 manageupdate matched · unmatched high → new trackSORT: one stage on all boxes; ByteTrack adds step 5single-object tracking: SiamFC (Bertinetto et al., 2016)z: exemplar127×127×3x: search255×255×3φ → 6×6×128φ → 22×22×128same AlexNet-style φ⋆score map 17×17f(z, x) = φ(z) ⋆ φ(x) + bpeak × stride 8 → new boxno detector, no class
fig.One frame of tracking by detection. Three Kalman-predicted tracks (dashed) meet four detections. ByteTrack first matches the high-score boxes d1 (0.92) and d2 (0.81) to all tracks: IoU 0.727 with T1 and 0.726 with T2, while d2's 0.146 overlap with T3 loses. The leftover T3 is then matched to the low-score, partly occluded d3 (0.35) at IoU 0.509. The clutter box d4 (0.12) matches nothing and is dropped. SORT would apply one stage to all boxes above its detector threshold. Bottom: SiamFC correlates a 6×6×128 exemplar embedding with a 22×22×128 search embedding to give a 17×17 score map.

SORT ran at 260 Hz excluding detection, but motion and IoU alone fail when objects are occluded for longer than the Kalman prediction stays accurate. DeepSORT adds a 128-d appearance embedding from a re-identification CNN trained on the MARS person dataset. The cost combines a Mahalanobis distance from the Kalman filter, used as a gate at the 95% χ2\chi^2 quantile (9.4877 for 4 degrees of freedom), with the smallest cosine distance to the track's last 100 embeddings. A matching cascade gives priority to recently seen tracks, and lost tracks are kept for up to 30 frames.

ByteTrack observed that discarding low-confidence detections throws away occluded objects. It splits detections at a score of 0.6. High boxes are matched first. The remaining tracks are then matched by IoU to low boxes above 0.1, which recovers partly hidden objects, and unmatched low boxes are discarded as background. New tracks start only from unmatched high boxes. With a YOLOX detector, it reached 80.3 MOTA, 77.3 IDF1 and 63.1 HOTA on the MOT17 test set at 30 FPS.

Metrics: detection versus identity

MOTA=1−∑t(FNt+FPt+IDSWt)∑tGTt,IDF1=2 IDTP2 IDTP+IDFP+IDFN,HOTAα=DetAα⋅AssAα\mathrm{MOTA}=1-\frac{\sum_t(\mathrm{FN}_t+\mathrm{FP}_t+\mathrm{IDSW}_t)}{\sum_t\mathrm{GT}_t},\qquad \mathrm{IDF1}=\frac{2\,\mathrm{IDTP}}{2\,\mathrm{IDTP}+\mathrm{IDFP}+\mathrm{IDFN}},\qquad \mathrm{HOTA}_\alpha=\sqrt{\mathrm{DetA}_\alpha\cdot\mathrm{AssA}_\alpha}
IDSW\mathrm{IDSW}
frames where a ground-truth object's matched track ID changes (CLEAR MOT, Bernardin and Stiefelhagen, 2008)
IDTP\mathrm{IDTP}
box matches under the single best one-to-one mapping between whole GT and predicted trajectories (Ristani et al., 2016); 2·IDTP + IDFP + IDFN = GT boxes + predicted boxes
HOTA\mathrm{HOTA}
detection accuracy times association accuracy, square-rooted and averaged over IoU thresholds α = 0.05 … 0.95 (Luiten et al., IJCV 2021)

MOTA counts every error as one event per frame. Because missed and false boxes usually outnumber identity switches, MOTA mostly measures the detector. IDF1 asks how long each object keeps one ID, so it punishes switches and fragmentation. HOTA balances the two.

Benchmarks and reported numbers

MOT17
14 pedestrian sequences (7 train, 7 test), each released with public detections from DPM, Faster R-CNN and SDP. MOT20 adds very crowded scenes.
SOT benchmarks
OTB, VOT and LaSOT: one target per sequence, given in the first frame; scored by overlap success and precision, or by accuracy and robustness in VOT
SORT
Kalman filter + Hungarian on IoU, IoU_min = 0.3; 260 Hz for the tracking step alone
ByteTrack (YOLOX-X)
MOT17 test: 80.3 MOTA, 77.3 IDF1, 63.1 HOTA, 30 FPS
frame 13/23 · Kalman constant velocity · gate IoU ≥ 0.30 · max age 3ABd0d1#0#1IoU(predicted track, detection), frame 13d0d1#00.920 ✓0.235#1 (miss 3)0.2020.853 ✓id matched to each person per frame (IoU ≥ 0.5)ABempty = FN · ▌ = identity switch · frames 0–23FN 3 · FP 0 · IDSW 0 · GT 48: MOTA = 1 − (3 + 0 + 0) / 48 = 0.938IDF1 = 2·IDTP / (GT + outputs) = 2·45 / (48 + 45) = 0.968 · identities used: 2B undetected in frames 10–12 · dashed: GT (grey), prediction (colour) · thin: detections
motion model
fig.Two people (24 × 48 boxes, 6 px per frame) walk towards each other and cross. B is not detected in frames 10–12. Default: Kalman filter, IoU gate 0.3, max age 3. B's track coasts on its predictions to x = 93.3 and re-associates in frame 13 at IoU 0.853 (B is at 92). The result has 0 ID switches, MOTA 0.938 and IDF1 0.968. With max age 1 the track is deleted and B returns as #2: MOTA only drops to 0.917, but IDF1 falls to 0.753. With no motion model, B's stale box at x = 116.9 takes A's detection in frame 13, because the swapped assignment has total IoU 0.815 against 0.654 for the correct one. That gives 2 switches, MOTA 0.896 and IDF1 0.495. A 0.7 gate breaks on ordinary detector jitter: 35 switches, MOTA 0.208.

Worked example: one switch, two metrics

TrackerFNFPIDSWMOTAIDTPIDF1IDs
Kalman, gate 0.3, max age 33001 − 3/48 = 0.9384590/93 = 0.9682
Kalman, max age 13011 − 4/48 = 0.9173570/93 = 0.7533
No motion model, max age 33021 − 5/48 = 0.8962346/93 = 0.4952
Kalman, gate 0.730351 − 38/48 = 0.208510/93 = 0.10837
The figure's scenario (2 people × 24 frames = 48 GT boxes, B hidden for 3 frames, 45 detections). Computed by the figure's code and checked in Python.

IDF1's denominator is GT + predicted boxes =48+45=93=48+45=93. With max age 1, B's trajectory is split into a 10-frame and an 11-frame piece. Only one piece can map to B, so 10 correct boxes stop counting and IDF1 drops by 0.215. MOTA sees just one extra error out of 48 and drops by 0.021.

python
1from collections import defaultdict
2from ultralytics import YOLO
3
4model = YOLO("yolo11n.pt") # any detection checkpoint
5tracks = defaultdict(list)
6
7# persist=True keeps tracker state between calls; botsort.yaml adds camera-motion compensation
8for r in model.track("street.mp4", tracker="bytetrack.yaml", classes=[0], stream=True, persist=True):
9 if r.boxes.id is None: # no confirmed tracks in this frame
10 continue
11 for tid, xywh in zip(r.boxes.id.int().tolist(), r.boxes.xywh.tolist()):
12 tracks[tid].append(xywh) # centre x, y, width, height
13
14print(len(tracks), "identities;", sum(map(len, tracks.values())), "boxes")
Multi-object tracking with a YOLO detector and ByteTrack in Ultralytics (not run here)

Check yourself

Why does ByteTrack match low-score detections only to tracks left unmatched after the high-score stage?

Check yourself

A tracker has the same FN and FP counts as a second tracker, but its tracks are fragmented into twice as many IDs. Which metric shows the difference most clearly?

Remember

  • ✓SOT follows one target given its first box. SiamFC scores φ(z)⋆φ(x)\varphi(z)\star\varphi(x): a 6×6×1286\times6\times128 exemplar slid over a 22×22×12822\times22\times128 search region gives a 17×1717\times17 map.
  • ✓MOT is mostly tracking by detection: detect, predict each track with a constant-velocity Kalman filter, and match with the Hungarian algorithm on IoU (SORT, IoU_min = 0.3).
  • ✓DeepSORT adds 128-d re-ID embeddings with Mahalanobis gating and a matching cascade. ByteTrack adds a second association stage for low-score boxes.
  • ✓MOTA = 1 − (FN + FP + IDSW)/GT is dominated by detection. IDF1 and HOTA measure whether identities persist.
  • ✓In the figure, one track deletion costs 0.021 MOTA but 0.215 IDF1.
← back to Deep Learning roadmap