# Object tracking

> Tracking keeps object identities across frames. SiamFC matches a template; SORT predicts tracks with a Kalman filter and matches by IoU, DeepSORT adds appearance, ByteTrack adds low-score boxes.

Source: https://www.mlfoundry.in/learn/object-tracking  
Section: Deep Learning › Deep Learning for Computer Vision  
Level: core · 8 min read · updated 2026-10-15

**Object tracking:** estimates where each object is in every frame of a video while keeping its identity. **Single-object tracking** (SOT) follows one target given only its box in the first frame. SiamFC (Bertinetto, Valmadre, Henriques, Vedaldi and Torr, ECCV 2016 workshops) is the standard deep baseline. **Multi-object tracking** (MOT) follows every object of a class, usually by *tracking by detection*: SORT (Bewley et al., ICIP 2016), DeepSORT (Wojke, Bewley and Paulus, ICIP 2017) and ByteTrack (Zhang et al., ECCV 2022).

The output is a set of trajectories $\{(k, t, b_{k,t})\}$: an identity $k$ and a box $b$ per frame. A detector alone gives boxes but no identities, and it misses objects that are occluded, blurred or small. A tracker must link boxes over time and bridge such gaps. It must also avoid swapping identities when two objects cross, and it has to work online, frame by frame, at video rate.

Before deep features, SOT was dominated by correlation filters (MOSSE in 2010, KCF in 2015), which learn a template online in the Fourier domain. MOT relied on hand-crafted motion and appearance models with multiple-hypothesis or network-flow data association. The deep era kept the overall structure and replaced the parts with learned detectors and embeddings.

## Single object: Siamese matching

SiamFC learns a similarity function offline and does not update it while tracking. One shared fully convolutional network $\varphi$ embeds a $127\times127$ exemplar $z$ (the first-frame target) and a $255\times255$ search region $x$ centred on the last position. Their cross-correlation is a score map:

$$
f(z,x)=\varphi(z)\star\varphi(x)+b\,\mathbb 1,\qquad \underbrace{6\times6\times128}_{\varphi(z)}\ \star\ \underbrace{22\times22\times128}_{\varphi(x)}\ \to\ 17\times17
$$

- $\star$: $\varphi(z)$ slid over $\varphi(x)$ as a convolution kernel, giving one similarity per position
- $\varphi$: AlexNet-style network with total stride 8, trained with a logistic loss on frame pairs from ImageNet VID
- $b$: a learned bias; the peak, upsampled and mapped back through the stride, gives the new centre

Scale is handled by scoring the search region at a few scales. SiamRPN (Li et al., CVPR 2018) added a region-proposal head that regresses the box directly. Later trackers use deeper backbones and transformer cross-attention between template and search region (see **Cross-attention**).

## Many objects: detect, predict, associate

SORT combines an off-the-shelf detector (Faster R-CNN in the paper, see **Faster R-CNN**) with two classical tools. Each track carries a Kalman filter with state $[u,v,s,r,\dot u,\dot v,\dot s]$: the box centre, area and aspect ratio, plus the velocities of the first three, with the aspect ratio held constant. Every frame the filter predicts each track forward, and the Hungarian algorithm matches predictions to detections by IoU.

$$
\hat x_{t\mid t-1}=F\hat x_{t-1},\quad P_{t\mid t-1}=FP_{t-1}F^\top+Q;\qquad K=P_{t\mid t-1}H^\top\big(HP_{t\mid t-1}H^\top+R\big)^{-1},\quad \hat x_t=\hat x_{t\mid t-1}+K\big(z_t-H\hat x_{t\mid t-1}\big)
$$

- $F$: constant-velocity transition: position += velocity each frame
- $Q,\ R$: process and measurement noise; a large initial velocity variance lets the first matches set the velocity
- $K$: Kalman gain: how far the matched detection z_t pulls the prediction

$$
\pi^\star=\arg\max_{\pi}\sum_{(i,j)\in\pi}\mathrm{IoU}\big(H\hat x^{(i)}_{t\mid t-1},\,d_j\big),\qquad \text{reject pairs with }\mathrm{IoU}<\mathrm{IoU}_{\min}=0.3
$$

- $\pi$: one-to-one assignment of tracks i to detections j (Hungarian algorithm on cost 1 − IoU)
- $d_j$: unmatched detections start new tracks; unmatched tracks are deleted after a few missed frames (1 in SORT)

> Figure: One frame of tracking by detection. Three Kalman-predicted tracks (dashed) meet four detections. ByteTrack first matches the high-score boxes d1 (0.92) and d2 (0.81) to all tracks: IoU 0.727 with T1 and 0.726 with T2, while d2's 0.146 overlap with T3 loses. The leftover T3 is then matched to the low-score, partly occluded d3 (0.35) at IoU 0.509. The clutter box d4 (0.12) matches nothing and is dropped. SORT would apply one stage to all boxes above its detector threshold. Bottom: SiamFC correlates a 6×6×128 exemplar embedding with a 22×22×128 search embedding to give a 17×17 score map.

SORT ran at 260 Hz excluding detection, but motion and IoU alone fail when objects are occluded for longer than the Kalman prediction stays accurate. **DeepSORT** adds a 128-d appearance embedding from a re-identification CNN trained on the MARS person dataset. The cost combines a Mahalanobis distance from the Kalman filter, used as a gate at the 95% $\chi^2$ quantile (9.4877 for 4 degrees of freedom), with the smallest cosine distance to the track's last 100 embeddings. A matching cascade gives priority to recently seen tracks, and lost tracks are kept for up to 30 frames.

**ByteTrack** observed that discarding low-confidence detections throws away occluded objects. It splits detections at a score of 0.6. High boxes are matched first. The remaining tracks are then matched by IoU to low boxes above 0.1, which recovers partly hidden objects, and unmatched low boxes are discarded as background. New tracks start only from unmatched high boxes. With a YOLOX detector, it reached 80.3 MOTA, 77.3 IDF1 and 63.1 HOTA on the MOT17 test set at 30 FPS.

## Metrics: detection versus identity

$$
\mathrm{MOTA}=1-\frac{\sum_t(\mathrm{FN}_t+\mathrm{FP}_t+\mathrm{IDSW}_t)}{\sum_t\mathrm{GT}_t},\qquad \mathrm{IDF1}=\frac{2\,\mathrm{IDTP}}{2\,\mathrm{IDTP}+\mathrm{IDFP}+\mathrm{IDFN}},\qquad \mathrm{HOTA}_\alpha=\sqrt{\mathrm{DetA}_\alpha\cdot\mathrm{AssA}_\alpha}
$$

- $\mathrm{IDSW}$: frames where a ground-truth object's matched track ID changes (CLEAR MOT, Bernardin and Stiefelhagen, 2008)
- $\mathrm{IDTP}$: box matches under the single best one-to-one mapping between whole GT and predicted trajectories (Ristani et al., 2016); 2·IDTP + IDFP + IDFN = GT boxes + predicted boxes
- $\mathrm{HOTA}$: detection accuracy times association accuracy, square-rooted and averaged over IoU thresholds α = 0.05 … 0.95 (Luiten et al., IJCV 2021)

MOTA counts every error as one event per frame. Because missed and false boxes usually outnumber identity switches, MOTA mostly measures the detector. IDF1 asks how long each object keeps one ID, so it punishes switches and fragmentation. HOTA balances the two.

**Benchmarks and reported numbers**

- **MOT17:** 14 pedestrian sequences (7 train, 7 test), each released with public detections from DPM, Faster R-CNN and SDP. MOT20 adds very crowded scenes.
- **SOT benchmarks:** OTB, VOT and LaSOT: one target per sequence, given in the first frame; scored by overlap success and precision, or by accuracy and robustness in VOT
- **SORT:** Kalman filter + Hungarian on IoU, IoU_min = 0.3; 260 Hz for the tracking step alone
- **ByteTrack (YOLOX-X):** MOT17 test: 80.3 MOTA, 77.3 IDF1, 63.1 HOTA, 30 FPS

> Figure: Two people (24 × 48 boxes, 6 px per frame) walk towards each other and cross. B is not detected in frames 10–12. Default: Kalman filter, IoU gate 0.3, max age 3. B's track coasts on its predictions to x = 93.3 and re-associates in frame 13 at IoU 0.853 (B is at 92). The result has 0 ID switches, MOTA 0.938 and IDF1 0.968. With max age 1 the track is deleted and B returns as #2: MOTA only drops to 0.917, but IDF1 falls to 0.753. With no motion model, B's stale box at x = 116.9 takes A's detection in frame 13, because the swapped assignment has total IoU 0.815 against 0.654 for the correct one. That gives 2 switches, MOTA 0.896 and IDF1 0.495. A 0.7 gate breaks on ordinary detector jitter: 35 switches, MOTA 0.208.

## Worked example: one switch, two metrics

The figure's scenario (2 people × 24 frames = 48 GT boxes, B hidden for 3 frames, 45 detections). Computed by the figure's code and checked in Python.

| Tracker | FN | FP | IDSW | MOTA | IDTP | IDF1 | IDs |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Kalman, gate 0.3, max age 3 | 3 | 0 | 0 | 1 − 3/48 = 0.938 | 45 | 90/93 = 0.968 | 2 |
| Kalman, max age 1 | 3 | 0 | 1 | 1 − 4/48 = 0.917 | 35 | 70/93 = 0.753 | 3 |
| No motion model, max age 3 | 3 | 0 | 2 | 1 − 5/48 = 0.896 | 23 | 46/93 = 0.495 | 2 |
| Kalman, gate 0.7 | 3 | 0 | 35 | 1 − 38/48 = 0.208 | 5 | 10/93 = 0.108 | 37 |

IDF1's denominator is GT + predicted boxes $=48+45=93$. With max age 1, B's trajectory is split into a 10-frame and an 11-frame piece. Only one piece can map to B, so 10 correct boxes stop counting and IDF1 drops by 0.215. MOTA sees just one extra error out of 48 and drops by 0.021.

```python
from collections import defaultdict
from ultralytics import YOLO

model = YOLO("yolo11n.pt")                          # any detection checkpoint
tracks = defaultdict(list)

# persist=True keeps tracker state between calls; botsort.yaml adds camera-motion compensation
for r in model.track("street.mp4", tracker="bytetrack.yaml", classes=[0], stream=True, persist=True):
    if r.boxes.id is None:                          # no confirmed tracks in this frame
        continue
    for tid, xywh in zip(r.boxes.id.int().tolist(), r.boxes.xywh.tolist()):
        tracks[tid].append(xywh)                    # centre x, y, width, height

print(len(tracks), "identities;", sum(map(len, tracks.values())), "boxes")
```

Multi-object tracking with a YOLO detector and ByteTrack in Ultralytics (not run here)

> **Do not tune for MOTA alone:** A tracker can gain MOTA by reporting more boxes, such as coasting tracks through occlusion, while fragmenting identities. Report IDF1 or HOTA alongside it. Also state whether you used public or private detections, since detector quality dominates every MOT metric.

> **The motion model does the identity work:** In the figure the detector is identical across settings, and only the prediction step changes. A constant-velocity Kalman filter keeps a hidden track in the right place, so IoU alone can re-associate it. Appearance embeddings (DeepSORT) take over when occlusions outlast the motion model or the camera moves.

**Question:** Why does ByteTrack match low-score detections only to tracks left unmatched after the high-score stage?

- A. Low-score boxes are often occluded objects, and checking them against existing tracks keeps them while rejecting background boxes that match nothing
- B. Low-score boxes have more accurate coordinates
- C. The Hungarian algorithm cannot handle more than one score range at a time
- D. So that new tracks can be started from low-score boxes

**Answer:** A. Low-score boxes are often occluded objects, and checking them against existing tracks keeps them while rejecting background boxes that match nothing — A partly hidden person gets a low detector score but still overlaps their predicted track, so the second stage recovers them. Background clutter rarely overlaps a live track and is discarded. New tracks are started only from high-score boxes, which avoids spawning tracks from clutter.

**Question:** A tracker has the same FN and FP counts as a second tracker, but its tracks are fragmented into twice as many IDs. Which metric shows the difference most clearly?

- A. MOTA
- B. IDF1
- C. Detector AP
- D. Frames per second

**Answer:** B. IDF1 — MOTA adds one error per identity switch, which is small next to FN + FP. IDF1 maps each GT trajectory to a single predicted ID, so every fragment beyond the first stops counting as IDTP.

**Key points**

- SOT follows one target given its first box. SiamFC scores $\varphi(z)\star\varphi(x)$: a $6\times6\times128$ exemplar slid over a $22\times22\times128$ search region gives a $17\times17$ map.
- MOT is mostly tracking by detection: detect, predict each track with a constant-velocity Kalman filter, and match with the Hungarian algorithm on IoU (SORT, IoU_min = 0.3).
- DeepSORT adds 128-d re-ID embeddings with Mahalanobis gating and a matching cascade. ByteTrack adds a second association stage for low-score boxes.
- MOTA = 1 − (FN + FP + IDSW)/GT is dominated by detection. IDF1 and HOTA measure whether identities persist.
- In the figure, one track deletion costs 0.021 MOTA but 0.215 IDF1.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
