DETR

DETR (Carion et al., 2020) treats detection as set prediction: a CNN + transformer turns 100 learned queries into 100 boxes, Hungarian-matched one-to-one to the ground truth, so it needs no anchors or NMS; 42.0 COCO AP after 500 epochs.

8 min read822 wordsadvanced0 viewsupdated 2026-10-15

Definition

DETR (DEtection TRansformer; Carion, Massa, Synnaeve, Usunier, Kirillov and Zagoruyko, "End-to-End Object Detection with Transformers", ECCV 2020) is an object detector that outputs a fixed-size set of N=100N=100 predictions in one pass. A CNN backbone feeds a transformer encoder–decoder, and a bipartite (Hungarian) matching assigns each ground-truth object to exactly one prediction during training. Every other prediction is trained to say "no object" (∅\varnothing).

Input: an image. Output: a set of (class, box) pairs, with no fixed count and no order. Earlier detectors (Faster R-CNN, RetinaNet, YOLO) reach this set indirectly. They place thousands of anchors or proposals, label each one by an IoU threshold, regress offsets, and then remove duplicates with non-maximum suppression (NMS). The anchor sizes, IoU thresholds and NMS threshold are hand-tuned, and NMS is not part of the trained model.

DETR removes all of these steps. The model predicts the set directly, and the loss is defined on sets. That needs two things: a loss that does not depend on the order of the predictions, and an architecture whose outputs can talk to each other, so that two slots do not both claim the same object.

Architecture

A ResNet turns a 3×H×W3\times H\times W image into its C5 map, 2048×H32×W322048\times\frac{H}{32}\times\frac{W}{32}. A 1×11\times1 conv reduces this to d=256d=256 channels, and the map is flattened into HW/322HW/32^2 tokens: 950 for an 800×1216800\times1216 input. A 6-layer transformer encoder (8 heads, FFN width 2048) applies Self-attention over all tokens, with fixed 2-D sine positional encodings added to queries and keys in every layer.

The decoder (6 layers) takes N=100N=100 object queries, learned embeddings that act as positional codes for output slots. In each layer the queries self-attend, so slots can coordinate, and then read the image through Cross-attention to the encoder memory. All 100 slots are decoded in parallel, not autoregressively. A shared head turns each output into class probabilities over KK classes plus ∅\varnothing (linear + softmax) and a normalised box (cx,cy,w,h)∈[0,1]4(c_x,c_y,w,h)\in[0,1]^4 (3-layer MLP + sigmoid).

DETR: a set of 100 predictions from a CNN + transformer, matched 1-to-1 to ground truthcatdogimage3×800×1216CNN backboneResNet-50C5 2048×25×38stride 321×1 conv256×25×38flatten →950 × 256encoder × 6self-attentionover 950 tokens8 heads, d = 256decoder × 6self-attn: 100 queriescross-attn: 950 tokensall slots in parallel+ 2-D sine positional encoding added to Q, K in every layerobject queries100 learned × 256shared FFN headsclass: linear → softmaxover K classes + ∅box: 3-layer MLP → σ100 slots: (class, p, cx cy w h)target1cat0.97(.25,.55,.28,.56)g12∅0.98∅3dog0.94(.79,.62,.30,.44)g24cat0.31(.29,.54,.33,.58)∅5∅0.99∅…100∅0.97∅Hungarian matching (train)σ* = argmin Σ_i C(y_i, ŷ_σ(i))C = −p̂(c) + 5·L1 − 2·GIoUGT g1 cat, g2 dog; pad to100 with ∅ (no object)loss: CE + 5·L1 + 2·L_GIoU,∅ class weighted 0.1inference: keep p > threshold,no anchors, no NMSSlot 4 (a duplicate cat) is matched to ∅, so training teaches the set to suppress duplicates itself.
fig.DETR end to end. The backbone and 1×1 conv give 950 tokens of width 256 for an 800×1216 image; the encoder relates them globally; the decoder turns 100 learned queries into 100 slots in parallel; one shared head gives each slot a class (or ∅) and a box. During training a Hungarian matching assigns every ground-truth object to one slot. The duplicate cat in slot 4 is assigned ∅, so the model learns to suppress duplicates itself.

Hungarian matching and the set loss

Pad the ground-truth set yy with ∅\varnothing up to NN elements. Find the permutation of the predictions y^\hat y with the lowest total matching cost. This is a linear assignment problem, solved exactly by the Hungarian algorithm (in practice scipy.optimize.linear_sum_assignment, O(N3)O(N^3)):

σ^=arg⁡min⁡σ∈SN∑i=1NLmatch(yi,y^σ(i)),Lmatch=−1{ci≠∅} p^σ(i)(ci)+1{ci≠∅} Lbox(bi,b^σ(i))\hat\sigma=\arg\min_{\sigma\in\mathfrak S_N}\sum_{i=1}^{N}\mathcal L_{\text{match}}\big(y_i,\hat y_{\sigma(i)}\big),\qquad \mathcal L_{\text{match}}=-\mathbb 1_{\{c_i\neq\varnothing\}}\,\hat p_{\sigma(i)}(c_i)+\mathbb 1_{\{c_i\neq\varnothing\}}\,\mathcal L_{\text{box}}\big(b_i,\hat b_{\sigma(i)}\big)
SN\mathfrak S_N
all permutations of the NN prediction slots
p^σ(i)(ci)\hat p_{\sigma(i)}(c_i)
predicted probability of the true class cic_i; the probability, not its log, is used in the cost
Lbox\mathcal L_{\text{box}}
λL1∥bi−b^σ(i)∥1+λiou LGIoU\lambda_{L1}\lVert b_i-\hat b_{\sigma(i)}\rVert_1+\lambda_{\text{iou}}\,\mathcal L_{\text{GIoU}}, with λL1=5\lambda_{L1}=5, λiou=2\lambda_{\text{iou}}=2 and LGIoU=1−GIoU\mathcal L_{\text{GIoU}}=1-\text{GIoU}

With the matching fixed, the Hungarian loss is a normal detection loss on the matched pairs. Every slot gets cross-entropy on its target class. Slots matched to ∅\varnothing get no box loss, and their class term is down-weighted by 10 against class imbalance (most of the 100 slots are empty):

LHungarian(y,y^)=∑i=1N[−log⁡p^σ^(i)(ci)+1{ci≠∅} Lbox(bi,b^σ^(i))]\mathcal L_{\text{Hungarian}}(y,\hat y)=\sum_{i=1}^{N}\Big[-\log\hat p_{\hat\sigma(i)}(c_i)+\mathbb 1_{\{c_i\neq\varnothing\}}\,\mathcal L_{\text{box}}\big(b_i,\hat b_{\hat\sigma(i)}\big)\Big]
σ^\hat\sigma
the optimal assignment from the matching step; no gradient flows through it
GIoU\text{GIoU}
generalised IoU, IoU−∣C∖(A∪B)∣/∣C∣\text{IoU}-|C\setminus(A\cup B)|/|C| with CC the smallest enclosing box (see Intersection over Union (IoU) loss); scale-invariant, unlike L1

The same loss is applied after every decoder layer, through the shared head (auxiliary losses). One-to-one matching is what removes NMS: a second slot on the same object is a ∅\varnothing target, so the decoder learns to keep only one. At inference you keep the slots whose top class is not ∅\varnothing, or whose score passes a threshold.

Hungarian one-to-one matching · 3 positives, 5 → ∅g1 catg2 catg3 dogq1q2q3q4q5q6q7q8solid = ground truth · dashed = 8 of N = 100 queriescost = −1·p(c) + 5·L1 − 2·GIoUg1 catg2 catg3 dogq1→-2.421.114.32q2∅-1.681.124.35q3→1.47-2.042.86q4∅0.28-0.693.44q5→4.322.89-2.49q6∅5.473.75-0.88q7∅6.934.215.86q8∅5.586.789.15min total cost -6.952 over 336 optionseach GT gets exactly one queryg1 ← q1 (p=0.88, IoU 0.897)g2 ← q3 (p=0.52, IoU 0.887)g3 ← q5 (p=0.90, IoU 0.896)loss (sum) = CE 0.887 + box 1.348 + 0.1·∅-CE 0.534 = 2.769box loss = λ_L1·L1 + λ_GIoU·(1 − GIoU); unmatched queries are trained to predict ∅
assignment
fig.Three objects, eight of the 100 query slots. Each cell is DETR's matching cost −λ_cls·p(c) + λ_L1·L1 − λ_GIoU·GIoU. At the default weights (1, 5, 2), the minimum over all 336 one-to-one assignments is −6.952: g1←q1, g2←q3, g3←q5, and the other five slots get ∅. The summed loss is 2.769. Raise λ_cls to 5.5 and g2 switches to q4, the confident but loose box (p = 0.83, IoU 0.541). The loss rises to 3.852. Switch to anchor-style IoU ≥ 0.5 assignment and 6 slots become positives for 3 objects, the duplicates that NMS would otherwise have to remove.

Configuration and COCO val results (Carion et al., 2020)

Model
ResNet-50 + 6 encoder + 6 decoder layers, d=256d=256, 8 heads, N=100N=100 queries; 41M parameters, 86 GFLOPs, 28 FPS on a V100
Training
AdamW, lr 10−410^{-4} (transformer) and 10−510^{-5} (backbone); 300 epochs take 3 days on 16 V100s at batch 64; the 500-epoch schedule (lr drop at 400) adds 1.5 AP
DETR R50, 500 ep
42.0 AP; AP_S 20.5, AP_L 61.1
Faster R-CNN-FPN+ R50
42.0 AP; AP_S 26.6, AP_L 53.4 (a strengthened baseline with GIoU loss and a longer schedule)
Larger DETR
DC5-R50 43.3 AP (dilated C5, stride 16); R101 43.5 AP

So DETR matches a well-tuned Faster R-CNN overall, but in a different way. It is much better on large objects (+7.7 AP_L), because the encoder reasons globally. It is much worse on small ones (−6.1 AP_S), because it sees only a stride-32 map with no feature pyramid. Adding an FPN is not easy: encoder self-attention costs O((HW)2d)O((HW)^2d), and going from stride 32 to stride 16 already multiplies the tokens by 4 and the attention cost by 16.

Worked example: matching two objects

Ground truth: cat A=(0.30,0.50,0.20,0.40)A=(0.30,0.50,0.20,0.40) and dog B=(0.70,0.50,0.20,0.40)B=(0.70,0.50,0.20,0.40) in (cx,cy,w,h)(c_x,c_y,w,h). Three slots: q1=(0.32,0.50,0.20,0.40)q_1=(0.32,0.50,0.20,0.40) with p(cat)=0.90p(\text{cat})=0.90; q2=(0.50,0.50,0.40,0.40)q_2=(0.50,0.50,0.40,0.40) with p(cat)=0.60p(\text{cat})=0.60, p(dog)=0.30p(\text{dog})=0.30, a wide box over both animals; q3=(0.68,0.52,0.24,0.40)q_3=(0.68,0.52,0.24,0.40) with p(dog)=0.80p(\text{dog})=0.80. Cost =−p+5 L1−2 GIoU=-p+5\,\text{L1}-2\,\text{GIoU}:

Pairp(c)L1GIoUCost
q1 → A (cat)0.900.020.8182−0.90 + 0.10 − 1.6364 = −2.4364
q2 → A (cat)0.600.400.2000−0.60 + 2.00 − 0.40 = 1.0000
q2 → B (dog)0.300.400.2000−0.30 + 2.00 − 0.40 = 1.3000
q3 → B (dog)0.800.080.7521−0.80 + 0.40 − 1.5041 = −1.9041
q1 → B, q3 → A0.05, 0.100.38, 0.44−0.3103, −0.30162.4707, 2.7032
Matching costs (computed in Python); IoU of q1 with A = 0.072 / 0.088

Of the 6 one-to-one assignments, {q1→A, q3→B}\{q_1\to A,\ q_3\to B\} costs −2.4364−1.9041=−4.3405-2.4364-1.9041=-4.3405; the next best, {q1→A, q2→B}\{q_1\to A,\ q_2\to B\}, costs −1.1364-1.1364. So q2q_2 becomes a ∅\varnothing target, and its loss term is −0.1log⁡p^2(∅)-0.1\log\hat p_2(\varnothing). The negative GIoU of the disjoint pairs (−0.31-0.31) still says how far apart the boxes are, which plain IoU (0 for both) cannot.

python
1import torch
2from PIL import Image
3from scipy.optimize import linear_sum_assignment
4from torchvision.ops import box_convert, generalized_box_iou
5from transformers import DetrForObjectDetection, DetrImageProcessor
6
7proc = DetrImageProcessor.from_pretrained("facebook/detr-resnet-50")
8model = DetrForObjectDetection.from_pretrained("facebook/detr-resnet-50").eval()
9image = Image.open("street.jpg").convert("RGB")
10with torch.no_grad():
11 out = model(**proc(images=image, return_tensors="pt"))
12print(out.logits.shape, out.pred_boxes.shape) # (1, 100, 92) = 91 COCO ids + no-object, (1, 100, 4)
13det = proc.post_process_object_detection(out, threshold=0.9, target_sizes=[image.size[::-1]])[0]
14for s, l, b in zip(det["scores"], det["labels"], det["boxes"]): # no NMS anywhere
15 print(model.config.id2label[l.item()], f"{s:.2f}", [round(v) for v in b.tolist()])
16
17def hungarian_match(logits, boxes, tgt_labels, tgt_boxes, w_cls=1.0, w_l1=5.0, w_giou=2.0):
18 """logits (N, K+1), boxes (N, 4) cxcywh in [0, 1]; returns (pred_idx, gt_idx)."""
19 prob = logits.softmax(-1)
20 xyxy = lambda b: box_convert(b, "cxcywh", "xyxy")
21 cost = (-w_cls * prob[:, tgt_labels] # (N, M)
22 + w_l1 * torch.cdist(boxes, tgt_boxes, p=1)
23 - w_giou * generalized_box_iou(xyxy(boxes), xyxy(tgt_boxes)))
24 return linear_sum_assignment(cost.detach().cpu().numpy())
Pretrained DETR inference with Hugging Face, plus DETR's matcher (not run here: needs torch, transformers and scipy)

Use when

  • +No anchors, no proposal stage, no NMS: the loss is defined on the final output set
  • +Global reasoning in the encoder gives strong large-object AP (61.1 AP_L vs 53.4 for the R50 Faster R-CNN baseline)
  • +The same query decoder extends to panoptic segmentation with a mask head, and to later set-prediction models such as Mask2Former

Avoid when

  • −Very slow training: 500 epochs for 42.0 AP
  • −Weak on small objects (20.5 AP_S): a single stride-32 map, and attention cost is quadratic in the number of tokens
  • −N must exceed the number of objects in any image; 100 suits COCO but caps dense scenes

Check yourself

An image has 3 objects and DETR uses N = 100 queries. During training, how many slots get a box loss and how many are trained to predict ∅?

Check yourself

Why can DETR skip non-maximum suppression?

Remember

  • ✓Detection as set prediction: CNN → 950 tokens (stride 32, d=256d=256) → 6-layer encoder → 6-layer decoder over N=100N=100 learned queries → class (incl. ∅\varnothing) + box per slot.
  • ✓Hungarian matching with cost −p^(c)+5 L1−2 GIoU-\hat p(c)+5\,\text{L1}-2\,\text{GIoU} assigns each object to one slot. The loss is CE + 5 L1+2(1−GIoU)5\,\text{L1}+2(1-\text{GIoU}), with ∅\varnothing weighted 0.1, applied at every decoder layer.
  • ✓No anchors and no NMS: duplicates are trained to predict ∅.
  • ✓R50, 500 epochs: 42.0 AP, the same as a strong Faster R-CNN, with +7.7 AP_L but −6.1 AP_S.
  • ✓Main weaknesses: slow convergence and small objects. Deformable DETR addresses both with sparse multi-scale attention.
← back to Deep Learning roadmap