Definition
Deformable DETR (Zhu, Su, Lu, Li, Wang and Dai, "Deformable DETR: Deformable Transformers for End-to-End Object Detection", ICLR 2021) is a DETR-style set-prediction detector in which every attention module over image features is replaced by multi-scale deformable attention. A query no longer compares itself with every pixel. It predicts a few sampling offsets around a reference point, reads the features there by bilinear interpolation, and averages them with weights it also predicts.
DETR showed that detection can be end to end, with Hungarian matching and no anchors or NMS. It had two problems. It needs 500 epochs on COCO, and its small-object AP is weak (20.5 AP_S). Both come from dense attention. At initialisation each query spreads its attention almost evenly over all pixels, and learning to focus on a few takes many epochs. The encoder cost is with , quadratic in the number of pixels, so DETR can only afford the stride-32 map. High-resolution, multi-scale features are exactly what small objects need.
The fix borrows from Deformable convolution: sample a small set of learned, input-dependent locations instead of a fixed or exhaustive set. Deformable attention keeps the transformer's multi-head form and applies this idea to it.
Multi-scale deformable attention
Take query with feature and a normalised reference point . In the encoder, every pixel of every level is a query and its reference point is its own position. In the decoder, is predicted from the object query by a linear layer and a sigmoid. With feature maps :
- heads, feature levels, sampling points per head per level; , ,
- 2-D sampling offset in level- pixels, from a linear layer on ( outputs); fractional, so is read by bilinear interpolation
- attention weight from another linear layer on , softmax-normalised over the points of head ; there is no query–key dot product
- rescales the normalised to the coordinates of level , so one query samples the same image region on every scale
- output and value projections of head , as in standard Multi-head attention
Each query reads values in total, whatever the image size. The paper gives the cost as for one level. In the encoder () this is , linear in the number of pixels. In the decoder's cross-attention it does not depend on at all. The decoder's self-attention among the object queries stays dense: it is cheap, and it lets queries suppress duplicates.
The rest of the detector
The levels share one transformer, so a learned level embedding is added to the positional encoding to say which scale a token comes from. Sampling points of one query span all levels, so information already moves across scales. The paper found that adding an FPN gave no further gain. The decoder uses queries instead of 100, and focal loss (weight 2) for classification. Each box is predicted relative to the query's reference point: for the centre. Hungarian matching and the L1 + GIoU box loss are kept from DETR.
Two variants improve accuracy. Iterative bounding-box refinement: each decoder layer refines the box of the previous layer and uses it as the new reference, so later layers sample inside the current box estimate. Two-stage: the encoder output first acts as a dense proposal generator, and the top-scoring proposals become the decoder's object queries.
COCO val2017, ResNet-50 (Zhu et al., 2021)
- DETR, 500 epochs
- 42.0 AP, AP_S 20.5; 41M params, 86 GFLOPs
- DETR-DC5, 500 / 50 epochs
- 43.3 AP at 500 epochs; only 35.3 AP when trained for 50
- Deformable DETR, 50 epochs
- 43.8 AP, AP_S 26.4; 40M params, 173 GFLOPs
- + iterative box refinement
- 45.4 AP
- ++ two-stage
- 46.2 AP, AP_S 28.8
- Defaults
- heads, levels, points, 6 + 6 layers, , 300 queries
So in a tenth of the epochs it beats DETR overall, by 5.9 AP on small objects. FLOPs are higher than DETR's, because it processes the stride-8 map. They are lower than DETR-DC5's 187 G, which works at stride 16 only.
Worked example: one head, one level, K = 2
Sampling C5 (25×38, from an 800×1216 image) for a query at p̂ = (0.40, 0.56)
- 1Scale the reference point to level pixels: .
- 2Point 1, offset : location , normalised . Cell centres sit at , so the grid index is . That is exactly column 16, of the way from row 12 to row 13. With and : .
- 3Point 2, offset : location , index , halfway between columns 12 and 13 on row 14. With values and : .
- 4Attention logits from the linear layer: softmax gives .
- 5Head output before : . Gradients reach through the bilinear weights and reach directly, so both are learned from the detection loss.
| Features | Tokens | Dense pairs (N²) | Deformable (16 per query) | Ratio |
|---|---|---|---|---|
| C5 only (DETR) | 25×42 = 1,050 | 1,102,500 | 16,800 | 66× |
| dilated C5, stride 16 (DETR-DC5) | 50×84 = 4,200 | 17,640,000 | 67,200 | 262.5× |
| C3–C5 + C6 (Deformable DETR) | 22,223 | 493,861,729 | 355,568 | 1,389× |
1import torch2import torch.nn.functional as F34def ms_deform_attn(value, shapes, loc, attn):5 """value: (B, S, M, D) all levels flattened, S = sum(H_l * W_l)6 shapes: [(H_l, W_l)] * L; loc: (B, Q, M, L, K, 2) normalised (x, y) in [0, 1]7 attn: (B, Q, M, L, K), softmax-ed over L*K. Returns (B, Q, M*D)."""8 B, S, M, D = value.shape9 _, Q, _, L, K, _ = loc.shape10 grids = 2 * loc - 1 # grid_sample wants [-1, 1]11 sampled = []12 for l, v in enumerate(value.split([h * w for h, w in shapes], dim=1)):13 h, w = shapes[l]14 v = v.flatten(2).transpose(1, 2).reshape(B * M, D, h, w)15 g = grids[:, :, :, l].transpose(1, 2).flatten(0, 1) # (B*M, Q, K, 2)16 sampled.append(F.grid_sample(v, g, mode="bilinear", padding_mode="zeros", align_corners=False))17 a = attn.transpose(1, 2).reshape(B * M, 1, Q, L * K)18 out = (torch.stack(sampled, dim=-2).flatten(-2) * a).sum(-1) # (B*M, D, Q)19 return out.view(B, M * D, Q).transpose(1, 2)2021shapes = [(100, 167), (50, 84), (25, 42), (13, 21)] # 800x1333 input, strides 8..6422S, M, D, L, K, Q = sum(h * w for h, w in shapes), 8, 32, 4, 4, 30023z = torch.randn(1, Q, 256) # decoder queries24ref = torch.rand(1, Q, 1, 1, 1, 2) # reference points25offsets = torch.nn.Linear(256, M * L * K * 2)(z).view(1, Q, M, L, K, 2)26norm = torch.tensor([[w, h] for h, w in shapes], dtype=torch.float).view(1, 1, 1, L, 1, 2)27loc = ref + offsets / norm # offsets are in level pixels28attn = torch.nn.Linear(256, M * L * K)(z).view(1, Q, M, L * K).softmax(-1).view(1, Q, M, L, K)29out = ms_deform_attn(torch.randn(1, S, M, D), shapes, loc, attn)30print(S, out.shape) # 22223 torch.Size([1, 300, 256])Check yourself
In deformable attention, how are the attention weights A_mlqk computed?
Check yourself
Why can Deformable DETR use the stride-8 map when DETR could not?
Remember
- ✓Each query samples points per head on each of levels (strides 8–64) around its reference point: 128 bilinear reads, weights and offsets both predicted from the query.
- ✓Encoder cost becomes linear in , so multi-scale, high-resolution features fit, and no FPN is needed.
- ✓43.8 COCO AP in 50 epochs vs DETR's 42.0 in 500; AP_S 26.4 vs 20.5; 46.2 AP with box refinement and two stages.
- ✓300 queries, focal loss, boxes relative to reference points; Hungarian matching and L1 + GIoU are kept from DETR.
- ✓The same multi-scale deformable attention became a standard pixel decoder and encoder block in later detectors and segmenters, such as Mask2Former.