Deformable convolution

A convolution whose sampling grid bends to the input: a side conv predicts a 2-D fractional offset for every kernel tap at every location, values are read by bilinear interpolation, and the offsets are trained end-to-end. DCNv2 adds a learned weight per tap.

13 min read863 wordsadvanced0 viewsupdated 2026-10-11

Definition

Deformable convolution is a convolution in which tap kk of the kernel, at output location p0p_0, samples the input at p0+pk+Δpkp_0+p_k+\Delta p_k instead of the fixed grid position p0+pkp_0+p_k. The offsets Δpk∈R2\Delta p_k\in\mathbb R^2 are produced by an ordinary convolution on the same input, so they differ per location and per image. Fractional positions are read by bilinear interpolation, which makes the output differentiable in the offsets.

A standard convolution samples a rigid rectangle. Pooling, striding and Dilated convolution change its size, but every location in a layer still uses the same shape. Objects do not cooperate: they appear at different scales, poses and deformations, and the useful evidence for a pixel on a cat's tail lies along a curve, not inside a square. Data augmentation can teach a rigid CNN some invariance, but only by spending capacity to memorise many transformed copies.

Dai, Qi, Xiong, Li, Zhang, Hu and Wei introduced deformable convolution and deformable RoI pooling in "Deformable Convolutional Networks" (ICCV 2017, DCNv1). The geometry becomes something the network learns from the task loss, with no extra supervision. Zhu, Hu, Lin and Dai's "Deformable ConvNets v2: More Deformable, Better Results" (CVPR 2019) added a learned modulation scalar per tap, used deformable layers in more stages of the backbone, and trained with feature mimicking from an R-CNN teacher. Wang et al.'s InternImage (CVPR 2023) redesigned the operator as DCNv3 to build large foundation backbones, and Xiong et al.'s DCNv4 (CVPR 2024) made it much faster.

Formula

y(p0)=∑k=1Kwk  mk  x(p0+pk+Δpk),x(q)=∑pG(q,p) x(p),G(q,p)=g(qy,py) g(qx,px),  g(a,b)=max⁡(0, 1−∣a−b∣)y(p_0)=\sum_{k=1}^{K} w_k\;m_k\;x\big(p_0+p_k+\Delta p_k\big),\qquad x(q)=\sum_{p} G(q,p)\,x(p),\quad G(q,p)=g(q_y,p_y)\,g(q_x,p_x),\ \ g(a,b)=\max(0,\,1-|a-b|)
pkp_k
the regular grid offsets, e.g. {−1,0,1}2\{-1,0,1\}^2 for a 3×33\times3 kernel (times the dilation)
Δpk\Delta p_k
learned fractional offset (Δyk,Δxk)(\Delta y_k,\Delta x_k) of tap kk at location p0p_0; 2K2K numbers per location
mkm_k
DCNv2 modulation scalar σ(uk)∈(0,1)\sigma(u_k)\in(0,1), also predicted per location; DCNv1 is mk=1m_k=1
wkw_k
the ordinary kernel weights (a Cout×CinC_{out}\times C_{in} matrix per tap), shared over all locations
GG
bilinear kernel: only the 4 integer pixels around qq have non-zero weight

The offsets come from a separate convolution with the same kernel size, stride and dilation as the deformable layer, applied to the same input. It outputs 2K2K channels (2⋅3⋅3=182\cdot3\cdot3=18 for a 3×33\times3 kernel), or 3K=273K=27 for DCNv2 (18 offsets plus 9 mask logits). Both papers initialise this conv to zero, so training starts from an ordinary convolution (Δp=0\Delta p=0) and, in DCNv2, from mk=σ(0)=0.5m_k=\sigma(0)=0.5.

Gradients through the sampling position

The kernel weights get the usual gradient, with xx replaced by the interpolated samples. The new part is the gradient with respect to the offsets. Bilinear interpolation is piecewise linear in qq, so its derivative is a difference of the four neighbouring pixels:

∂y(p0)∂Δxk=wk mk ∂x(qk)∂qx,∂x(q)∂qx=(1−ay)(x01−x00)+ay(x11−x10),∂y(p0)∂mk=wk x(qk)\frac{\partial y(p_0)}{\partial \Delta x_k}=w_k\,m_k\,\frac{\partial x(q_k)}{\partial q_x},\qquad \frac{\partial x(q)}{\partial q_x}=(1-a_y)\big(x_{01}-x_{00}\big)+a_y\big(x_{11}-x_{10}\big),\qquad \frac{\partial y(p_0)}{\partial m_k}=w_k\,x(q_k)
x00,x01,x10,x11x_{00},x_{01},x_{10},x_{11}
the four pixels around qq: top-left, top-right, bottom-left, bottom-right
ay, axa_y,\ a_x
fractional parts of qyq_y and qxq_x; ∂x(q)/∂qy\partial x(q)/\partial q_y is the same formula with the roles swapped

So a tap moves toward the side where the input changes in the direction that lowers the loss. The gradient is local: it only sees the 2×2 cell the tap sits in, which is why offsets start at zero and are trained with the rest of the network rather than searched globally. When qq lands exactly on a pixel the derivative is one-sided; implementations just use the formula above.

At a glance

Extra parameters
the offset conv: K⋅Cin⋅2KK\cdot C_{in}\cdot 2K weights (+ K⋅Cin⋅KK\cdot C_{in}\cdot K for DCNv2 masks); e.g. 9⋅256⋅27=62,2089\cdot256\cdot27=62{,}208 next to 589,824589{,}824 for a 256→256 3×3 layer
Offset tensor (torchvision)
shape (B, 2 Goff khkw, Hout, Wout)(B,\ 2\,G_{\text{off}}\,k_h k_w,\ H_{out},\ W_{out}); for tap k=i kw+jk=i\,k_w+j, channel 2k2k is Δy\Delta y and 2k+12k+1 is Δx\Delta x
Mask tensor
shape (B, Goff khkw, Hout, Wout)(B,\ G_{\text{off}}\,k_h k_w,\ H_{out},\ W_{out}), values usually σ(⋅)\sigma(\cdot)
Init
offset/mask conv weights and bias = 0 (starts as a plain conv, masks 0.5)
Typical placement
the 3×33\times3 convs of the last stages of a ResNet (conv3–conv5 in DCNv2), in detectors and segmenters
Cost
a gather with bilinear weights per tap: memory-bound and slower than a dense conv of the same FLOPs
p₀ = (20, 20) · click the map to movestep 0 / 80 · J = -0.010regular grid y = 0.249DCNv1 y = Σx(qₖ)/9 = 0.249kΔy chΔx chmₖx(qₖ)0 0.00 0 0.00 110.2231 0.00 2 0.00 310.0842 0.00 4 0.00 510.4023 0.00 6 0.00 710.0844 0.00 8 0.00 910.0075 0.00 10 0.00 1110.2236 0.00 12 0.00 1310.4027 0.00 14 0.00 1510.2238 0.00 16 0.00 1710.591tap 2: q = (17.00, 23.00) · bilinear over 2×2(17,23) 0.402 × 1.000(17,24) 0.651 × 0.000(18,23) 0.280 × 0.000(18,24) 0.529 × 0.000= 0.402 · ∂/∂y -0.122 · ∂/∂x 0.249offset tensor: 18 ch = (Δy, Δx) per tapascent on J = Σ(x(qₖ) − 0.25) − λ/2·Σ‖Δpₖ‖²○ regular grid ● sampled qₖ = p₀ + pₖ + Δpₖ □ 2×2 cell of tap 2illustration: offsets fitted by gradient ascent; in a network a conv layer predicts them
fig.One output location p₀ = (20, 20) of a 3×3 deformable conv on a drawn 40×40 ring (regular grid spacing 3, hollow circles). Here the 18 offsets are fitted directly by gradient ascent on J = Σ mₖ(x(qₖ) − 0.25) − λ/2·Σ‖Δpₖ‖², so you can watch the gradient path in isolation; in a network a conv layer would predict them. Press fit offsets: with λ = 0.02 the mean sampled value rises from 0.249 on the regular grid to 0.966 after 80 steps as the taps slide onto the ring. With DCNv2 masks on, the centre tap, which sits in the empty middle, moves only 0.28 px and switches itself off (m = 0.14) instead of travelling 5.8 px as in DCNv1. Raise λ to 0.2 and the taps stay close to the grid (response 0.518). The table lists each tap's Δy/Δx with its torchvision channel index (2k, 2k+1) and the 2×2 bilinear read of the selected tap; click the map to move p₀ or switch to the curve and bar shapes.

Worked example: one tap

Let the input contain x(1,1)=8, x(1,2)=6, x(2,1)=3, x(2,2)=9x(1,1)=8,\ x(1,2)=6,\ x(2,1)=3,\ x(2,2)=9 and let tap kk land at q=(1.6, 1.75)q=(1.6,\,1.75) (for instance grid position (1,2)(1,2) plus offset Δpk=(0.6, −0.25)\Delta p_k=(0.6,\,-0.25)). Then y0=1, x0=1y_0=1,\ x_0=1, ay=0.6a_y=0.6, ax=0.75a_x=0.75:

neighbourvalueweightcontribution
(1, 1)80.4⋅0.25=0.100.4\cdot0.25=0.100.80
(1, 2)60.4⋅0.75=0.300.4\cdot0.75=0.301.80
(2, 1)30.6⋅0.25=0.150.6\cdot0.25=0.150.45
(2, 2)90.6⋅0.75=0.450.6\cdot0.75=0.454.05
x(q)x(q)1.007.10
Bilinear read and its gradient (verified with the figure's library)
  1. 1∂x(q)/∂qy=(1−0.75)(3−8)+0.75(9−6)=−1.25+2.25=1.0\partial x(q)/\partial q_y=(1-0.75)(3-8)+0.75(9-6)=-1.25+2.25=1.0.
  2. 2∂x(q)/∂qx=(1−0.6)(6−8)+0.6(9−3)=−0.8+3.6=2.8\partial x(q)/\partial q_x=(1-0.6)(6-8)+0.6(9-3)=-0.8+3.6=2.8.
  3. 3With a scalar weight wk=0.5w_k=0.5, mk=1m_k=1 and upstream gradient ∂L/∂y=1\partial L/\partial y=1: ∂L/∂Δyk=0.5\partial L/\partial\Delta y_k=0.5, ∂L/∂Δxk=1.4\partial L/\partial\Delta x_k=1.4, ∂L/∂wk=7.1\partial L/\partial w_k=7.1. These flow back into the offset conv's weights.

Deformable RoI pooling

For two-stage detectors, DCNv1 also deforms RoI pooling. The region is split into K×KK\times K bins as usual; a fully connected layer on the pooled features predicts a normalised offset Δp^ij\Delta\hat p_{ij} per bin, which is scaled by the RoI's width and height (and a factor γ=0.1\gamma=0.1) so the offsets are independent of the RoI size. Each bin is then average-pooled over bilinearly sampled points at its shifted position. A position-sensitive variant does the same for R-FCN. DCNv2 adds a modulation scalar per bin here too.

DCNv2, v3 and v4

Where it's used

DCNv2 (Zhu et al., 2019)

Adds mkm_k, so a tap can not only move but also fade out: the layer can shrink its support or ignore background without contorting the offsets. Stacks deformable layers in all 3×33\times3 convs of ResNet stages conv3–conv5, and distils from an R-CNN on cropped regions ("feature mimicking") so the support focuses on the object.

DCNv3 / InternImage (Wang et al., 2023)

Separates the per-tap weights into a depthwise-like shared projection plus point-wise mixing (as in Depthwise separable convolution), splits channels into groups that each predict their own offsets, and normalises the modulation scalars with a softmax over the KK taps instead of independent sigmoids. InternImage-H reported 65.4 box AP on COCO test-dev and 62.9 mIoU on ADE20K.

DCNv4 (Xiong et al., 2024)

Drops the softmax normalisation (letting the aggregation weights be unbounded, like a convolution's) and rewrites the kernel for memory efficiency, making the operator several times faster than DCNv3.

python
1import torch, torch.nn as nn, torch.nn.functional as F
2from torchvision.ops import DeformConv2d, deform_conv2d
3
4class ModulatedDeformConv(nn.Module):
5 """DCNv2: offsets (2*k*k channels) and masks (k*k channels) predicted from the input."""
6 def __init__(self, cin, cout, k=3, stride=1, padding=1, dilation=1):
7 super().__init__()
8 self.k = k
9 self.pred = nn.Conv2d(cin, 3 * k * k, k, stride, padding, dilation)
10 nn.init.zeros_(self.pred.weight) # start as a plain conv: offsets 0, masks 0.5
11 nn.init.zeros_(self.pred.bias)
12 self.dcn = DeformConv2d(cin, cout, k, stride, padding, dilation)
13
14 def forward(self, x):
15 o = self.pred(x)
16 offset = o[:, : 2 * self.k * self.k] # channel 2k = dy, 2k+1 = dx of tap k = i*kw + j
17 mask = o[:, 2 * self.k * self.k :].sigmoid()
18 return self.dcn(x, offset, mask)
19
20m = ModulatedDeformConv(16, 32)
21x = torch.randn(2, 16, 32, 32)
22print(m(x).shape, m.pred(x).shape) # torch.Size([2, 32, 32, 32]) torch.Size([2, 27, 32, 32])
23
24# Sanity checks with the functional op
25w = torch.randn(32, 16, 3, 3)
26zero = torch.zeros(2, 18, 32, 32)
27print(torch.allclose(deform_conv2d(x, zero, w, padding=1), F.conv2d(x, w, padding=1), atol=1e-4)) # True
28
29# Moving every tap by exactly +1 in x (channels 1, 3, 5, ...) = reading the input one column to the right
30shift = torch.zeros(2, 18, 32, 32); shift[:, 1::2] = 1.0
31xs = F.pad(x, (0, 2, 1, 1)) # pad left 0, right 2, top 1, bottom 1, then a 'valid' conv
32print(torch.allclose(deform_conv2d(x, shift, w, padding=1), F.conv2d(xs, w), atol=1e-4)) # True
DCNv2 with torchvision.ops: zero-initialised offset/mask predictor and the offset layout

Use when

  • +Learns spatial adaptation (scale, pose, deformation) from the task loss, no extra labels
  • +Drop-in replacement for 3×3 convs; starts identical to them with zero-init
  • +Consistent gains in detection and segmentation, where geometry varies most
  • +DCNv2 masks let taps switch off irrelevant context

Avoid when

  • −Irregular memory access: slow on GPUs without custom kernels, awkward for export and edge runtimes
  • −Offset gradients are local (one 2×2 cell), so training can be unstable without zero-init and a small learning rate
  • −Extra hyperparameters: where to place layers, offset groups, mask on/off
  • −Library-specific offset layouts complicate weight porting

Check yourself

A torchvision DeformConv2d with a 3 × 3 kernel and one offset group runs on a 64 × 64 output. What is the shape of the offset tensor for a batch of 2?

Check yourself

Why does deformable convolution use bilinear interpolation rather than rounding the sampling position to the nearest pixel?

Remember

  • ✓Each tap samples at p0+pk+Δpkp_0+p_k+\Delta p_k; the offsets are predicted per location by a conv with 2K2K outputs (18 for 3×33\times3).
  • ✓Bilinear interpolation makes the samples differentiable in Δp\Delta p; the gradient comes from the 4 neighbouring pixels.
  • ✓DCNv2 adds a modulation mk=σ(uk)m_k=\sigma(u_k) per tap (27 predicted channels); zero-init gives Δp=0\Delta p=0, m=0.5m=0.5.
  • ✓torchvision: deform_conv2d(input, offset, weight, mask=...), offset channels (2k,2k+1)=(Δy,Δx)(2k,2k+1)=(\Delta y,\Delta x) of tap kk.
  • ✓DCNv3 (InternImage) shares weights across taps, groups the offsets and softmax-normalises the masks; DCNv4 drops the softmax and speeds it up.
← back to Deep Learning roadmap