Definition
Deformable convolution is a convolution in which tap of the kernel, at output location , samples the input at instead of the fixed grid position . The offsets are produced by an ordinary convolution on the same input, so they differ per location and per image. Fractional positions are read by bilinear interpolation, which makes the output differentiable in the offsets.
A standard convolution samples a rigid rectangle. Pooling, striding and Dilated convolution change its size, but every location in a layer still uses the same shape. Objects do not cooperate: they appear at different scales, poses and deformations, and the useful evidence for a pixel on a cat's tail lies along a curve, not inside a square. Data augmentation can teach a rigid CNN some invariance, but only by spending capacity to memorise many transformed copies.
Dai, Qi, Xiong, Li, Zhang, Hu and Wei introduced deformable convolution and deformable RoI pooling in "Deformable Convolutional Networks" (ICCV 2017, DCNv1). The geometry becomes something the network learns from the task loss, with no extra supervision. Zhu, Hu, Lin and Dai's "Deformable ConvNets v2: More Deformable, Better Results" (CVPR 2019) added a learned modulation scalar per tap, used deformable layers in more stages of the backbone, and trained with feature mimicking from an R-CNN teacher. Wang et al.'s InternImage (CVPR 2023) redesigned the operator as DCNv3 to build large foundation backbones, and Xiong et al.'s DCNv4 (CVPR 2024) made it much faster.
Formula
- the regular grid offsets, e.g. for a kernel (times the dilation)
- learned fractional offset of tap at location ; numbers per location
- DCNv2 modulation scalar , also predicted per location; DCNv1 is
- the ordinary kernel weights (a matrix per tap), shared over all locations
- bilinear kernel: only the 4 integer pixels around have non-zero weight
The offsets come from a separate convolution with the same kernel size, stride and dilation as the deformable layer, applied to the same input. It outputs channels ( for a kernel), or for DCNv2 (18 offsets plus 9 mask logits). Both papers initialise this conv to zero, so training starts from an ordinary convolution () and, in DCNv2, from .
Gradients through the sampling position
The kernel weights get the usual gradient, with replaced by the interpolated samples. The new part is the gradient with respect to the offsets. Bilinear interpolation is piecewise linear in , so its derivative is a difference of the four neighbouring pixels:
- the four pixels around : top-left, top-right, bottom-left, bottom-right
- fractional parts of and ; is the same formula with the roles swapped
So a tap moves toward the side where the input changes in the direction that lowers the loss. The gradient is local: it only sees the 2×2 cell the tap sits in, which is why offsets start at zero and are trained with the rest of the network rather than searched globally. When lands exactly on a pixel the derivative is one-sided; implementations just use the formula above.
At a glance
- Extra parameters
- the offset conv: weights (+ for DCNv2 masks); e.g. next to for a 256→256 3×3 layer
- Offset tensor (torchvision)
- shape ; for tap , channel is and is
- Mask tensor
- shape , values usually
- Init
- offset/mask conv weights and bias = 0 (starts as a plain conv, masks 0.5)
- Typical placement
- the convs of the last stages of a ResNet (conv3–conv5 in DCNv2), in detectors and segmenters
- Cost
- a gather with bilinear weights per tap: memory-bound and slower than a dense conv of the same FLOPs
Worked example: one tap
Let the input contain and let tap land at (for instance grid position plus offset ). Then , , :
| neighbour | value | weight | contribution |
|---|---|---|---|
| (1, 1) | 8 | 0.80 | |
| (1, 2) | 6 | 1.80 | |
| (2, 1) | 3 | 0.45 | |
| (2, 2) | 9 | 4.05 | |
| 1.00 | 7.10 |
- 1.
- 2.
- 3With a scalar weight , and upstream gradient : , , . These flow back into the offset conv's weights.
Deformable RoI pooling
For two-stage detectors, DCNv1 also deforms RoI pooling. The region is split into bins as usual; a fully connected layer on the pooled features predicts a normalised offset per bin, which is scaled by the RoI's width and height (and a factor ) so the offsets are independent of the RoI size. Each bin is then average-pooled over bilinearly sampled points at its shifted position. A position-sensitive variant does the same for R-FCN. DCNv2 adds a modulation scalar per bin here too.
DCNv2, v3 and v4
Where it's used
DCNv2 (Zhu et al., 2019)
Adds , so a tap can not only move but also fade out: the layer can shrink its support or ignore background without contorting the offsets. Stacks deformable layers in all convs of ResNet stages conv3–conv5, and distils from an R-CNN on cropped regions ("feature mimicking") so the support focuses on the object.
DCNv3 / InternImage (Wang et al., 2023)
Separates the per-tap weights into a depthwise-like shared projection plus point-wise mixing (as in Depthwise separable convolution), splits channels into groups that each predict their own offsets, and normalises the modulation scalars with a softmax over the taps instead of independent sigmoids. InternImage-H reported 65.4 box AP on COCO test-dev and 62.9 mIoU on ADE20K.
DCNv4 (Xiong et al., 2024)
Drops the softmax normalisation (letting the aggregation weights be unbounded, like a convolution's) and rewrites the kernel for memory efficiency, making the operator several times faster than DCNv3.
1import torch, torch.nn as nn, torch.nn.functional as F2from torchvision.ops import DeformConv2d, deform_conv2d34class ModulatedDeformConv(nn.Module):5 """DCNv2: offsets (2*k*k channels) and masks (k*k channels) predicted from the input."""6 def __init__(self, cin, cout, k=3, stride=1, padding=1, dilation=1):7 super().__init__()8 self.k = k9 self.pred = nn.Conv2d(cin, 3 * k * k, k, stride, padding, dilation)10 nn.init.zeros_(self.pred.weight) # start as a plain conv: offsets 0, masks 0.511 nn.init.zeros_(self.pred.bias)12 self.dcn = DeformConv2d(cin, cout, k, stride, padding, dilation)1314 def forward(self, x):15 o = self.pred(x)16 offset = o[:, : 2 * self.k * self.k] # channel 2k = dy, 2k+1 = dx of tap k = i*kw + j17 mask = o[:, 2 * self.k * self.k :].sigmoid()18 return self.dcn(x, offset, mask)1920m = ModulatedDeformConv(16, 32)21x = torch.randn(2, 16, 32, 32)22print(m(x).shape, m.pred(x).shape) # torch.Size([2, 32, 32, 32]) torch.Size([2, 27, 32, 32])2324# Sanity checks with the functional op25w = torch.randn(32, 16, 3, 3)26zero = torch.zeros(2, 18, 32, 32)27print(torch.allclose(deform_conv2d(x, zero, w, padding=1), F.conv2d(x, w, padding=1), atol=1e-4)) # True2829# Moving every tap by exactly +1 in x (channels 1, 3, 5, ...) = reading the input one column to the right30shift = torch.zeros(2, 18, 32, 32); shift[:, 1::2] = 1.031xs = F.pad(x, (0, 2, 1, 1)) # pad left 0, right 2, top 1, bottom 1, then a 'valid' conv32print(torch.allclose(deform_conv2d(x, shift, w, padding=1), F.conv2d(xs, w), atol=1e-4)) # TrueUse when
- +Learns spatial adaptation (scale, pose, deformation) from the task loss, no extra labels
- +Drop-in replacement for 3×3 convs; starts identical to them with zero-init
- +Consistent gains in detection and segmentation, where geometry varies most
- +DCNv2 masks let taps switch off irrelevant context
Avoid when
- −Irregular memory access: slow on GPUs without custom kernels, awkward for export and edge runtimes
- −Offset gradients are local (one 2×2 cell), so training can be unstable without zero-init and a small learning rate
- −Extra hyperparameters: where to place layers, offset groups, mask on/off
- −Library-specific offset layouts complicate weight porting
Check yourself
A torchvision DeformConv2d with a 3 × 3 kernel and one offset group runs on a 64 × 64 output. What is the shape of the offset tensor for a batch of 2?
Check yourself
Why does deformable convolution use bilinear interpolation rather than rounding the sampling position to the nearest pixel?
Remember
- ✓Each tap samples at ; the offsets are predicted per location by a conv with outputs (18 for ).
- ✓Bilinear interpolation makes the samples differentiable in ; the gradient comes from the 4 neighbouring pixels.
- ✓DCNv2 adds a modulation per tap (27 predicted channels); zero-init gives , .
- ✓torchvision:
deform_conv2d(input, offset, weight, mask=...), offset channels of tap . - ✓DCNv3 (InternImage) shares weights across taps, groups the offsets and softmax-normalises the masks; DCNv4 drops the softmax and speeds it up.