# Deformable convolution

> A convolution whose sampling grid bends to the input: a side conv predicts a 2-D fractional offset for every kernel tap at every location, values are read by bilinear interpolation, and the offsets are trained end-to-end. DCNv2 adds a learned weight per tap.

Source: https://www.mlfoundry.in/learn/deformable-convolution  
Section: Deep Learning › Convolutional Neural Networks  
Level: advanced · 13 min read · updated 2026-10-11

**Deformable convolution:** is a convolution in which tap $k$ of the kernel, at output location $p_0$, samples the input at $p_0+p_k+\Delta p_k$ instead of the fixed grid position $p_0+p_k$. The offsets $\Delta p_k\in\mathbb R^2$ are produced by an ordinary convolution on the same input, so they differ per location and per image. Fractional positions are read by **bilinear interpolation**, which makes the output differentiable in the offsets.

A standard convolution samples a rigid rectangle. Pooling, striding and **Dilated convolution** change its size, but every location in a layer still uses the same shape. Objects do not cooperate: they appear at different scales, poses and deformations, and the useful evidence for a pixel on a cat's tail lies along a curve, not inside a square. Data augmentation can teach a rigid CNN some invariance, but only by spending capacity to memorise many transformed copies.

Dai, Qi, Xiong, Li, Zhang, Hu and Wei introduced **deformable convolution** and **deformable RoI pooling** in "Deformable Convolutional Networks" (ICCV 2017, DCNv1). The geometry becomes something the network learns from the task loss, with no extra supervision. Zhu, Hu, Lin and Dai's "Deformable ConvNets v2: More Deformable, Better Results" (CVPR 2019) added a learned **modulation** scalar per tap, used deformable layers in more stages of the backbone, and trained with feature mimicking from an R-CNN teacher. Wang et al.'s InternImage (CVPR 2023) redesigned the operator as **DCNv3** to build large foundation backbones, and Xiong et al.'s DCNv4 (CVPR 2024) made it much faster.

## Formula

$$
y(p_0)=\sum_{k=1}^{K} w_k\;m_k\;x\big(p_0+p_k+\Delta p_k\big),\qquad x(q)=\sum_{p} G(q,p)\,x(p),\quad G(q,p)=g(q_y,p_y)\,g(q_x,p_x),\ \ g(a,b)=\max(0,\,1-|a-b|)
$$

- $p_k$: the regular grid offsets, e.g. $\{-1,0,1\}^2$ for a $3\times3$ kernel (times the dilation)
- $\Delta p_k$: learned fractional offset $(\Delta y_k,\Delta x_k)$ of tap $k$ at location $p_0$; $2K$ numbers per location
- $m_k$: DCNv2 modulation scalar $\sigma(u_k)\in(0,1)$, also predicted per location; DCNv1 is $m_k=1$
- $w_k$: the ordinary kernel weights (a $C_{out}\times C_{in}$ matrix per tap), shared over all locations
- $G$: bilinear kernel: only the 4 integer pixels around $q$ have non-zero weight

The offsets come from a separate convolution with the same kernel size, stride and dilation as the deformable layer, applied to the same input. It outputs $2K$ channels ($2\cdot3\cdot3=18$ for a $3\times3$ kernel), or $3K=27$ for DCNv2 (18 offsets plus 9 mask logits). Both papers **initialise this conv to zero**, so training starts from an ordinary convolution ($\Delta p=0$) and, in DCNv2, from $m_k=\sigma(0)=0.5$.

## Gradients through the sampling position

The kernel weights get the usual gradient, with $x$ replaced by the interpolated samples. The new part is the gradient with respect to the offsets. Bilinear interpolation is piecewise linear in $q$, so its derivative is a difference of the four neighbouring pixels:

$$
\frac{\partial y(p_0)}{\partial \Delta x_k}=w_k\,m_k\,\frac{\partial x(q_k)}{\partial q_x},\qquad \frac{\partial x(q)}{\partial q_x}=(1-a_y)\big(x_{01}-x_{00}\big)+a_y\big(x_{11}-x_{10}\big),\qquad \frac{\partial y(p_0)}{\partial m_k}=w_k\,x(q_k)
$$

- $x_{00},x_{01},x_{10},x_{11}$: the four pixels around $q$: top-left, top-right, bottom-left, bottom-right
- $a_y,\ a_x$: fractional parts of $q_y$ and $q_x$; $\partial x(q)/\partial q_y$ is the same formula with the roles swapped

So a tap moves toward the side where the input changes in the direction that lowers the loss. The gradient is local: it only sees the 2×2 cell the tap sits in, which is why offsets start at zero and are trained with the rest of the network rather than searched globally. When $q$ lands exactly on a pixel the derivative is one-sided; implementations just use the formula above.

**At a glance**

- **Extra parameters:** the offset conv: $K\cdot C_{in}\cdot 2K$ weights (+ $K\cdot C_{in}\cdot K$ for DCNv2 masks); e.g. $9\cdot256\cdot27=62{,}208$ next to $589{,}824$ for a 256→256 3×3 layer
- **Offset tensor (torchvision):** shape $(B,\ 2\,G_{\text{off}}\,k_h k_w,\ H_{out},\ W_{out})$; for tap $k=i\,k_w+j$, channel $2k$ is $\Delta y$ and $2k+1$ is $\Delta x$
- **Mask tensor:** shape $(B,\ G_{\text{off}}\,k_h k_w,\ H_{out},\ W_{out})$, values usually $\sigma(\cdot)$
- **Init:** offset/mask conv weights and bias = 0 (starts as a plain conv, masks 0.5)
- **Typical placement:** the $3\times3$ convs of the last stages of a ResNet (conv3–conv5 in DCNv2), in detectors and segmenters
- **Cost:** a gather with bilinear weights per tap: memory-bound and slower than a dense conv of the same FLOPs

> Figure: One output location p₀ = (20, 20) of a 3×3 deformable conv on a drawn 40×40 ring (regular grid spacing 3, hollow circles). Here the 18 offsets are fitted directly by gradient ascent on J = Σ mₖ(x(qₖ) − 0.25) − λ/2·Σ‖Δpₖ‖², so you can watch the gradient path in isolation; in a network a conv layer would predict them. Press fit offsets: with λ = 0.02 the mean sampled value rises from 0.249 on the regular grid to 0.966 after 80 steps as the taps slide onto the ring. With DCNv2 masks on, the centre tap, which sits in the empty middle, moves only 0.28 px and switches itself off (m = 0.14) instead of travelling 5.8 px as in DCNv1. Raise λ to 0.2 and the taps stay close to the grid (response 0.518). The table lists each tap's Δy/Δx with its torchvision channel index (2k, 2k+1) and the 2×2 bilinear read of the selected tap; click the map to move p₀ or switch to the curve and bar shapes.

## Worked example: one tap

Let the input contain $x(1,1)=8,\ x(1,2)=6,\ x(2,1)=3,\ x(2,2)=9$ and let tap $k$ land at $q=(1.6,\,1.75)$ (for instance grid position $(1,2)$ plus offset $\Delta p_k=(0.6,\,-0.25)$). Then $y_0=1,\ x_0=1$, $a_y=0.6$, $a_x=0.75$:

Bilinear read and its gradient (verified with the figure's library)

| neighbour | value | weight | contribution |
| --- | --- | --- | --- |
| (1, 1) | 8 | $0.4\cdot0.25=0.10$ | 0.80 |
| (1, 2) | 6 | $0.4\cdot0.75=0.30$ | 1.80 |
| (2, 1) | 3 | $0.6\cdot0.25=0.15$ | 0.45 |
| (2, 2) | 9 | $0.6\cdot0.75=0.45$ | 4.05 |
| $x(q)$ |  | 1.00 | **7.10** |

1. $\partial x(q)/\partial q_y=(1-0.75)(3-8)+0.75(9-6)=-1.25+2.25=1.0$.
2. $\partial x(q)/\partial q_x=(1-0.6)(6-8)+0.6(9-3)=-0.8+3.6=2.8$.
3. With a scalar weight $w_k=0.5$, $m_k=1$ and upstream gradient $\partial L/\partial y=1$: $\partial L/\partial\Delta y_k=0.5$, $\partial L/\partial\Delta x_k=1.4$, $\partial L/\partial w_k=7.1$. These flow back into the offset conv's weights.

## Deformable RoI pooling

For two-stage detectors, DCNv1 also deforms **RoI pooling**. The region is split into $K\times K$ bins as usual; a fully connected layer on the pooled features predicts a normalised offset $\Delta\hat p_{ij}$ per bin, which is scaled by the RoI's width and height (and a factor $\gamma=0.1$) so the offsets are independent of the RoI size. Each bin is then average-pooled over bilinearly sampled points at its shifted position. A position-sensitive variant does the same for R-FCN. DCNv2 adds a modulation scalar per bin here too.

## DCNv2, v3 and v4

- **DCNv2 (Zhu et al., 2019):** Adds $m_k$, so a tap can not only move but also fade out: the layer can shrink its support or ignore background without contorting the offsets. Stacks deformable layers in all $3\times3$ convs of ResNet stages conv3–conv5, and distils from an R-CNN on cropped regions ("feature mimicking") so the support focuses on the object.
- **DCNv3 / InternImage (Wang et al., 2023):** Separates the per-tap weights into a depthwise-like shared projection plus point-wise mixing (as in **Depthwise separable convolution**), splits channels into groups that each predict their own offsets, and normalises the modulation scalars with a softmax over the $K$ taps instead of independent sigmoids. InternImage-H reported 65.4 box AP on COCO test-dev and 62.9 mIoU on ADE20K.
- **DCNv4 (Xiong et al., 2024):** Drops the softmax normalisation (letting the aggregation weights be unbounded, like a convolution's) and rewrites the kernel for memory efficiency, making the operator several times faster than DCNv3.

```python
import torch, torch.nn as nn, torch.nn.functional as F
from torchvision.ops import DeformConv2d, deform_conv2d

class ModulatedDeformConv(nn.Module):
    """DCNv2: offsets (2*k*k channels) and masks (k*k channels) predicted from the input."""
    def __init__(self, cin, cout, k=3, stride=1, padding=1, dilation=1):
        super().__init__()
        self.k = k
        self.pred = nn.Conv2d(cin, 3 * k * k, k, stride, padding, dilation)
        nn.init.zeros_(self.pred.weight)     # start as a plain conv: offsets 0, masks 0.5
        nn.init.zeros_(self.pred.bias)
        self.dcn = DeformConv2d(cin, cout, k, stride, padding, dilation)

    def forward(self, x):
        o = self.pred(x)
        offset = o[:, : 2 * self.k * self.k]          # channel 2k = dy, 2k+1 = dx of tap k = i*kw + j
        mask = o[:, 2 * self.k * self.k :].sigmoid()
        return self.dcn(x, offset, mask)

m = ModulatedDeformConv(16, 32)
x = torch.randn(2, 16, 32, 32)
print(m(x).shape, m.pred(x).shape)   # torch.Size([2, 32, 32, 32]) torch.Size([2, 27, 32, 32])

# Sanity checks with the functional op
w = torch.randn(32, 16, 3, 3)
zero = torch.zeros(2, 18, 32, 32)
print(torch.allclose(deform_conv2d(x, zero, w, padding=1), F.conv2d(x, w, padding=1), atol=1e-4))  # True

# Moving every tap by exactly +1 in x (channels 1, 3, 5, ...) = reading the input one column to the right
shift = torch.zeros(2, 18, 32, 32); shift[:, 1::2] = 1.0
xs = F.pad(x, (0, 2, 1, 1))           # pad left 0, right 2, top 1, bottom 1, then a 'valid' conv
print(torch.allclose(deform_conv2d(x, shift, w, padding=1), F.conv2d(xs, w), atol=1e-4))  # True
```

DCNv2 with torchvision.ops: zero-initialised offset/mask predictor and the offset layout

> **Initialise the offset predictor to zero:** Random offsets of a few pixels on the first step scramble the pretrained features and can diverge. Zero-init the offset/mask conv (as both papers do), and many codebases also lower its learning rate. Watch for offsets that grow to the size of the feature map: they mean the taps have escaped into zero padding.

> **Offset layout differs between libraries:** torchvision interleaves $(\Delta y,\Delta x)$ per tap. The original DCNv2 / mmcv code computes three chunks `o1, o2, mask` and concatenates `o1, o2` as the offset, and DCNv3 uses yet another layout. When porting weights, check the order of the 18 channels; a mismatch still trains, but pretrained offsets become garbage.

> **Dilation is a special case:** A constant offset field $\Delta p_k=(d-1)\,p_k$ turns a $3\times3$ conv into a dilated one with rate $d$. Deformable conv lets every location pick its own, possibly non-integer, non-rectangular dilation; in the DCNv1 paper the learned sampling points spread out on large objects and contract on small ones.

> **Cost in practice:** The FLOPs are those of the conv plus the small offset conv, but the irregular gather defeats im2col/implicit-GEMM tiling and caching. Expect the deformable layers to be noticeably slower than dense convolutions of the same size, which is why they are placed only in the later, low-resolution stages.

**Strengths**

- Learns spatial adaptation (scale, pose, deformation) from the task loss, no extra labels
- Drop-in replacement for 3×3 convs; starts identical to them with zero-init
- Consistent gains in detection and segmentation, where geometry varies most
- DCNv2 masks let taps switch off irrelevant context

**Limitations**

- Irregular memory access: slow on GPUs without custom kernels, awkward for export and edge runtimes
- Offset gradients are local (one 2×2 cell), so training can be unstable without zero-init and a small learning rate
- Extra hyperparameters: where to place layers, offset groups, mask on/off
- Library-specific offset layouts complicate weight porting

**Question:** A torchvision DeformConv2d with a 3 × 3 kernel and one offset group runs on a 64 × 64 output. What is the shape of the offset tensor for a batch of 2?

- A. (2, 9, 64, 64)
- B. (2, 18, 64, 64)
- C. (2, 27, 64, 64)
- D. (2, 2, 3, 3)

**Answer:** B. (2, 18, 64, 64) — Two numbers $(\Delta y,\Delta x)$ per tap, $3\cdot3=9$ taps, at every output location: $2\cdot9=18$ channels. The DCNv2 mask is a separate $(2,9,64,64)$ tensor; 27 is the width of a combined offset+mask predictor.

**Question:** Why does deformable convolution use bilinear interpolation rather than rounding the sampling position to the nearest pixel?

- A. Rounding is slower
- B. Bilinear makes the output a differentiable function of the fractional offsets, so they can be learned by backprop
- C. Rounding cannot handle negative offsets
- D. Bilinear interpolation removes the need for the offset conv

**Answer:** B. Bilinear makes the output a differentiable function of the fractional offsets, so they can be learned by backprop — With rounding, $x(q)$ is piecewise constant in $\Delta p$ and its gradient is zero almost everywhere. Bilinear interpolation is piecewise linear, with gradient $w_k m_k\,\partial x(q)/\partial q$ from the 2×2 neighbourhood.

**Key points**

- Each tap samples at $p_0+p_k+\Delta p_k$; the offsets are predicted per location by a conv with $2K$ outputs (18 for $3\times3$).
- Bilinear interpolation makes the samples differentiable in $\Delta p$; the gradient comes from the 4 neighbouring pixels.
- DCNv2 adds a modulation $m_k=\sigma(u_k)$ per tap (27 predicted channels); zero-init gives $\Delta p=0$, $m=0.5$.
- torchvision: `deform_conv2d(input, offset, weight, mask=...)`, offset channels $(2k,2k+1)=(\Delta y,\Delta x)$ of tap $k$.
- DCNv3 (InternImage) shares weights across taps, groups the offsets and softmax-normalises the masks; DCNv4 drops the softmax and speeds it up.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
