Dilated convolution

A convolution whose kernel taps are spread d pixels apart. A 3×3 kernel with dilation d covers a (2d+1)×(2d+1) window with only 9 weights, so stacking d = 1, 2, 4, 8 grows the receptive field exponentially without pooling, striding or extra parameters.

11 min read985 wordscore0 viewsupdated 2026-10-11

Definition

Dilated convolution (also atrous convolution, from the French à trous, "with holes") is a convolution whose kernel reads the input at taps spaced dd pixels apart instead of adjacent pixels: y[i]=∑tw[t] x[i+d t]y[i]=\sum_t w[t]\,x[i+d\,t]. The integer d≥1d\ge1 is the dilation rate; d=1d=1 is the ordinary convolution. The kernel keeps its kk weights per axis, but they span k+(k−1)(d−1)k+(k-1)(d-1) pixels.

Dense prediction tasks such as semantic segmentation, depth estimation and audio generation need two things at once: a large receptive field (to see context) and full resolution (to output one value per pixel or sample). The classic way to grow a receptive field is to downsample with pooling or strided convolution (see Stride and padding and Receptive field), but that throws away resolution, which then has to be recovered with upsampling. Dilation grows the receptive field while keeping stride 1 and the parameter count unchanged.

The idea is old. The algorithme à trous of Holschneider, Kronland-Martinet, Morlet and Tchamitchian (1989) computes the undecimated wavelet transform by inserting zeros into the filters instead of subsampling the signal. In deep learning, Chen, Papandreou, Kokkinos, Murphy and Yuille (DeepLab, ICLR 2015) used atrous convolution to remove the last downsampling steps of an ImageNet-pretrained VGG while keeping its weights valid. Yu and Koltun ("Multi-Scale Context Aggregation by Dilated Convolutions", ICLR 2016) named it dilated convolution, built a context module from stacked layers with exponentially increasing dilation, and proved the exponential receptive-field growth. The same year van den Oord et al.'s WaveNet (2016) used causal dilated 1-D convolutions to model raw audio.

Formula

y[i,j]=∑u=0k−1∑v=0k−1w[u,v]  x[i+d u,  j+d v],keff=k+(k−1)(d−1)y[i,j]=\sum_{u=0}^{k-1}\sum_{v=0}^{k-1} w[u,v]\;x\big[i+d\,u,\;j+d\,v\big],\qquad k_{\text{eff}}=k+(k-1)(d-1)
dd
dilation rate: the spacing between neighbouring taps (d−1d-1 skipped pixels, the "holes")
kk
kernel size along one axis; the number of weights is still k2k^2 per (input, output) channel pair
keffk_{\text{eff}}
effective kernel size, the span of the taps: 3×33\times3 with d=2d=2 covers 5×55\times5, d=4d=4 covers 9×99\times9, d=8d=8 covers 17×1717\times17
ww
the same learned weights as an ordinary k×kk\times k conv; equivalently a keff×keffk_{\text{eff}}\times k_{\text{eff}} kernel with zeros inserted between the taps

Output size uses the effective kernel. PyTorch's rule for one axis is ⌊(n+2p−d(k−1)−1)/s⌋+1\lfloor (n+2p-d(k-1)-1)/s\rfloor+1. With stride 1 and k=3k=3, choosing p=dp=d keeps the size: a 32×3232\times32 map with d=2, p=2d=2,\ p=2 stays 32×3232\times32, and a 65×6565\times65 map with d=6, p=6d=6,\ p=6 stays 65×6565\times65. With p=0p=0 each dilated layer trims d(k−1)d(k-1) pixels instead of k−1k-1.

Gradients

Dilation changes only which input pixels a tap touches, so the backward pass is the same bookkeeping as an ordinary convolution with the same spacing. Writing δ[i]=∂L/∂y[i]\delta[i]=\partial L/\partial y[i] in 1-D:

∂L∂w[t]=∑iδ[i] x[i+d t],∂L∂x[j]=∑t w[t]  δ[j−d t]\frac{\partial L}{\partial w[t]}=\sum_i \delta[i]\,x[i+d\,t],\qquad \frac{\partial L}{\partial x[j]}=\sum_{t}\,w[t]\;\delta[j-d\,t]
∂L/∂w[t]\partial L/\partial w[t]
correlation of the input with the error map, sampled at spacing dd
∂L/∂x[j]\partial L/\partial x[j]
a full (zero-padded) correlation of δ\delta with the flipped kernel at the same dilation: the transposed dilated convolution

At a glance

Parameters
k2CinCoutk^2 C_{in} C_{out} (+ bias), independent of dd
FLOPs
same as the undilated conv on the same output size: k2k^2 multiply-adds per output per channel pair
Receptive field (stride 1)
1+(k−1)∑ldl1+(k-1)\sum_l d_l after a stack of layers
Padding to keep size
p=d (k−1)/2p=d\,(k-1)/2 for odd kk, stride 1 (PyTorch: padding="same" also works)
Main uses
semantic segmentation (DeepLab, Yu & Koltun's context module), audio and time series (WaveNet, TCNs), keeping resolution in late backbone stages
Known failure
gridding: stacks whose rates share a common factor leave input pixels that no path reaches

Exponential receptive field

With stride 1, a k×kk\times k layer at dilation dd adds (k−1)d(k-1)d to the receptive field. Plain 3×33\times3 layers add 2 each, so the field grows linearly: 3, 5, 7, 9. Doubling the dilation each layer, d=1,2,4,8d=1,2,4,8, makes it grow exponentially: 3, 7, 15, 31, i.e. 2L+1−12^{L+1}-1 after LL layers. A plain stack needs 15 layers (15×9=13515\times9=135 weights per channel pair) to reach the 31×3131\times31 field that four dilated layers reach with 36. Yu and Koltun's context module uses rates 1,1,2,4,8,16,11,1,2,4,8,16,1 and reaches 67×6767\times67 with seven 3×33\times3 layers.

paths: input pixel → centre output1-D path counts (one row of the map); red = never readlayerdk_effRF113×33×3225×57×7349×915×154817×1731×31receptive field31 × 31pixels read961 / 961coverage100.0%max paths (centre)25weights per ch. pair36 (4 × 3×3)plain 3×3 layers forsame RF: 15□ taps of layer 4: offsets −8, 0, 8k_eff = k + (k−1)(d−1) · RF = 1 + (k−1)Σdevery pixel in the field is read
presets
d11d22d34d48
fig.A stack of 3×3 convolutions with all-ones weights. The heat map counts, for every input pixel, how many paths lead to the centre output (darker blue = more paths; red = zero paths, never read), and the orange squares are the 9 taps of the last layer shown. The default rates 1, 2, 4, 8 give per-layer effective kernels 3, 5, 9, 17 and receptive fields 3, 7, 15, 31: all 961 pixels of the 31×31 field are read, with 36 weights, where plain 3×3 layers would need 15 layers. Try the preset 2, 2, 2, 2: the field is 17×17 but only 9 of every 17 rows and columns are reached, 81 of 289 pixels (28.0%). 2, 4, 8 is worse (26.8%), while the hybrid rates 1, 2, 5 cover the whole 17×17 field. Use the ± buttons to set each layer's rate and the slider to reveal layers one at a time.

Gridding, and how to avoid it

Dilation has a price: a single dilated layer only looks at a sparse lattice of pixels. If every layer of a stack uses rates that are all multiples of some g>1g>1, every path moves in steps of gg, so the output depends only on input pixels whose offset from it is a multiple of gg. Neighbouring outputs then read disjoint sub-lattices of the input. In 1-D two layers with d=2d=2 have path counts 1,0,2,0,3,0,2,0,11,0,2,0,3,0,2,0,1 over a 9-pixel field: every odd offset is a hole. In 2-D the holes form a grid, and the outputs show a matching checkerboard of inconsistent predictions, especially on thin structures.

Wang et al. ("Understanding Convolution for Semantic Segmentation", WACV 2018) named this gridding and proposed hybrid dilated convolution (HDC): use a repeating group of rates with no common factor greater than 1, such as 1,2,51,2,5 or 1,2,31,2,3 instead of 2,2,22,2,2 or 2,4,82,4,8, chosen so the group's combined footprint has no holes. Yu, Koltun and Funkhouser's Dilated Residual Networks (2017) attack the same artefact by appending undilated layers ("degridding"). Starting a doubling sequence at d=1d=1, as in 1,2,4,81,2,4,8, also avoids holes: the first layer fills the gaps the later ones skip.

ratesreceptive fieldpixels readweights / channel pair
1, 1, 1, 19 × 9100%36
1, 2, 4, 831 × 31100%36
2, 2, 2, 217 × 1728.0% (81 / 289)36
2, 4, 829 × 2926.8% (225 / 841)27
1, 2, 5 (HDC)17 × 17100%27
1, 1, 2, 4, 8, 16, 1 (Yu & Koltun)67 × 67100%63
Stacks of 3×3 layers (stride 1), computed by the figure's library

Worked example: one dilated layer in 1-D

Take x=(3,1,4,1,5,9,2,6,5,3)x=(3,1,4,1,5,9,2,6,5,3), kernel w=(1,2,−1)w=(1,2,-1), dilation d=2d=2, no padding. The effective kernel spans 3+2⋅1=53+2\cdot1=5 samples, so the output has 10−5+1=610-5+1=6 values. Each output reads x[i], x[i+2], x[i+4]x[i],\,x[i+2],\,x[i+4]:

Forward, then backward with L = Σ y (so every δ[i] = 1)

  1. 1y[0]=1⋅3+2⋅4−1⋅5=6y[0]=1\cdot3+2\cdot4-1\cdot5=6, y[1]=1+2⋅1−9=−6y[1]=1+2\cdot1-9=-6, y[2]=4+2⋅5−2=12y[2]=4+2\cdot5-2=12.
  2. 2y[3]=1+2⋅9−6=13y[3]=1+2\cdot9-6=13, y[4]=5+2⋅2−5=4y[4]=5+2\cdot2-5=4, y[5]=9+2⋅6−3=18y[5]=9+2\cdot6-3=18. So y=(6,−6,12,13,4,18)y=(6,-6,12,13,4,18). With d=1d=1 the same kernel gives 8 outputs, (1,8,1,2,21,7,9,13)(1,8,1,2,21,7,9,13).
  3. 3Weight gradients: ∂L/∂w[t]=∑i=05x[i+2t]\partial L/\partial w[t]=\sum_{i=0}^{5}x[i+2t], i.e. sums of x[0..5]x[0..5], x[2..7]x[2..7], x[4..9]x[4..9]: (23, 27, 30)(23,\,27,\,30).
  4. 4Input gradients: ∂L/∂x[j]=∑tw[t] 1[0≤j−2t≤5]=(1,1,3,3,2,2,1,1,−1,−1)\partial L/\partial x[j]=\sum_t w[t]\,\mathbb 1[0\le j-2t\le5]=(1,1,3,3,2,2,1,1,-1,-1). Each x[j]x[j] gets the weights of the taps that touched it; the values match a finite-difference check.
python
1import torch, torch.nn as nn, torch.nn.functional as F
2
3# Same 9 weights, 5x5 footprint. padding = dilation keeps H and W for k = 3.
4conv = nn.Conv2d(64, 64, kernel_size=3, dilation=2, padding=2)
5print(conv(torch.randn(1, 64, 32, 32)).shape) # torch.Size([1, 64, 32, 32])
6print(sum(p.numel() for p in conv.parameters())) # 36928 = 64*64*9 + 64, same as dilation=1
7
8# 1-D check of the worked example
9x = torch.tensor([[[3., 1, 4, 1, 5, 9, 2, 6, 5, 3]]], requires_grad=True)
10w = torch.tensor([[[1., 2, -1]]], requires_grad=True)
11y = F.conv1d(x, w, dilation=2)
12y.sum().backward()
13print(y.flatten().tolist()) # [6.0, -6.0, 12.0, 13.0, 4.0, 18.0]
14print(w.grad.flatten().tolist()) # [23.0, 27.0, 30.0]
15
16# WaveNet-style causal dilated stack (kernel 2, rates 1..512): receptive field 1024 samples
17class CausalConv1d(nn.Conv1d):
18 def forward(self, x):
19 pad = self.dilation[0] * (self.kernel_size[0] - 1)
20 return super().forward(F.pad(x, (pad, 0))) # pad on the left only: no peeking ahead
21
22stack = nn.Sequential(*[CausalConv1d(32, 32, kernel_size=2, dilation=2**i) for i in range(10)])
23
24# Atrous spatial pyramid pooling: parallel 3x3 branches at several rates
25class ASPP(nn.Module):
26 def __init__(self, cin, cout, rates=(6, 12, 18)):
27 super().__init__()
28 self.branches = nn.ModuleList(
29 [nn.Conv2d(cin, cout, 1)] +
30 [nn.Conv2d(cin, cout, 3, padding=r, dilation=r) for r in rates])
31 self.pool = nn.Sequential(nn.AdaptiveAvgPool2d(1), nn.Conv2d(cin, cout, 1))
32 self.project = nn.Conv2d(cout * (len(rates) + 2), cout, 1)
33 def forward(self, x):
34 feats = [b(x) for b in self.branches]
35 feats.append(F.interpolate(self.pool(x), size=x.shape[-2:], mode="bilinear"))
36 return self.project(torch.cat(feats, dim=1))
37
38# torchvision's DeepLabV3 dilates the last two ResNet stages (output stride 8)
39from torchvision.models import resnet50
40backbone = resnet50(replace_stride_with_dilation=[False, True, True])
Dilated convolutions in PyTorch, a WaveNet-style causal stack, and DeepLab's ASPP

Where it is used

Where it's used

Semantic segmentation

DeepLab replaces the strides of the last backbone stages with dilation (output stride 16 or 8 instead of 32) so the pretrained filters keep their receptive field at higher resolution. ASPP (atrous spatial pyramid pooling, DeepLabv2) runs parallel 3×33\times3 branches at several rates; DeepLabv3 uses rates 6,12,186,12,18 plus a 1×11\times1 branch and global image pooling at output stride 16, doubled to 12,24,3612,24,36 at stride 8.

Audio and sequences

WaveNet stacks causal dilated convolutions with rates 1,2,4,…,5121,2,4,\dots,512, repeated in several blocks, giving each block a 1024-sample receptive field. Temporal convolutional networks (Bai, Kolter and Koltun, 2018) use the same recipe for general sequence modelling, and ByteNet (Kalchbrenner et al., 2016) for translation.

Backbones and detectors

Dilated ResNets (Yu, Koltun and Funkhouser, 2017) keep late-stage resolution for localisation; detectors such as DetNet and TridentNet use dilation to enlarge or vary receptive fields without pooling.

Use when

  • +Exponential receptive-field growth with no extra parameters
  • +Keeps full resolution: no pooling, no information-losing striding
  • +A drop-in change: same weights, so pretrained filters can be dilated (DeepLab)
  • +Natural fit for causal 1-D models (WaveNet, TCN)

Avoid when

  • −Gridding holes when rates share a common factor
  • −Sparse sampling ignores local detail at large rates
  • −Compute and memory of full-resolution maps are high compared with downsampling
  • −Strided memory access is less cache-friendly than dense 3×3 kernels on some hardware

Check yourself

Four 3×3 convolutions with stride 1 and dilation rates 1, 2, 4, 8 are stacked. What is the receptive field of one output?

Check yourself

Why does the stack 2, 2, 2, 2 cover only 28% of its 17 × 17 receptive field?

Remember

  • ✓Dilation dd spreads the kk taps dd apart: keff=k+(k−1)(d−1)k_{\text{eff}}=k+(k-1)(d-1), with the same k2k^2 weights and FLOPs.
  • ✓Stride-1 stacks have receptive field 1+(k−1)∑dl1+(k-1)\sum d_l: rates 1,2,4,81,2,4,8 give 31 with four layers, versus 9 for plain 3×33\times3.
  • ✓Keep the size with p=d(k−1)/2p=d(k-1)/2; PyTorch: nn.Conv2d(c, c, 3, padding=d, dilation=d).
  • ✓Rates that share a common factor cause gridding; hybrid rates (1, 2, 5) or a leading d = 1 layer avoid it.
  • ✓Used for dense prediction at high resolution (DeepLab, ASPP) and causal sequence models (WaveNet, TCN).
← back to Deep Learning roadmap