Definition
Dilated convolution (also atrous convolution, from the French à trous, "with holes") is a convolution whose kernel reads the input at taps spaced pixels apart instead of adjacent pixels: . The integer is the dilation rate; is the ordinary convolution. The kernel keeps its weights per axis, but they span pixels.
Dense prediction tasks such as semantic segmentation, depth estimation and audio generation need two things at once: a large receptive field (to see context) and full resolution (to output one value per pixel or sample). The classic way to grow a receptive field is to downsample with pooling or strided convolution (see Stride and padding and Receptive field), but that throws away resolution, which then has to be recovered with upsampling. Dilation grows the receptive field while keeping stride 1 and the parameter count unchanged.
The idea is old. The algorithme à trous of Holschneider, Kronland-Martinet, Morlet and Tchamitchian (1989) computes the undecimated wavelet transform by inserting zeros into the filters instead of subsampling the signal. In deep learning, Chen, Papandreou, Kokkinos, Murphy and Yuille (DeepLab, ICLR 2015) used atrous convolution to remove the last downsampling steps of an ImageNet-pretrained VGG while keeping its weights valid. Yu and Koltun ("Multi-Scale Context Aggregation by Dilated Convolutions", ICLR 2016) named it dilated convolution, built a context module from stacked layers with exponentially increasing dilation, and proved the exponential receptive-field growth. The same year van den Oord et al.'s WaveNet (2016) used causal dilated 1-D convolutions to model raw audio.
Formula
- dilation rate: the spacing between neighbouring taps ( skipped pixels, the "holes")
- kernel size along one axis; the number of weights is still per (input, output) channel pair
- effective kernel size, the span of the taps: with covers , covers , covers
- the same learned weights as an ordinary conv; equivalently a kernel with zeros inserted between the taps
Output size uses the effective kernel. PyTorch's rule for one axis is . With stride 1 and , choosing keeps the size: a map with stays , and a map with stays . With each dilated layer trims pixels instead of .
Gradients
Dilation changes only which input pixels a tap touches, so the backward pass is the same bookkeeping as an ordinary convolution with the same spacing. Writing in 1-D:
- correlation of the input with the error map, sampled at spacing
- a full (zero-padded) correlation of with the flipped kernel at the same dilation: the transposed dilated convolution
At a glance
- Parameters
- (+ bias), independent of
- FLOPs
- same as the undilated conv on the same output size: multiply-adds per output per channel pair
- Receptive field (stride 1)
- after a stack of layers
- Padding to keep size
- for odd , stride 1 (PyTorch:
padding="same"also works) - Main uses
- semantic segmentation (DeepLab, Yu & Koltun's context module), audio and time series (WaveNet, TCNs), keeping resolution in late backbone stages
- Known failure
- gridding: stacks whose rates share a common factor leave input pixels that no path reaches
Exponential receptive field
With stride 1, a layer at dilation adds to the receptive field. Plain layers add 2 each, so the field grows linearly: 3, 5, 7, 9. Doubling the dilation each layer, , makes it grow exponentially: 3, 7, 15, 31, i.e. after layers. A plain stack needs 15 layers ( weights per channel pair) to reach the field that four dilated layers reach with 36. Yu and Koltun's context module uses rates and reaches with seven layers.
Gridding, and how to avoid it
Dilation has a price: a single dilated layer only looks at a sparse lattice of pixels. If every layer of a stack uses rates that are all multiples of some , every path moves in steps of , so the output depends only on input pixels whose offset from it is a multiple of . Neighbouring outputs then read disjoint sub-lattices of the input. In 1-D two layers with have path counts over a 9-pixel field: every odd offset is a hole. In 2-D the holes form a grid, and the outputs show a matching checkerboard of inconsistent predictions, especially on thin structures.
Wang et al. ("Understanding Convolution for Semantic Segmentation", WACV 2018) named this gridding and proposed hybrid dilated convolution (HDC): use a repeating group of rates with no common factor greater than 1, such as or instead of or , chosen so the group's combined footprint has no holes. Yu, Koltun and Funkhouser's Dilated Residual Networks (2017) attack the same artefact by appending undilated layers ("degridding"). Starting a doubling sequence at , as in , also avoids holes: the first layer fills the gaps the later ones skip.
| rates | receptive field | pixels read | weights / channel pair |
|---|---|---|---|
| 1, 1, 1, 1 | 9 × 9 | 100% | 36 |
| 1, 2, 4, 8 | 31 × 31 | 100% | 36 |
| 2, 2, 2, 2 | 17 × 17 | 28.0% (81 / 289) | 36 |
| 2, 4, 8 | 29 × 29 | 26.8% (225 / 841) | 27 |
| 1, 2, 5 (HDC) | 17 × 17 | 100% | 27 |
| 1, 1, 2, 4, 8, 16, 1 (Yu & Koltun) | 67 × 67 | 100% | 63 |
Worked example: one dilated layer in 1-D
Take , kernel , dilation , no padding. The effective kernel spans samples, so the output has values. Each output reads :
Forward, then backward with L = Σ y (so every δ[i] = 1)
- 1, , .
- 2, , . So . With the same kernel gives 8 outputs, .
- 3Weight gradients: , i.e. sums of , , : .
- 4Input gradients: . Each gets the weights of the taps that touched it; the values match a finite-difference check.
1import torch, torch.nn as nn, torch.nn.functional as F23# Same 9 weights, 5x5 footprint. padding = dilation keeps H and W for k = 3.4conv = nn.Conv2d(64, 64, kernel_size=3, dilation=2, padding=2)5print(conv(torch.randn(1, 64, 32, 32)).shape) # torch.Size([1, 64, 32, 32])6print(sum(p.numel() for p in conv.parameters())) # 36928 = 64*64*9 + 64, same as dilation=178# 1-D check of the worked example9x = torch.tensor([[[3., 1, 4, 1, 5, 9, 2, 6, 5, 3]]], requires_grad=True)10w = torch.tensor([[[1., 2, -1]]], requires_grad=True)11y = F.conv1d(x, w, dilation=2)12y.sum().backward()13print(y.flatten().tolist()) # [6.0, -6.0, 12.0, 13.0, 4.0, 18.0]14print(w.grad.flatten().tolist()) # [23.0, 27.0, 30.0]1516# WaveNet-style causal dilated stack (kernel 2, rates 1..512): receptive field 1024 samples17class CausalConv1d(nn.Conv1d):18 def forward(self, x):19 pad = self.dilation[0] * (self.kernel_size[0] - 1)20 return super().forward(F.pad(x, (pad, 0))) # pad on the left only: no peeking ahead2122stack = nn.Sequential(*[CausalConv1d(32, 32, kernel_size=2, dilation=2**i) for i in range(10)])2324# Atrous spatial pyramid pooling: parallel 3x3 branches at several rates25class ASPP(nn.Module):26 def __init__(self, cin, cout, rates=(6, 12, 18)):27 super().__init__()28 self.branches = nn.ModuleList(29 [nn.Conv2d(cin, cout, 1)] +30 [nn.Conv2d(cin, cout, 3, padding=r, dilation=r) for r in rates])31 self.pool = nn.Sequential(nn.AdaptiveAvgPool2d(1), nn.Conv2d(cin, cout, 1))32 self.project = nn.Conv2d(cout * (len(rates) + 2), cout, 1)33 def forward(self, x):34 feats = [b(x) for b in self.branches]35 feats.append(F.interpolate(self.pool(x), size=x.shape[-2:], mode="bilinear"))36 return self.project(torch.cat(feats, dim=1))3738# torchvision's DeepLabV3 dilates the last two ResNet stages (output stride 8)39from torchvision.models import resnet5040backbone = resnet50(replace_stride_with_dilation=[False, True, True])Where it is used
Where it's used
Semantic segmentation
DeepLab replaces the strides of the last backbone stages with dilation (output stride 16 or 8 instead of 32) so the pretrained filters keep their receptive field at higher resolution. ASPP (atrous spatial pyramid pooling, DeepLabv2) runs parallel branches at several rates; DeepLabv3 uses rates plus a branch and global image pooling at output stride 16, doubled to at stride 8.
Audio and sequences
WaveNet stacks causal dilated convolutions with rates , repeated in several blocks, giving each block a 1024-sample receptive field. Temporal convolutional networks (Bai, Kolter and Koltun, 2018) use the same recipe for general sequence modelling, and ByteNet (Kalchbrenner et al., 2016) for translation.
Backbones and detectors
Dilated ResNets (Yu, Koltun and Funkhouser, 2017) keep late-stage resolution for localisation; detectors such as DetNet and TridentNet use dilation to enlarge or vary receptive fields without pooling.
Use when
- +Exponential receptive-field growth with no extra parameters
- +Keeps full resolution: no pooling, no information-losing striding
- +A drop-in change: same weights, so pretrained filters can be dilated (DeepLab)
- +Natural fit for causal 1-D models (WaveNet, TCN)
Avoid when
- −Gridding holes when rates share a common factor
- −Sparse sampling ignores local detail at large rates
- −Compute and memory of full-resolution maps are high compared with downsampling
- −Strided memory access is less cache-friendly than dense 3×3 kernels on some hardware
Check yourself
Four 3×3 convolutions with stride 1 and dilation rates 1, 2, 4, 8 are stacked. What is the receptive field of one output?
Check yourself
Why does the stack 2, 2, 2, 2 cover only 28% of its 17 × 17 receptive field?
Remember
- ✓Dilation spreads the taps apart: , with the same weights and FLOPs.
- ✓Stride-1 stacks have receptive field : rates give 31 with four layers, versus 9 for plain .
- ✓Keep the size with ; PyTorch:
nn.Conv2d(c, c, 3, padding=d, dilation=d). - ✓Rates that share a common factor cause gridding; hybrid rates (1, 2, 5) or a leading d = 1 layer avoid it.
- ✓Used for dense prediction at high resolution (DeepLab, ASPP) and causal sequence models (WaveNet, TCN).