# Dilated convolution

> A convolution whose kernel taps are spread d pixels apart. A 3×3 kernel with dilation d covers a (2d+1)×(2d+1) window with only 9 weights, so stacking d = 1, 2, 4, 8 grows the receptive field exponentially without pooling, striding or extra parameters.

Source: https://www.mlfoundry.in/learn/dilated-convolution  
Section: Deep Learning › Convolutional Neural Networks  
Level: core · 11 min read · updated 2026-10-11

**Dilated convolution:** (also **atrous convolution**, from the French *à trous*, "with holes") is a convolution whose kernel reads the input at taps spaced $d$ pixels apart instead of adjacent pixels: $y[i]=\sum_t w[t]\,x[i+d\,t]$. The integer $d\ge1$ is the **dilation rate**; $d=1$ is the ordinary convolution. The kernel keeps its $k$ weights per axis, but they span $k+(k-1)(d-1)$ pixels.

Dense prediction tasks such as semantic segmentation, depth estimation and audio generation need two things at once: a **large receptive field** (to see context) and **full resolution** (to output one value per pixel or sample). The classic way to grow a receptive field is to downsample with pooling or strided convolution (see **Stride and padding** and **Receptive field**), but that throws away resolution, which then has to be recovered with upsampling. Dilation grows the receptive field while keeping stride 1 and the parameter count unchanged.

The idea is old. The *algorithme à trous* of Holschneider, Kronland-Martinet, Morlet and Tchamitchian (1989) computes the undecimated wavelet transform by inserting zeros into the filters instead of subsampling the signal. In deep learning, Chen, Papandreou, Kokkinos, Murphy and Yuille (DeepLab, ICLR 2015) used atrous convolution to remove the last downsampling steps of an ImageNet-pretrained VGG while keeping its weights valid. Yu and Koltun ("Multi-Scale Context Aggregation by Dilated Convolutions", ICLR 2016) named it *dilated convolution*, built a context module from stacked layers with exponentially increasing dilation, and proved the exponential receptive-field growth. The same year van den Oord et al.'s **WaveNet** (2016) used causal dilated 1-D convolutions to model raw audio.

## Formula

$$
y[i,j]=\sum_{u=0}^{k-1}\sum_{v=0}^{k-1} w[u,v]\;x\big[i+d\,u,\;j+d\,v\big],\qquad k_{\text{eff}}=k+(k-1)(d-1)
$$

- $d$: dilation rate: the spacing between neighbouring taps ($d-1$ skipped pixels, the "holes")
- $k$: kernel size along one axis; the number of weights is still $k^2$ per (input, output) channel pair
- $k_{\text{eff}}$: effective kernel size, the span of the taps: $3\times3$ with $d=2$ covers $5\times5$, $d=4$ covers $9\times9$, $d=8$ covers $17\times17$
- $w$: the same learned weights as an ordinary $k\times k$ conv; equivalently a $k_{\text{eff}}\times k_{\text{eff}}$ kernel with zeros inserted between the taps

Output size uses the effective kernel. PyTorch's rule for one axis is $\lfloor (n+2p-d(k-1)-1)/s\rfloor+1$. With stride 1 and $k=3$, choosing $p=d$ keeps the size: a $32\times32$ map with $d=2,\ p=2$ stays $32\times32$, and a $65\times65$ map with $d=6,\ p=6$ stays $65\times65$. With $p=0$ each dilated layer trims $d(k-1)$ pixels instead of $k-1$.

## Gradients

Dilation changes only *which* input pixels a tap touches, so the backward pass is the same bookkeeping as an ordinary convolution with the same spacing. Writing $\delta[i]=\partial L/\partial y[i]$ in 1-D:

$$
\frac{\partial L}{\partial w[t]}=\sum_i \delta[i]\,x[i+d\,t],\qquad \frac{\partial L}{\partial x[j]}=\sum_{t}\,w[t]\;\delta[j-d\,t]
$$

- $\partial L/\partial w[t]$: correlation of the input with the error map, sampled at spacing $d$
- $\partial L/\partial x[j]$: a full (zero-padded) correlation of $\delta$ with the flipped kernel at the same dilation: the transposed dilated convolution

**At a glance**

- **Parameters:** $k^2 C_{in} C_{out}$ (+ bias), independent of $d$
- **FLOPs:** same as the undilated conv on the same output size: $k^2$ multiply-adds per output per channel pair
- **Receptive field (stride 1):** $1+(k-1)\sum_l d_l$ after a stack of layers
- **Padding to keep size:** $p=d\,(k-1)/2$ for odd $k$, stride 1 (PyTorch: `padding="same"` also works)
- **Main uses:** semantic segmentation (DeepLab, Yu & Koltun's context module), audio and time series (WaveNet, TCNs), keeping resolution in late backbone stages
- **Known failure:** **gridding**: stacks whose rates share a common factor leave input pixels that no path reaches

## Exponential receptive field

With stride 1, a $k\times k$ layer at dilation $d$ adds $(k-1)d$ to the receptive field. Plain $3\times3$ layers add 2 each, so the field grows **linearly**: 3, 5, 7, 9. Doubling the dilation each layer, $d=1,2,4,8$, makes it grow **exponentially**: 3, 7, 15, 31, i.e. $2^{L+1}-1$ after $L$ layers. A plain stack needs 15 layers ($15\times9=135$ weights per channel pair) to reach the $31\times31$ field that four dilated layers reach with 36. Yu and Koltun's context module uses rates $1,1,2,4,8,16,1$ and reaches $67\times67$ with seven $3\times3$ layers.

> Figure: A stack of 3×3 convolutions with all-ones weights. The heat map counts, for every input pixel, how many paths lead to the centre output (darker blue = more paths; red = zero paths, never read), and the orange squares are the 9 taps of the last layer shown. The default rates 1, 2, 4, 8 give per-layer effective kernels 3, 5, 9, 17 and receptive fields 3, 7, 15, 31: all 961 pixels of the 31×31 field are read, with 36 weights, where plain 3×3 layers would need 15 layers. Try the preset 2, 2, 2, 2: the field is 17×17 but only 9 of every 17 rows and columns are reached, 81 of 289 pixels (28.0%). 2, 4, 8 is worse (26.8%), while the hybrid rates 1, 2, 5 cover the whole 17×17 field. Use the ± buttons to set each layer's rate and the slider to reveal layers one at a time.

## Gridding, and how to avoid it

Dilation has a price: a single dilated layer only looks at a sparse lattice of pixels. If every layer of a stack uses rates that are all multiples of some $g>1$, every path moves in steps of $g$, so the output depends only on input pixels whose offset from it is a multiple of $g$. Neighbouring outputs then read disjoint sub-lattices of the input. In 1-D two layers with $d=2$ have path counts $1,0,2,0,3,0,2,0,1$ over a 9-pixel field: every odd offset is a hole. In 2-D the holes form a grid, and the outputs show a matching checkerboard of inconsistent predictions, especially on thin structures.

Wang et al. ("Understanding Convolution for Semantic Segmentation", WACV 2018) named this **gridding** and proposed **hybrid dilated convolution (HDC)**: use a repeating group of rates with no common factor greater than 1, such as $1,2,5$ or $1,2,3$ instead of $2,2,2$ or $2,4,8$, chosen so the group's combined footprint has no holes. Yu, Koltun and Funkhouser's Dilated Residual Networks (2017) attack the same artefact by appending undilated layers ("degridding"). Starting a doubling sequence at $d=1$, as in $1,2,4,8$, also avoids holes: the first layer fills the gaps the later ones skip.

Stacks of 3×3 layers (stride 1), computed by the figure's library

| rates | receptive field | pixels read | weights / channel pair |
| --- | --- | --- | --- |
| 1, 1, 1, 1 | 9 × 9 | 100% | 36 |
| 1, 2, 4, 8 | 31 × 31 | 100% | 36 |
| 2, 2, 2, 2 | 17 × 17 | 28.0% (81 / 289) | 36 |
| 2, 4, 8 | 29 × 29 | 26.8% (225 / 841) | 27 |
| 1, 2, 5 (HDC) | 17 × 17 | 100% | 27 |
| 1, 1, 2, 4, 8, 16, 1 (Yu & Koltun) | 67 × 67 | 100% | 63 |

## Worked example: one dilated layer in 1-D

Take $x=(3,1,4,1,5,9,2,6,5,3)$, kernel $w=(1,2,-1)$, dilation $d=2$, no padding. The effective kernel spans $3+2\cdot1=5$ samples, so the output has $10-5+1=6$ values. Each output reads $x[i],\,x[i+2],\,x[i+4]$:

**Forward, then backward with L = Σ y (so every δ[i] = 1)**

1. $y[0]=1\cdot3+2\cdot4-1\cdot5=6$, $y[1]=1+2\cdot1-9=-6$, $y[2]=4+2\cdot5-2=12$.
2. $y[3]=1+2\cdot9-6=13$, $y[4]=5+2\cdot2-5=4$, $y[5]=9+2\cdot6-3=18$. So $y=(6,-6,12,13,4,18)$. With $d=1$ the same kernel gives 8 outputs, $(1,8,1,2,21,7,9,13)$.
3. Weight gradients: $\partial L/\partial w[t]=\sum_{i=0}^{5}x[i+2t]$, i.e. sums of $x[0..5]$, $x[2..7]$, $x[4..9]$: $(23,\,27,\,30)$.
4. Input gradients: $\partial L/\partial x[j]=\sum_t w[t]\,\mathbb 1[0\le j-2t\le5]=(1,1,3,3,2,2,1,1,-1,-1)$. Each $x[j]$ gets the weights of the taps that touched it; the values match a finite-difference check.

```python
import torch, torch.nn as nn, torch.nn.functional as F

# Same 9 weights, 5x5 footprint. padding = dilation keeps H and W for k = 3.
conv = nn.Conv2d(64, 64, kernel_size=3, dilation=2, padding=2)
print(conv(torch.randn(1, 64, 32, 32)).shape)     # torch.Size([1, 64, 32, 32])
print(sum(p.numel() for p in conv.parameters()))  # 36928 = 64*64*9 + 64, same as dilation=1

# 1-D check of the worked example
x = torch.tensor([[[3., 1, 4, 1, 5, 9, 2, 6, 5, 3]]], requires_grad=True)
w = torch.tensor([[[1., 2, -1]]], requires_grad=True)
y = F.conv1d(x, w, dilation=2)
y.sum().backward()
print(y.flatten().tolist())       # [6.0, -6.0, 12.0, 13.0, 4.0, 18.0]
print(w.grad.flatten().tolist())  # [23.0, 27.0, 30.0]

# WaveNet-style causal dilated stack (kernel 2, rates 1..512): receptive field 1024 samples
class CausalConv1d(nn.Conv1d):
    def forward(self, x):
        pad = self.dilation[0] * (self.kernel_size[0] - 1)
        return super().forward(F.pad(x, (pad, 0)))    # pad on the left only: no peeking ahead

stack = nn.Sequential(*[CausalConv1d(32, 32, kernel_size=2, dilation=2**i) for i in range(10)])

# Atrous spatial pyramid pooling: parallel 3x3 branches at several rates
class ASPP(nn.Module):
    def __init__(self, cin, cout, rates=(6, 12, 18)):
        super().__init__()
        self.branches = nn.ModuleList(
            [nn.Conv2d(cin, cout, 1)] +
            [nn.Conv2d(cin, cout, 3, padding=r, dilation=r) for r in rates])
        self.pool = nn.Sequential(nn.AdaptiveAvgPool2d(1), nn.Conv2d(cin, cout, 1))
        self.project = nn.Conv2d(cout * (len(rates) + 2), cout, 1)
    def forward(self, x):
        feats = [b(x) for b in self.branches]
        feats.append(F.interpolate(self.pool(x), size=x.shape[-2:], mode="bilinear"))
        return self.project(torch.cat(feats, dim=1))

# torchvision's DeepLabV3 dilates the last two ResNet stages (output stride 8)
from torchvision.models import resnet50
backbone = resnet50(replace_stride_with_dilation=[False, True, True])
```

Dilated convolutions in PyTorch, a WaveNet-style causal stack, and DeepLab's ASPP

## Where it is used

- **Semantic segmentation:** DeepLab replaces the strides of the last backbone stages with dilation (output stride 16 or 8 instead of 32) so the pretrained filters keep their receptive field at higher resolution. **ASPP** (atrous spatial pyramid pooling, DeepLabv2) runs parallel $3\times3$ branches at several rates; DeepLabv3 uses rates $6,12,18$ plus a $1\times1$ branch and global image pooling at output stride 16, doubled to $12,24,36$ at stride 8.
- **Audio and sequences:** WaveNet stacks causal dilated convolutions with rates $1,2,4,\dots,512$, repeated in several blocks, giving each block a 1024-sample receptive field. Temporal convolutional networks (Bai, Kolter and Koltun, 2018) use the same recipe for general sequence modelling, and ByteNet (Kalchbrenner et al., 2016) for translation.
- **Backbones and detectors:** Dilated ResNets (Yu, Koltun and Funkhouser, 2017) keep late-stage resolution for localisation; detectors such as DetNet and TridentNet use dilation to enlarge or vary receptive fields without pooling.

> **Gridding from equal or doubling rates without a d = 1 layer:** Rates like $2,2,2,2$ or $2,4,8$ share the factor 2, so only about a quarter of the pixels in the receptive field (28.0% and 26.8% in the figure) ever reach a given output. The symptom is a fine checkerboard in the predictions. Mix in rates with no common factor (HDC: $1,2,5$), start sequences at $d=1$, or add undilated layers at the end.

> **Large rates on small maps:** When $d$ approaches the feature-map size, most taps fall into zero padding and a $3\times3$ kernel degenerates to its centre tap, a $1\times1$ conv. DeepLabv3 observed this and added a global-average-pooling branch to ASPP to supply image-level context instead.

> **Dilation versus downsampling:** Striding and dilation reach the same receptive field, but striding also cuts the number of outputs by $s^2$ and so the compute of every later layer. Dilation keeps every output, so the compute and activation memory of the dilated stages grow: DeepLab at output stride 8 spends several times the FLOPs of the same backbone at stride 32. Use dilation where resolution is worth paying for.

> **Relation to the other convolutions:** The gradient with respect to the input of a dilated conv is a dilated **Transposed convolution**. **Deformable convolution** generalises dilation: instead of a fixed spacing $d$, every tap gets its own learned, fractional offset.

**Strengths**

- Exponential receptive-field growth with no extra parameters
- Keeps full resolution: no pooling, no information-losing striding
- A drop-in change: same weights, so pretrained filters can be dilated (DeepLab)
- Natural fit for causal 1-D models (WaveNet, TCN)

**Limitations**

- Gridding holes when rates share a common factor
- Sparse sampling ignores local detail at large rates
- Compute and memory of full-resolution maps are high compared with downsampling
- Strided memory access is less cache-friendly than dense 3×3 kernels on some hardware

**Question:** Four 3×3 convolutions with stride 1 and dilation rates 1, 2, 4, 8 are stacked. What is the receptive field of one output?

- A. 9 × 9
- B. 17 × 17
- C. 31 × 31
- D. 81 × 81

**Answer:** C. 31 × 31 — Each layer adds $(k-1)d=2d$: $1+2(1+2+4+8)=31$. A plain stack would give $1+2\cdot4=9$; the last layer alone has an effective kernel of $17\times17$.

**Question:** Why does the stack 2, 2, 2, 2 cover only 28% of its 17 × 17 receptive field?

- A. Padding zeros out the borders
- B. All rates are even, so every path moves in steps of 2 and odd offsets are never reached
- C. Dilated kernels have fewer weights
- D. The ReLU between layers kills half the paths

**Answer:** B. All rates are even, so every path moves in steps of 2 and odd offsets are never reached — Each tap offset is a multiple of 2, so any sum of offsets is too. Per axis only 9 of 17 positions are reachable, so $81/289=28.0\%$ of the pixels. Rates with no common factor, or a $d=1$ layer, fill the holes.

**Key points**

- Dilation $d$ spreads the $k$ taps $d$ apart: $k_{\text{eff}}=k+(k-1)(d-1)$, with the same $k^2$ weights and FLOPs.
- Stride-1 stacks have receptive field $1+(k-1)\sum d_l$: rates $1,2,4,8$ give 31 with four layers, versus 9 for plain $3\times3$.
- Keep the size with $p=d(k-1)/2$; PyTorch: `nn.Conv2d(c, c, 3, padding=d, dilation=d)`.
- Rates that share a common factor cause gridding; hybrid rates (1, 2, 5) or a leading d = 1 layer avoid it.
- Used for dense prediction at high resolution (DeepLab, ASPP) and causal sequence models (WaveNet, TCN).

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
