DeepLab

DeepLab (Chen et al., 2015–2018) keeps feature maps dense with atrous convolution, captures multiple scales with ASPP (rates 6, 12, 18 at output stride 16), and in v3+ adds a light decoder: 89.0% mIoU on PASCAL VOC 2012 test.

8 min read865 wordscore0 viewsupdated 2026-10-15

Definition

DeepLab is a family of semantic segmentation networks led by Liang-Chieh Chen and George Papandreou. v1 ("Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs", Chen, Papandreou, Kokkinos, Murphy and Yuille, ICLR 2015) added atrous convolution and a fully connected CRF to VGG-16. v2 (TPAMI 2018, arXiv 2016) added atrous spatial pyramid pooling (ASPP) on ResNet-101. v3 ("Rethinking Atrous Convolution for Semantic Image Segmentation", Chen, Papandreou, Schroff and Adam, 2017) improved ASPP and dropped the CRF. v3+ ("Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation", Chen, Zhu, Papandreou, Schroff and Adam, ECCV 2018) added a decoder.

Semantic segmentation maps an image to one class label per pixel. Classification backbones are a poor fit as they stand: five stride-2 stages shrink a 513×513513\times513 crop to 17×1717\times17 (output stride 32), and pooling makes features invariant to small shifts, which blurs boundaries. FCN recovered resolution with learned upsampling and skip connections.

DeepLab takes the other route. Remove the stride from the last stages so the map stays at output stride (OS) 16 or 8, and dilate the following convolutions so each filter still sees as much context as before. The weights of a pretrained network can be kept unchanged.

Atrous convolution

An atrous (dilated) convolution reads its input with gaps of r−1r-1 pixels between taps. The name comes from the algorithme à trous ("with holes") of wavelet analysis. See Dilated convolution for the operator itself; here it matters for what it does to resolution.

y[i]=∑kx[i+r⋅k] w[k],keff=k+(k−1)(r−1),F=S−1OS+1y[i]=\sum_{k}x[i+r\cdot k]\,w[k],\qquad k_{\text{eff}}=k+(k-1)(r-1),\qquad F=\frac{S-1}{\text{OS}}+1
rr
atrous rate; r=1r=1 is an ordinary convolution
keffk_{\text{eff}}
effective kernel size: a 3×33\times3 filter with r=12r=12 spans 25×2525\times25 cells but still has 9 weights
F, SF,\ S
feature-map side and crop side; DeepLab crops are 513513, so F=65, 33, 17F=65,\,33,\,17 at OS 8, 16, 328,\,16,\,32

To go from OS 32 to OS 8 in ResNet, set the stride of the last two stages to 1 and dilate their 3×33\times3 convolutions by 2 and 4. The cost is that every later layer now runs on 652=4,22565^2=4{,}225 positions instead of 172=28917^2=289. That is why v3 trains at OS 16, where the map is 33×3333\times33, and evaluates at OS 8 with all rates doubled.

ASPP and the decoder

Objects appear at many scales. ASPP runs several atrous convolutions with different rates in parallel on the same map and fuses the results. v2 used four 3×33\times3 branches with rates {6,12,18,24}\{6,12,18,24\} and summed their class scores. v3 uses a 1×11\times1 convolution, three 3×33\times3 convolutions with rates {6,12,18}\{6,12,18\} at OS 16, and an image-level pooling branch, each with 256 filters and batch normalization, then concatenates them and projects back to 256 channels.

DeepLabv3+: atrous encoder with ASPP, light decoder (513×513 input, output stride 16)v3 = encoder only (bilinear ↑16); v1/v2: OS 8, bilinear ↑8, then a fully connected CRF513×513×3backboneResNet-101 oraligned Xceptionstrides → dilationOS 32 → 1633×33×20481×1 conv2563×3 rate 62563×3 rate 122563×3 rate 18256image pool1×1, 256, ↑ASPP (rates ×2 at OS 8)concat 12801×1 conv → 25633×33×256bilinear ↑4low-level, OS 4129×129×2561×1 conv → 48129×129×48129×129×256encoder, ↑4concat 304 → two 3×3, 2561×1 → 21 classes (VOC)↑4513×513×21decoder recoversboundaries fromstride-4 featuresF = 33: a rate-183×3 spans 37 cells,more than the map
fig.DeepLabv3+ on a 513×513 crop. The atrous backbone stops at output stride 16 (33×33×2048). ASPP runs a 1×1 conv, three 3×3 atrous convs (rates 6, 12, 18; the dot patterns show the tap spacing) and global image pooling, 256 channels each. The 1,280 concatenated channels are projected to 256 and upsampled 4×. The decoder concatenates them with stride-4 backbone features reduced to 48 channels, refines with two 3×3 convs and upsamples 4× to per-pixel scores. v3 stops after ASPP and upsamples 16× directly.

The image-pooling branch fixes a failure that the figure below shows. As the rate approaches the map size, most taps of a 3×33\times3 filter fall into the zero padding, and in the limit only the centre weight remains: the filter becomes a 1×11\times1 convolution. Global average pooling, a 1×11\times1 convolution and bilinear upsampling supply the image-wide context instead.

v3+ adds a simple decoder. The encoder output is upsampled 4×4\times and concatenated with stride-4 features from the backbone (reduced to 48 channels so they do not dominate), followed by two 3×33\times3 convolutions and a final 4×4\times upsampling. It also uses depthwise separable atrous convolutions in ASPP and the decoder, on a modified aligned Xception backbone.

The CRF in v1 and v2

Early DeepLab score maps were smooth and blob-like. v1 and v2 sharpened them with the fully connected CRF of Krähenbühl and Koltun (2011), with labels xix_i for every pixel and a pairwise term between all pixel pairs:

E(x)=∑i−log⁡P(xi)+∑i<jμ(xi,xj)[w1e−∥pi−pj∥22σα2−∥Ii−Ij∥22σβ2+w2e−∥pi−pj∥22σγ2]E(\mathbf x)=\sum_i -\log P(x_i)+\sum_{i<j}\mu(x_i,x_j)\Big[w_1e^{-\frac{\|p_i-p_j\|^2}{2\sigma_\alpha^2}-\frac{\|I_i-I_j\|^2}{2\sigma_\beta^2}}+w_2e^{-\frac{\|p_i-p_j\|^2}{2\sigma_\gamma^2}}\Big]
P(xi)P(x_i)
the network's class probability at pixel ii
μ\mu
Potts compatibility: 11 if xi≠xjx_i\ne x_j, else 00
p, Ip,\ I
pixel position and RGB colour; the first kernel makes nearby pixels of similar colour agree, the second smooths locally

Mean-field inference with fast high-dimensional Gaussian filtering makes the dense CRF tractable despite its O(N2)O(N^2) pairs. v3 showed that a better backbone and ASPP made the CRF unnecessary, and later versions drop it.

VersionBackboneWhat it addedVOC test mIoU
v1 (2015)VGG-16, OS 8atrous conv, dense CRF71.6%
v2 (2016/2018)ResNet-101, OS 8ASPP (rates 6, 12, 18, 24), CRF79.7%
v3 (2017)ResNet-101, OS 16/8ASPP + image pooling + BN, multi-grid, no CRF85.7%
v3+ (2018)aligned Xception-65decoder, separable atrous convs89.0% (JFT pretraining)
The DeepLab series on the PASCAL VOC 2012 test set (21 classes, mean IoU)

v3+ also reported 82.1% mIoU on the Cityscapes test set. The later scores rely on extra pretraining (COCO and, for the top v3+ entry, Google's internal JFT-300M), so the rows mix architecture and data gains.

3×3 atrous conv, rate 12, on the 33×33 map of a 513×513 crop at output stride 16shade = valid taps at each position (zero padding outside) · click the map to move the kernelkernel at (8, 8): 4/9 taps inside · dashed = in the zero padding (pinned to the edge)mean valid taps (of 9) vs rate13579161218243036ASPP rates at OS 16: 6, 12, 18OS8OS16OS32effective kernel 25×25 cells · outer taps 384 px apart in the input (75% of the 513 crop)mean valid taps 5.17/9 · all 9 valid at 7.4% of positions · centre only at 0.0%1089 positions; OS 8 has 4225, OS 16 has 1089: halving the stride costs 3.9× per layer
output stride
fig.A 3×3 atrous convolution on the map a 513×513 crop leaves at output stride 8, 16 or 32, with zero padding; shading counts the taps that land inside the map. At the default (OS 16, 33×33, rate 12) the kernel at (8, 8) has 4 of 9 taps inside, the average is 5.17 of 9, and only 7.4% of positions see all nine. At rate 18 the average is 3.64 and no position sees all nine. From rate 33 every off-centre tap is padding: 1.00 of 9, a 1×1 filter. At OS 8, rate 24 gives 5.11, nearly the same as rate 12 at OS 16, which is why the rates double.

Worked example: ASPP rates on a 33×33 map

Rate rk_eff = 2r + 1Outer taps apart (input px)Mean valid tapsPositions with all 9
132 · 1 · 16 = 328.64 / 988.2%
6132 · 6 · 16 = 1926.95 / 940.5%
12252 · 12 · 16 = 3845.17 / 97.4%
18372 · 18 · 16 = 576 (> 513)3.64 / 90.0%
33671,0561.00 / 90.0%
3×3 atrous filters at output stride 16 (one feature cell = 16 input pixels); zero padding, averages over all 1,089 positions

In 1-D, a tap at p+rp+r is inside only if p+r≤32p+r\le32. For r=18r=18 that holds for p≤14p\le14, and the tap at p−18p-18 for p≥18p\ge18, so no position has both: an interior tap triple never fits. The rate-18 branch therefore always works with at most 4 of its 9 weights (2 per axis), and its outer taps span more than the crop.

python
1import torch
2from torchvision.io import read_image
3from torchvision.models.segmentation import deeplabv3_resnet101, DeepLabV3_ResNet101_Weights
4
5weights = DeepLabV3_ResNet101_Weights.DEFAULT # trained on a COCO subset with the 21 VOC classes
6model = deeplabv3_resnet101(weights=weights).eval() # ResNet-101 at output stride 8, ASPP rates 12/24/36
7img = read_image("street.jpg") # uint8 tensor (3, H, W)
8x = weights.transforms()(img).unsqueeze(0) # resize + normalise
9with torch.no_grad():
10 logits = model(x)["out"] # (1, 21, h, w), upsampled to the input size
11mask = logits.argmax(dim=1)[0] # class index per pixel
12
13# a 3x3 atrous branch keeps the map size when padding = dilation
14branch = torch.nn.Conv2d(2048, 256, kernel_size=3, padding=12, dilation=12, bias=False)
15print(branch(torch.randn(1, 2048, 33, 33)).shape) # torch.Size([1, 256, 33, 33])
Inference with torchvision's DeepLabv3 and one ASPP branch (not run here: no PyTorch in the sandbox)

Check yourself

DeepLabv3 at output stride 16 on a 513×513 crop has a 33×33 feature map. What does a 3×3 atrous convolution with rate 36 compute there?

Check yourself

You want ResNet-101 features at output stride 8 for segmentation. What is the DeepLab recipe?

Remember

  • ✓Atrous convolution: y[i]=∑kx[i+rk]w[k]y[i]=\sum_k x[i+rk]w[k]; removes downsampling while keeping the field of view, so features stay at OS 16 or 8.
  • ✓v1/v2: VGG-16 then ResNet-101 at OS 8, plus a fully connected CRF; v2 introduced ASPP (rates 6, 12, 18, 24).
  • ✓v3: ASPP = 1×1 + three 3×3 at rates 6, 12, 18 (OS 16; doubled at OS 8) + image pooling, 256 ch each, BN, no CRF; 85.7% VOC test.
  • ✓Large rates on small maps degenerate to 1×1: at rate 18 on 33×33 only 3.64 of 9 taps are valid on average.
  • ✓v3+: decoder fuses ×4-upsampled ASPP output with 48-ch stride-4 features; 89.0% VOC test, 82.1% Cityscapes test.
← back to Deep Learning roadmap