Definition
DeepLab is a family of semantic segmentation networks led by Liang-Chieh Chen and George Papandreou. v1 ("Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs", Chen, Papandreou, Kokkinos, Murphy and Yuille, ICLR 2015) added atrous convolution and a fully connected CRF to VGG-16. v2 (TPAMI 2018, arXiv 2016) added atrous spatial pyramid pooling (ASPP) on ResNet-101. v3 ("Rethinking Atrous Convolution for Semantic Image Segmentation", Chen, Papandreou, Schroff and Adam, 2017) improved ASPP and dropped the CRF. v3+ ("Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation", Chen, Zhu, Papandreou, Schroff and Adam, ECCV 2018) added a decoder.
Semantic segmentation maps an image to one class label per pixel. Classification backbones are a poor fit as they stand: five stride-2 stages shrink a crop to (output stride 32), and pooling makes features invariant to small shifts, which blurs boundaries. FCN recovered resolution with learned upsampling and skip connections.
DeepLab takes the other route. Remove the stride from the last stages so the map stays at output stride (OS) 16 or 8, and dilate the following convolutions so each filter still sees as much context as before. The weights of a pretrained network can be kept unchanged.
Atrous convolution
An atrous (dilated) convolution reads its input with gaps of pixels between taps. The name comes from the algorithme à trous ("with holes") of wavelet analysis. See Dilated convolution for the operator itself; here it matters for what it does to resolution.
- atrous rate; is an ordinary convolution
- effective kernel size: a filter with spans cells but still has 9 weights
- feature-map side and crop side; DeepLab crops are , so at OS
To go from OS 32 to OS 8 in ResNet, set the stride of the last two stages to 1 and dilate their convolutions by 2 and 4. The cost is that every later layer now runs on positions instead of . That is why v3 trains at OS 16, where the map is , and evaluates at OS 8 with all rates doubled.
ASPP and the decoder
Objects appear at many scales. ASPP runs several atrous convolutions with different rates in parallel on the same map and fuses the results. v2 used four branches with rates and summed their class scores. v3 uses a convolution, three convolutions with rates at OS 16, and an image-level pooling branch, each with 256 filters and batch normalization, then concatenates them and projects back to 256 channels.
The image-pooling branch fixes a failure that the figure below shows. As the rate approaches the map size, most taps of a filter fall into the zero padding, and in the limit only the centre weight remains: the filter becomes a convolution. Global average pooling, a convolution and bilinear upsampling supply the image-wide context instead.
v3+ adds a simple decoder. The encoder output is upsampled and concatenated with stride-4 features from the backbone (reduced to 48 channels so they do not dominate), followed by two convolutions and a final upsampling. It also uses depthwise separable atrous convolutions in ASPP and the decoder, on a modified aligned Xception backbone.
The CRF in v1 and v2
Early DeepLab score maps were smooth and blob-like. v1 and v2 sharpened them with the fully connected CRF of Krähenbühl and Koltun (2011), with labels for every pixel and a pairwise term between all pixel pairs:
- the network's class probability at pixel
- Potts compatibility: if , else
- pixel position and RGB colour; the first kernel makes nearby pixels of similar colour agree, the second smooths locally
Mean-field inference with fast high-dimensional Gaussian filtering makes the dense CRF tractable despite its pairs. v3 showed that a better backbone and ASPP made the CRF unnecessary, and later versions drop it.
| Version | Backbone | What it added | VOC test mIoU |
|---|---|---|---|
| v1 (2015) | VGG-16, OS 8 | atrous conv, dense CRF | 71.6% |
| v2 (2016/2018) | ResNet-101, OS 8 | ASPP (rates 6, 12, 18, 24), CRF | 79.7% |
| v3 (2017) | ResNet-101, OS 16/8 | ASPP + image pooling + BN, multi-grid, no CRF | 85.7% |
| v3+ (2018) | aligned Xception-65 | decoder, separable atrous convs | 89.0% (JFT pretraining) |
v3+ also reported 82.1% mIoU on the Cityscapes test set. The later scores rely on extra pretraining (COCO and, for the top v3+ entry, Google's internal JFT-300M), so the rows mix architecture and data gains.
Worked example: ASPP rates on a 33×33 map
| Rate r | k_eff = 2r + 1 | Outer taps apart (input px) | Mean valid taps | Positions with all 9 |
|---|---|---|---|---|
| 1 | 3 | 2 · 1 · 16 = 32 | 8.64 / 9 | 88.2% |
| 6 | 13 | 2 · 6 · 16 = 192 | 6.95 / 9 | 40.5% |
| 12 | 25 | 2 · 12 · 16 = 384 | 5.17 / 9 | 7.4% |
| 18 | 37 | 2 · 18 · 16 = 576 (> 513) | 3.64 / 9 | 0.0% |
| 33 | 67 | 1,056 | 1.00 / 9 | 0.0% |
In 1-D, a tap at is inside only if . For that holds for , and the tap at for , so no position has both: an interior tap triple never fits. The rate-18 branch therefore always works with at most 4 of its 9 weights (2 per axis), and its outer taps span more than the crop.
1import torch2from torchvision.io import read_image3from torchvision.models.segmentation import deeplabv3_resnet101, DeepLabV3_ResNet101_Weights45weights = DeepLabV3_ResNet101_Weights.DEFAULT # trained on a COCO subset with the 21 VOC classes6model = deeplabv3_resnet101(weights=weights).eval() # ResNet-101 at output stride 8, ASPP rates 12/24/367img = read_image("street.jpg") # uint8 tensor (3, H, W)8x = weights.transforms()(img).unsqueeze(0) # resize + normalise9with torch.no_grad():10 logits = model(x)["out"] # (1, 21, h, w), upsampled to the input size11mask = logits.argmax(dim=1)[0] # class index per pixel1213# a 3x3 atrous branch keeps the map size when padding = dilation14branch = torch.nn.Conv2d(2048, 256, kernel_size=3, padding=12, dilation=12, bias=False)15print(branch(torch.randn(1, 2048, 33, 33)).shape) # torch.Size([1, 256, 33, 33])Check yourself
DeepLabv3 at output stride 16 on a 513×513 crop has a 33×33 feature map. What does a 3×3 atrous convolution with rate 36 compute there?
Check yourself
You want ResNet-101 features at output stride 8 for segmentation. What is the DeepLab recipe?
Remember
- ✓Atrous convolution: ; removes downsampling while keeping the field of view, so features stay at OS 16 or 8.
- ✓v1/v2: VGG-16 then ResNet-101 at OS 8, plus a fully connected CRF; v2 introduced ASPP (rates 6, 12, 18, 24).
- ✓v3: ASPP = 1×1 + three 3×3 at rates 6, 12, 18 (OS 16; doubled at OS 8) + image pooling, 256 ch each, BN, no CRF; 85.7% VOC test.
- ✓Large rates on small maps degenerate to 1×1: at rate 18 on 33×33 only 3.64 of 9 taps are valid on average.
- ✓v3+: decoder fuses ×4-upsampled ASPP output with 48-ch stride-4 features; 89.0% VOC test, 82.1% Cityscapes test.