# DeepLab

> DeepLab (Chen et al., 2015–2018) keeps feature maps dense with atrous convolution, captures multiple scales with ASPP (rates 6, 12, 18 at output stride 16), and in v3+ adds a light decoder: 89.0% mIoU on PASCAL VOC 2012 test.

Source: https://www.mlfoundry.in/learn/deeplab  
Section: Deep Learning › Deep Learning for Computer Vision  
Level: core · 8 min read · updated 2026-10-15

**DeepLab:** is a family of semantic segmentation networks led by Liang-Chieh Chen and George Papandreou. **v1** ("Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs", Chen, Papandreou, Kokkinos, Murphy and Yuille, ICLR 2015) added atrous convolution and a fully connected CRF to VGG-16. **v2** (TPAMI 2018, arXiv 2016) added atrous spatial pyramid pooling (ASPP) on ResNet-101. **v3** ("Rethinking Atrous Convolution for Semantic Image Segmentation", Chen, Papandreou, Schroff and Adam, 2017) improved ASPP and dropped the CRF. **v3+** ("Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation", Chen, Zhu, Papandreou, Schroff and Adam, ECCV 2018) added a decoder.

Semantic segmentation maps an image to one class label per pixel. Classification backbones are a poor fit as they stand: five stride-2 stages shrink a $513\times513$ crop to $17\times17$ (output stride 32), and pooling makes features invariant to small shifts, which blurs boundaries. **FCN** recovered resolution with learned upsampling and skip connections.

DeepLab takes the other route. Remove the stride from the last stages so the map stays at output stride (OS) 16 or 8, and dilate the following convolutions so each filter still sees as much context as before. The weights of a pretrained network can be kept unchanged.

## Atrous convolution

An atrous (dilated) convolution reads its input with gaps of $r-1$ pixels between taps. The name comes from the *algorithme à trous* ("with holes") of wavelet analysis. See **Dilated convolution** for the operator itself; here it matters for what it does to resolution.

$$
y[i]=\sum_{k}x[i+r\cdot k]\,w[k],\qquad k_{\text{eff}}=k+(k-1)(r-1),\qquad F=\frac{S-1}{\text{OS}}+1
$$

- $r$: atrous rate; $r=1$ is an ordinary convolution
- $k_{\text{eff}}$: effective kernel size: a $3\times3$ filter with $r=12$ spans $25\times25$ cells but still has 9 weights
- $F,\ S$: feature-map side and crop side; DeepLab crops are $513$, so $F=65,\,33,\,17$ at OS $8,\,16,\,32$

To go from OS 32 to OS 8 in ResNet, set the stride of the last two stages to 1 and dilate their $3\times3$ convolutions by 2 and 4. The cost is that every later layer now runs on $65^2=4{,}225$ positions instead of $17^2=289$. That is why v3 trains at OS 16, where the map is $33\times33$, and evaluates at OS 8 with all rates doubled.

## ASPP and the decoder

Objects appear at many scales. **ASPP** runs several atrous convolutions with different rates in parallel on the same map and fuses the results. v2 used four $3\times3$ branches with rates $\{6,12,18,24\}$ and summed their class scores. v3 uses a $1\times1$ convolution, three $3\times3$ convolutions with rates $\{6,12,18\}$ at OS 16, and an **image-level pooling** branch, each with 256 filters and batch normalization, then concatenates them and projects back to 256 channels.

> Figure: DeepLabv3+ on a 513×513 crop. The atrous backbone stops at output stride 16 (33×33×2048). ASPP runs a 1×1 conv, three 3×3 atrous convs (rates 6, 12, 18; the dot patterns show the tap spacing) and global image pooling, 256 channels each. The 1,280 concatenated channels are projected to 256 and upsampled 4×. The decoder concatenates them with stride-4 backbone features reduced to 48 channels, refines with two 3×3 convs and upsamples 4× to per-pixel scores. v3 stops after ASPP and upsamples 16× directly.

The image-pooling branch fixes a failure that the figure below shows. As the rate approaches the map size, most taps of a $3\times3$ filter fall into the zero padding, and in the limit only the centre weight remains: the filter becomes a $1\times1$ convolution. Global average pooling, a $1\times1$ convolution and bilinear upsampling supply the image-wide context instead.

v3+ adds a simple decoder. The encoder output is upsampled $4\times$ and concatenated with stride-4 features from the backbone (reduced to 48 channels so they do not dominate), followed by two $3\times3$ convolutions and a final $4\times$ upsampling. It also uses depthwise separable atrous convolutions in ASPP and the decoder, on a modified aligned **Xception** backbone.

## The CRF in v1 and v2

Early DeepLab score maps were smooth and blob-like. v1 and v2 sharpened them with the fully connected CRF of Krähenbühl and Koltun (2011), with labels $x_i$ for every pixel and a pairwise term between **all** pixel pairs:

$$
E(\mathbf x)=\sum_i -\log P(x_i)+\sum_{i<j}\mu(x_i,x_j)\Big[w_1e^{-\frac{\|p_i-p_j\|^2}{2\sigma_\alpha^2}-\frac{\|I_i-I_j\|^2}{2\sigma_\beta^2}}+w_2e^{-\frac{\|p_i-p_j\|^2}{2\sigma_\gamma^2}}\Big]
$$

- $P(x_i)$: the network's class probability at pixel $i$
- $\mu$: Potts compatibility: $1$ if $x_i\ne x_j$, else $0$
- $p,\ I$: pixel position and RGB colour; the first kernel makes nearby pixels of similar colour agree, the second smooths locally

Mean-field inference with fast high-dimensional Gaussian filtering makes the dense CRF tractable despite its $O(N^2)$ pairs. v3 showed that a better backbone and ASPP made the CRF unnecessary, and later versions drop it.

The DeepLab series on the PASCAL VOC 2012 test set (21 classes, mean IoU)

| Version | Backbone | What it added | VOC test mIoU |
| --- | --- | --- | --- |
| v1 (2015) | VGG-16, OS 8 | atrous conv, dense CRF | 71.6% |
| v2 (2016/2018) | ResNet-101, OS 8 | ASPP (rates 6, 12, 18, 24), CRF | 79.7% |
| v3 (2017) | ResNet-101, OS 16/8 | ASPP + image pooling + BN, multi-grid, no CRF | 85.7% |
| v3+ (2018) | aligned Xception-65 | decoder, separable atrous convs | 89.0% (JFT pretraining) |

v3+ also reported 82.1% mIoU on the Cityscapes test set. The later scores rely on extra pretraining (COCO and, for the top v3+ entry, Google's internal JFT-300M), so the rows mix architecture and data gains.

> Figure: A 3×3 atrous convolution on the map a 513×513 crop leaves at output stride 8, 16 or 32, with zero padding; shading counts the taps that land inside the map. At the default (OS 16, 33×33, rate 12) the kernel at (8, 8) has 4 of 9 taps inside, the average is 5.17 of 9, and only 7.4% of positions see all nine. At rate 18 the average is 3.64 and no position sees all nine. From rate 33 every off-centre tap is padding: 1.00 of 9, a 1×1 filter. At OS 8, rate 24 gives 5.11, nearly the same as rate 12 at OS 16, which is why the rates double.

## Worked example: ASPP rates on a 33×33 map

3×3 atrous filters at output stride 16 (one feature cell = 16 input pixels); zero padding, averages over all 1,089 positions

| Rate r | k_eff = 2r + 1 | Outer taps apart (input px) | Mean valid taps | Positions with all 9 |
| --- | --- | --- | --- | --- |
| 1 | 3 | 2 · 1 · 16 = 32 | 8.64 / 9 | 88.2% |
| 6 | 13 | 2 · 6 · 16 = 192 | 6.95 / 9 | 40.5% |
| 12 | 25 | 2 · 12 · 16 = 384 | 5.17 / 9 | 7.4% |
| 18 | 37 | 2 · 18 · 16 = 576 (> 513) | 3.64 / 9 | 0.0% |
| 33 | 67 | 1,056 | 1.00 / 9 | 0.0% |

In 1-D, a tap at $p+r$ is inside only if $p+r\le32$. For $r=18$ that holds for $p\le14$, and the tap at $p-18$ for $p\ge18$, so no position has both: an interior tap triple never fits. The rate-18 branch therefore always works with at most 4 of its 9 weights (2 per axis), and its outer taps span more than the crop.

```python
import torch
from torchvision.io import read_image
from torchvision.models.segmentation import deeplabv3_resnet101, DeepLabV3_ResNet101_Weights

weights = DeepLabV3_ResNet101_Weights.DEFAULT            # trained on a COCO subset with the 21 VOC classes
model = deeplabv3_resnet101(weights=weights).eval()      # ResNet-101 at output stride 8, ASPP rates 12/24/36
img = read_image("street.jpg")                           # uint8 tensor (3, H, W)
x = weights.transforms()(img).unsqueeze(0)               # resize + normalise
with torch.no_grad():
    logits = model(x)["out"]                             # (1, 21, h, w), upsampled to the input size
mask = logits.argmax(dim=1)[0]                           # class index per pixel

# a 3x3 atrous branch keeps the map size when padding = dilation
branch = torch.nn.Conv2d(2048, 256, kernel_size=3, padding=12, dilation=12, bias=False)
print(branch(torch.randn(1, 2048, 33, 33)).shape)        # torch.Size([1, 256, 33, 33])
```

Inference with torchvision's DeepLabv3 and one ASPP branch (not run here: no PyTorch in the sandbox)

> **Rates are tied to the output stride:** ASPP rates are measured in feature cells, not pixels. Rates $\{6,12,18\}$ are for OS 16; running the same weights at OS 8 needs $\{12,24,36\}$ to keep the same field of view, which is what DeepLabv3 does at evaluation. Changing the crop size has a similar effect: on a small map, large rates degenerate toward $1\times1$.

> **Resolution without throwing away the pretrained weights:** Atrous convolution lets a network pretrained for classification produce dense features without retraining from scratch: the filters are unchanged, only their sampling grid is spread out. The price is compute and memory, roughly $4\times$ per later layer for each halving of the output stride.

**Question:** DeepLabv3 at output stride 16 on a 513×513 crop has a 33×33 feature map. What does a 3×3 atrous convolution with rate 36 compute there?

- A. A 73×73 filter with 9 weights covering the whole image
- B. Effectively a 1×1 convolution, because every off-centre tap falls in the zero padding
- C. The same as rate 18, because the rate wraps around
- D. Global average pooling

**Answer:** B. Effectively a 1×1 convolution, because every off-centre tap falls in the zero padding — With $r=36>32$, a tap at $p\pm36$ is outside a 33-cell map for every $p$, so only the centre weight ever multiplies real data. This degeneration is why v3 adds an explicit image-pooling branch for global context.

**Question:** You want ResNet-101 features at output stride 8 for segmentation. What is the DeepLab recipe?

- A. Upsample the stride-32 output 4× with bilinear interpolation
- B. Remove the stride of the last two stages and dilate their 3×3 convolutions by 2 and 4
- C. Feed a 4× larger input image
- D. Replace every convolution with a 1×1 convolution

**Answer:** B. Remove the stride of the last two stages and dilate their 3×3 convolutions by 2 and 4 — Removing a stride doubles the map size; dilating the following convolutions by the accumulated factor (2, then 4) keeps their field of view, so the pretrained weights stay valid. Bilinear upsampling adds no detail, and a larger input changes the scale the network was trained for.

**Key points**

- Atrous convolution: $y[i]=\sum_k x[i+rk]w[k]$; removes downsampling while keeping the field of view, so features stay at OS 16 or 8.
- v1/v2: VGG-16 then ResNet-101 at OS 8, plus a fully connected CRF; v2 introduced ASPP (rates 6, 12, 18, 24).
- v3: ASPP = 1×1 + three 3×3 at rates 6, 12, 18 (OS 16; doubled at OS 8) + image pooling, 256 ch each, BN, no CRF; 85.7% VOC test.
- Large rates on small maps degenerate to 1×1: at rate 18 on 33×33 only 3.64 of 9 taps are valid on average.
- v3+: decoder fuses ×4-upsampled ASPP output with 48-ch stride-4 features; 89.0% VOC test, 82.1% Cityscapes test.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
