Definition
Xception is the CNN of François Chollet, "Xception: Deep Learning with Depthwise Separable Convolutions" (CVPR 2017). It replaces every Inception module by depthwise separable convolutions: a per-channel spatial conv followed by a conv that mixes channels. The 36 conv layers are arranged in an entry flow, a middle flow repeated 8 times and an exit flow, with residual connections around almost every block. The name is short for "Extreme Inception".
Chollet's starting point is a hypothesis about what makes Inception work. A regular conv layer learns filters over two spatial dimensions and the channel dimension at once: one kernel handles cross-channel and spatial correlations together. An Inception module first looks at cross-channel correlations with convs, mapping the input into a few smaller spaces, and then looks at spatial correlations in each of those spaces with or convs. The implicit assumption is that the two kinds of correlation are decoupled enough that it is better not to map them jointly.
A simplified Inception module (one conv, then convs on non-overlapping groups of its output channels) makes this a continuum. With 1 group it is a conv followed by a full conv. With 3 or 4 groups it resembles Inception's parallel towers. With one group per channel it becomes a depthwise separable convolution: the conv handles all cross-channel mixing and the conv handles only spatial mixing, one channel at a time. Xception asks whether the extreme end of the continuum works best, and finds that it does, at least at the scale tested.
Depthwise separable convs were not new. Sifre and Mallat studied them (2013–2014), Google Brain used them in 2014, and they were available as SeparableConv2D in TensorFlow and Keras, which Chollet created. MobileNets (Howard et al. 2017) appeared at about the same time and used them for efficiency on phones. Xception's contribution is the interpretation as extreme Inception, a clean architecture built from them, and the evidence that they match Inception at the same parameter count.
At a glance
- Paper
- François Chollet, CVPR 2017 (arXiv Oct. 2016)
- Building block
- depthwise conv (no nonlinearity) then pointwise conv, BN after each separable conv
- Layout
- 36 conv layers in 14 modules: entry flow (4), middle flow (8 identical blocks at 19×19×728), exit flow (2)
- Parameters
- 22,855,952 (paper, Table 3); Inception-v3 in the same table: 23,626,728
- Compute
- ≈8.4G multiply-adds at (computed from the layout)
- ImageNet
- 79.0% top-1, 94.5% top-5 single crop vs Inception-v3's 78.2% / 94.1% (paper, Table 1)
- In PyTorch
- not in torchvision;
timm.create_model("legacy_xception", pretrained=True)
Depthwise separable convolution
A regular conv computes each output channel from all input channels at once. The separable version splits it in two. The depthwise step filters each input channel with its own kernel . The pointwise step mixes the filtered channels with a conv :
- depthwise kernel of channel (a conv with )
- pointwise matrix, applied at every pixel
- cost ratio; for it approaches as grows
The separable conv is a regular conv whose kernel is constrained to : each output channel's filter on channel is a scaled copy of the same spatial kernel. Backprop follows the factorization: the pointwise gradient is , and the depthwise kernel of channel receives the back-mixed error , correlated with alone. The full treatment of the operation is in Depthwise separable convolution.
Xception differs from a literal "extreme Inception" module in two ways. Its order is depthwise first, then pointwise, the reverse of Inception's first; in a deep stack this hardly matters, because the layers alternate either way. And there is no nonlinearity between the depthwise and pointwise steps. The paper tested this directly: no activation in between converged fastest and reached the best accuracy, ELU was worse and ReLU worse still. Chollet's explanation is that the depthwise step works on 1-channel spaces, which are too shallow for a nonlinearity to help and lose information to it.
Entry, middle and exit flows
Xception at 299×299×3
- 1Entry flow. Two regular convs (/2 to 32, to 64), then three blocks of [two separable convs, /2 max-pool] going to 128, 256 and 728 channels. Each block's shortcut is a strided conv: .
- 2Middle flow. 8 identical blocks of [ReLU, separable conv 728] × 3 with an identity shortcut, all at . This is where most of the depth and 56% of the parameters sit.
- 3Exit flow. A block of separable convs to 728 and 1024 with max-pool and a projection shortcut (), then separable convs to 1536 and 2048 without shortcut, global average pooling, and a logistic-regression layer to 1000 classes.
- 4Everywhere. BatchNorm after every conv and separable conv; no biases; 36 conv layers, 34 of them separable.
The residual connections matter. The paper trained a version without them: it converged much more slowly and to a clearly worse ImageNet accuracy. Chollet adds that this does not make residual connections a requirement for stacks of separable convs: he also got excellent results with non-residual VGG-style models whose convs were all replaced by separable ones.
Worked example: one middle-flow layer
A middle-flow separable conv maps to . Weights and multiply-adds (weights × ), without BN:
| Variant | Depthwise 3×3 | Pointwise / full | Total weights | Mult-adds |
|---|---|---|---|---|
| regular 3×3 conv | – | 9 · 728 · 728 = 4,769,856 | 4,769,856 | 1,721.9M |
| 1×1 conv, 3×3 in 4 groups | 9 · 728 · 182 = 1,192,464 | 728 · 728 = 529,984 | 1,722,448 | 621.8M |
| depthwise separable | 9 · 728 = 6,552 | 728 · 728 = 529,984 | 536,536 | 193.7M |
Reading the table
- 1The cost ratio formula gives , the same as .
- 298.8% of the separable layer's weights are in the pointwise conv. The spatial filtering is nearly free; the channel mixing is what costs.
- 3With BN (2 × 728) the layer has 537,992 parameters, and a middle block of three such layers 1,613,976. Eight blocks: 12.9M of the network's 22.9M parameters.
- 4Because separable layers are so cheap, Xception can afford 728 channels at and still have fewer parameters than Inception-v3. It spends the saving on width, not on lower cost: at 8.4G it needs more multiply-adds than Inception-v3's 5.7G.
1import torch, torch.nn as nn23class SeparableConv2d(nn.Module):4 def __init__(self, cin, cout, k=3):5 super().__init__()6 self.depthwise = nn.Conv2d(cin, cin, k, padding=k // 2, groups=cin, bias=False) # one k x k filter per channel7 self.pointwise = nn.Conv2d(cin, cout, 1, bias=False) # 1 x 1 channel mixing8 def forward(self, x):9 return self.pointwise(self.depthwise(x)) # no nonlinearity in between (as in the paper)1011class MiddleBlock(nn.Module): # [ReLU, SepConv 728, BN] x 3, identity shortcut12 def __init__(self, c=728):13 super().__init__()14 self.f = nn.Sequential(*[m for _ in range(3) for m in15 (nn.ReLU(), SeparableConv2d(c, c), nn.BatchNorm2d(c))])16 def forward(self, x):17 return x + self.f(x)1819blk = MiddleBlock()20print(sum(p.numel() for p in blk.parameters())) # 161397621print(blk(torch.randn(1, 728, 19, 19)).shape) # torch.Size([1, 728, 19, 19])22print(sum(p.numel() for p in nn.Conv2d(728, 728, 3, bias=False).parameters())) # 47698562324# Pretrained Xception lives in timm (named "xception" in older timm releases)25# import timm; model = timm.create_model("legacy_xception", pretrained=True)Use when
- +Spatial and channel mixing are separated: about of the weights of a regular conv at the same width.
- +A simple, regular architecture (one block type per flow) instead of Inception's hand-tuned branches.
- +Slightly better ImageNet accuracy than Inception-v3 at fewer parameters, and a larger gain on JFT.
Avoid when
- −Depthwise convs are memory-bound and slower on GPUs than their FLOPs suggest.
- −The saving was spent on width: 8.4G multiply-adds, more than Inception-v3.
- −Not in torchvision; the separable constraint can hurt in very small networks.
Check yourself
A depthwise separable 3×3 conv maps 728 channels to 728. Roughly how many times fewer weights does it have than a regular 3×3 conv with the same channels?
Check yourself
What did the Xception paper find about putting a ReLU between the depthwise and the pointwise step?
Remember
- ✓Xception (Chollet 2017) = "extreme Inception": one spatial filter per channel, with all channel mixing done by 1×1 convs.
- ✓Depthwise separable conv: weights instead of , a ratio of .
- ✓36 conv layers (34 separable) in entry, middle (8 × 3 layers at 19×19×728) and exit flows, residual connections on all but the first and last modules, BN everywhere, no nonlinearity inside a separable conv.
- ✓22,855,952 parameters vs Inception-v3's 23,626,728; 79.0% vs 78.2% ImageNet top-1, a larger gain on JFT.
- ✓The saving is spent on width, so compute is not lower (≈8.4G mult-adds); depthwise convs are memory-bound on GPUs.