Definition
Depthwise separable convolution replaces one convolution from to channels by two cheaper ones: a depthwise convolution that filters every input channel separately (one kernel per channel, groups = C_in), followed by a pointwise convolution that mixes the filtered channels into outputs. Spatial filtering and channel mixing are done by different layers.
A standard convolution does two jobs at once. Each of its filters looks at a neighbourhood and at all channels, so it has weights and every output needs multiply-adds. The depthwise separable version assumes the two jobs can be done one after the other. That assumption costs some expressive power and saves most of the arithmetic.
Laurent Sifre introduced depthwise separable convolutions for image classification in his PhD thesis (Sifre, 2014, with Mallat) on rigid-motion scattering networks. Chollet's Xception (2017) built a full ImageNet network from them and framed them as an extreme Inception module. Howard et al.'s MobileNet (2017) made them the standard building block for mobile networks. They are also inside ShuffleNet, EfficientNet and ConvNeXt (which uses a depthwise conv), and Kaiser, Gomez and Chollet (2017) used them for machine translation.
At a glance
- Depthwise
- kernels of size , one per channel; params
- Pointwise
- a conv ; params
- Cost ratio
- , for params and for multiply-adds
- Typical saving
- 8–9× for with hundreds of channels; 24× for at 512 channels
- In PyTorch
nn.Conv2d(C, C, k, padding=k//2, groups=C)thennn.Conv2d(C, C_out, 1)- Not the same as
- spatially separable convs ( then ), which factor the kernel in space, not across channels
The two convolutions
- input channel ,
- the depthwise kernel of channel ; it never sees any other channel
- the filtered map of channel ; there are exactly of them
- the pointwise weight from filtered channel to output , a matrix applied at every pixel
Substitute the first formula into the second and you get an ordinary convolution with kernel . So a depthwise separable layer is a standard convolution whose kernel is constrained: for a fixed input channel , all filters use the same spatial pattern , scaled by . A standard layer has independent patterns; the separable one has patterns and a mixing matrix. In practice BatchNorm and an activation usually sit between the two halves (MobileNet does this), which makes the pair slightly more expressive than one rank-constrained conv.
With a channel multiplier the depthwise step produces maps per input channel (out_channels = m·C_in, groups = C_in). is by far the most common choice.
Gradients
Backprop runs through the two halves in reverse. Writing for the incoming error map and for the error at the filtered maps (ignoring any BN or activation between them):
- a conv's gradient: correlate each output error map with each filtered map
- error sent back to channel : the transpose of the pointwise mixing
- depends only on channel 's input and error; no cross-channel terms
The input gradient is the full convolution of with the flipped , again channel by channel. Every rule is the ordinary conv rule restricted to one group, so autograd needs nothing special.
How much it saves
- output resolution; it cancels, so the ratio is the same for parameters and for multiply-adds (also at stride 2 if the depthwise conv takes the stride)
- the price of the depthwise filters; negligible once is in the hundreds
- the price of the pointwise mixing; this sets the ceiling, = 9× for
The saving is bounded by no matter how wide the layer is. Larger kernels save more: at 512 channels a separable layer costs of the standard one, 23.8× less. That is why separable designs can afford big kernels. MobileNetV3 uses and ConvNeXt uses depthwise filters. Almost all the remaining cost is in the convolution.
Worked example: MobileNet's first separable layer
The first block of MobileNetV1 maps channels at with a kernel. Numbers from the figure's library (checked in Node):
| layer | parameters | multiply-adds | FLOP/byte (fp32, unfused) |
|---|---|---|---|
| standard 3×3 | 231,211,008 | 94.6 | |
| depthwise 3×3 | 3,612,672 | 2.25 | |
| pointwise 1×1 | 25,690,112 | 10.6 | |
| depthwise + pointwise | 2,336 | 29,302,784 | — |
Reading the table
- 1Ratio : 7.9× fewer multiply-adds and 7.9× fewer parameters.
- 2The pointwise conv holds 87.7% of the separable layer's multiply-adds (). Deeper in the network, at 512 channels, the depthwise share falls to 1.7%.
- 3Arithmetic intensity: the depthwise conv does multiply-adds per output and reads and writes about one value per output, so in fp32 it gets FLOP per byte whatever the width. The dense conv reuses each input value times and reaches 94.6.
- 4An A100-40GB needs about 12.5 FLOP/byte (FP32, 19.5 TFLOP/s over 1.555 TB/s) or 100 FLOP/byte (TF32 tensor cores) to be compute-bound. The depthwise conv is far below both, so its run time is set by memory traffic, not by its tiny FLOP count.
1import torch, torch.nn as nn, torch.nn.functional as F23class SeparableConv2d(nn.Module):4 """Depthwise k x k (groups = C_in) -> BN -> ReLU -> pointwise 1 x 1 -> BN -> ReLU."""5 def __init__(self, cin, cout, k=3, stride=1):6 super().__init__()7 self.dw = nn.Conv2d(cin, cin, k, stride, padding=k // 2, groups=cin, bias=False)8 self.bn1 = nn.BatchNorm2d(cin)9 self.pw = nn.Conv2d(cin, cout, 1, bias=False)10 self.bn2 = nn.BatchNorm2d(cout)1112 def forward(self, x):13 x = F.relu(self.bn1(self.dw(x)))14 return F.relu(self.bn2(self.pw(x)))1516count = lambda m: sum(p.numel() for p in m.parameters())17std = nn.Conv2d(32, 64, 3, padding=1, bias=False)18sep = SeparableConv2d(32, 64)19print(count(std)) # 1843220print(sep.dw.weight.shape) # torch.Size([32, 1, 3, 3]): one 3x3 kernel per channel21print(count(sep.dw) + count(sep.pw)) # 288 + 2048 = 2336 (BN adds 2*(32+64) more)2223# Without BN/ReLU in between, dw -> pw equals ONE conv with kernel W[o, c] = P[o, c] * K_c24dw = nn.Conv2d(32, 32, 3, padding=1, groups=32, bias=False)25pw = nn.Conv2d(32, 64, 1, bias=False)26W = pw.weight[:, :, 0, 0, None, None] * dw.weight[:, 0][None] # (64, 32, 3, 3)27x = torch.randn(2, 32, 28, 28)28with torch.no_grad():29 print(torch.allclose(F.conv2d(x, W, padding=1), pw(dw(x)), atol=1e-5)) # TrueUse when
- +Parameters and multiply-adds fall by , about 8–9× for
- +Large kernels become affordable (5×5, 7×7 cost little more than 3×3)
- +A big win on compute-bound hardware: mobile CPUs, DSPs, microcontrollers
- +Drop-in: just a grouped conv plus a 1×1 conv in any framework
Avoid when
- −Low arithmetic intensity ( FLOP/byte in fp32): memory-bound on GPUs and accelerators
- −Two kernels and two passes over activations unless fused
- −Less expressive than a full conv (rank-constrained kernel); slightly lower accuracy at equal depth and width
- −Depthwise weights quantize poorly (few weights per channel, wide ranges between channels)
Check yourself
A 3×3 conv maps 256 → 256 channels. Roughly how many times fewer multiply-adds does its depthwise separable replacement need?
Check yourself
Why does a depthwise convolution often run far below a GPU's peak throughput?
Remember
- ✓Depthwise: one kernel per channel (
groups=C_in). Pointwise: a conv that mixes channels. - ✓The pair equals a standard conv with rank-constrained kernel , and costs as much.
- ✓About 8–9× fewer multiply-adds for ; almost all remaining work is in the conv.
- ✓Depthwise convs are memory-bound ( FLOP/byte in fp32), so GPU speed-ups are smaller than FLOP counts suggest.
- ✓It is the building block of Xception, MobileNet, ShuffleNet, EfficientNet and ConvNeXt.