Depthwise separable convolution

Split a k×k convolution into a per-channel k×k filter (depthwise) and a 1×1 channel mixer (pointwise). Cost drops by 1/C_out + 1/k², about 8–9× for 3×3, but the depthwise half does so little work per byte that GPUs rarely see the full speed-up.

11 min read894 wordscore0 viewsupdated 2026-10-11

Definition

Depthwise separable convolution replaces one k×kk\times k convolution from CinC_{in} to CoutC_{out} channels by two cheaper ones: a depthwise k×kk\times k convolution that filters every input channel separately (one kernel per channel, groups = C_in), followed by a pointwise 1×11\times1 convolution that mixes the filtered channels into CoutC_{out} outputs. Spatial filtering and channel mixing are done by different layers.

A standard convolution does two jobs at once. Each of its CoutC_{out} filters looks at a k×kk\times k neighbourhood and at all CinC_{in} channels, so it has k2Cink^2C_{in} weights and every output needs k2Cink^2C_{in} multiply-adds. The depthwise separable version assumes the two jobs can be done one after the other. That assumption costs some expressive power and saves most of the arithmetic.

Laurent Sifre introduced depthwise separable convolutions for image classification in his PhD thesis (Sifre, 2014, with Mallat) on rigid-motion scattering networks. Chollet's Xception (2017) built a full ImageNet network from them and framed them as an extreme Inception module. Howard et al.'s MobileNet (2017) made them the standard building block for mobile networks. They are also inside ShuffleNet, EfficientNet and ConvNeXt (which uses a 7×77\times7 depthwise conv), and Kaiser, Gomez and Chollet (2017) used them for machine translation.

At a glance

Depthwise
CinC_{in} kernels of size k×kk\times k, one per channel; params k2Cink^2C_{in}
Pointwise
a 1×11\times1 conv Cin→CoutC_{in}\to C_{out}; params CinCoutC_{in}C_{out}
Cost ratio
k2Cin+CinCoutk2CinCout=1Cout+1k2\dfrac{k^2C_{in}+C_{in}C_{out}}{k^2C_{in}C_{out}}=\dfrac1{C_{out}}+\dfrac1{k^2}, for params and for multiply-adds
Typical saving
8–9× for 3×33\times3 with hundreds of channels; 24× for 5×55\times5 at 512 channels
In PyTorch
nn.Conv2d(C, C, k, padding=k//2, groups=C) then nn.Conv2d(C, C_out, 1)
Not the same as
spatially separable convs (k×1k\times1 then 1×k1\times k), which factor the kernel in space, not across channels

The two convolutions

dc(y,x)=∑i,jKc(i,j) Xc(y+i, x+j)⏟depthwise: one kernel per channelYo(y,x)=∑c=1CinPo,c  dc(y,x)⏟pointwise: 1×1 mixing\underbrace{d_c(y,x)=\sum_{i,j}K_c(i,j)\,X_c(y+i,\,x+j)}_{\text{depthwise: one kernel per channel}}\qquad\qquad \underbrace{Y_o(y,x)=\sum_{c=1}^{C_{in}}P_{o,c}\;d_c(y,x)}_{\text{pointwise: }1\times1\text{ mixing}}
XcX_c
input channel cc, c=1,…,Cinc=1,\dots,C_{in}
KcK_c
the k×kk\times k depthwise kernel of channel cc; it never sees any other channel
dcd_c
the filtered map of channel cc; there are exactly CinC_{in} of them
Po,cP_{o,c}
the pointwise weight from filtered channel cc to output oo, a Cout×CinC_{out}\times C_{in} matrix applied at every pixel

Substitute the first formula into the second and you get an ordinary convolution with kernel Wo,c(i,j)=Po,c Kc(i,j)W_{o,c}(i,j)=P_{o,c}\,K_c(i,j). So a depthwise separable layer is a standard convolution whose kernel is constrained: for a fixed input channel cc, all CoutC_{out} filters use the same spatial pattern KcK_c, scaled by Po,cP_{o,c}. A standard layer has CinCoutC_{in}C_{out} independent k×kk\times k patterns; the separable one has CinC_{in} patterns and a mixing matrix. In practice BatchNorm and an activation usually sit between the two halves (MobileNet does this), which makes the pair slightly more expressive than one rank-constrained conv.

With a channel multiplier mm the depthwise step produces mm maps per input channel (out_channels = m·C_in, groups = C_in). m=1m=1 is by far the most common choice.

Gradients

Backprop runs through the two halves in reverse. Writing δo=∂L/∂Yo\delta_o=\partial L/\partial Y_o for the incoming error map and Dc=∂L/∂dcD_c=\partial L/\partial d_c for the error at the filtered maps (ignoring any BN or activation between them):

∂L∂Po,c=∑y,xδo(y,x) dc(y,x),Dc=∑oPo,c δo,∂L∂Kc(i,j)=∑y,xDc(y,x) Xc(y+i,x+j)\frac{\partial L}{\partial P_{o,c}}=\sum_{y,x}\delta_o(y,x)\,d_c(y,x),\qquad D_c=\sum_{o}P_{o,c}\,\delta_o,\qquad \frac{\partial L}{\partial K_c(i,j)}=\sum_{y,x}D_c(y,x)\,X_c(y+i,x+j)
∂L/∂P\partial L/\partial P
a 1×11\times1 conv's gradient: correlate each output error map with each filtered map
DcD_c
error sent back to channel cc: the transpose of the pointwise mixing
∂L/∂Kc\partial L/\partial K_c
depends only on channel cc's input and error; no cross-channel terms

The input gradient ∂L/∂Xc\partial L/\partial X_c is the full convolution of DcD_c with the flipped KcK_c, again channel by channel. Every rule is the ordinary conv rule restricted to one group, so autograd needs nothing special.

input 3×8×8depthwise3×6×6pointwise 1×1 → 2×6×6K1edge x12.00K2edge y−12.00K3blur0.44out 1out 2click an output pixelpixel (2, 1) of out 2d1 = K1 ⋆ window = 12.00d2 = K2 ⋆ window = −12.00d3 = K3 ⋆ window = 0.44y = 0.5·d1 − 0.5·d2 + 1.0·d3 = 12.44standard 3×3 conv withW[o,c] = P[o,c]·K_c gives12.44 (identical)params here: dw 27 + pw 6 = 33standard: 543×3 conv, 512 → 512 channels, 14×14, stride 1, FP32 roofline (A100-40GB)MACs ratio = 0.1131 = 1/512 + 1/9 = 0.1131 → 8.84× fewerstandard2.36M paramsMACs462.42Mtime47.43 µs · 90.3 F/Bcomputedepthwise4.6k paramsMACs903.2ktime0.53 µs · 2.2 F/Bmempointwise262.1k paramsMACs51.38Mtime5.27 µs · 55.5 F/Bcomputedw + pw266.8k paramsMACs52.28Mtime5.80 µs · 39.1 F/Bcomputeideal speed-up 8.18× vs 8.84× fewer MACs · ridge 12.5 FLOP/B · depthwise = 1.7% of MACs
kernelpeak
fig.Top: a 3-channel 8×8 image (vertical bar, horizontal bar, diagonal ramp) is filtered by three depthwise 3×3 kernels, one per channel (vertical edge, horizontal edge, box blur), giving three 6×6 maps; a 1×1 pointwise conv mixes them into two outputs. Click any output pixel to see the arithmetic. At the default pixel (2, 1) of output 2 the depthwise values are 12, −12 and 0.44, and y = 0.5·12 − 0.5·(−12) + 1·0.44 = 12.44. A standard 3×3 conv with kernel W[o,c] = P[o,c]·K_c gives the identical number at every pixel. This tiny layer uses 33 weights instead of 54. Bottom: a cost calculator. At the default 3×3, 512 → 512 channels on 14×14, the separable pair needs 52.3M instead of 462.4M multiply-adds (8.84× fewer, = 1/512 + 1/9). An idealized FP32 roofline on an A100 still gives about 8.2× less time. Switch to TF32 and the speed-up drops to 3.8×. The depthwise half runs at 2.2 FLOP/byte, far below the GPU's ridge point, so it is memory-bound. At 32 → 64 channels on 112×112, TF32 makes the separable pair slower than the dense conv (0.61×).

How much it saves

separablestandard=k2CinHW+CinCoutHWk2CinCoutHW=1Cout+1k2\frac{\text{separable}}{\text{standard}}=\frac{k^2C_{in}HW+C_{in}C_{out}HW}{k^2C_{in}C_{out}HW}=\frac1{C_{out}}+\frac1{k^2}
HWH W
output resolution; it cancels, so the ratio is the same for parameters and for multiply-adds (also at stride 2 if the depthwise conv takes the stride)
1/Cout1/C_{out}
the price of the depthwise filters; negligible once CoutC_{out} is in the hundreds
1/k21/k^2
the price of the pointwise mixing; this sets the ceiling, k2k^2 = 9× for 3×33\times3

The saving is bounded by k2k^2 no matter how wide the layer is. Larger kernels save more: at 512 channels a 5×55\times5 separable layer costs 1/512+1/251/512+1/25 of the standard one, 23.8× less. That is why separable designs can afford big kernels. MobileNetV3 uses 5×55\times5 and ConvNeXt uses 7×77\times7 depthwise filters. Almost all the remaining cost is in the 1×11\times1 convolution.

Worked example: MobileNet's first separable layer

The first block of MobileNetV1 maps 32→6432\to64 channels at 112×112112\times112 with a 3×33\times3 kernel. Numbers from the figure's library (checked in Node):

layerparametersmultiply-addsFLOP/byte (fp32, unfused)
standard 3×39⋅32⋅64=18,4329\cdot32\cdot64=18{,}432231,211,00894.6
depthwise 3×39⋅32=2889\cdot32=2883,612,6722.25
pointwise 1×132⋅64=2,04832\cdot64=2{,}04825,690,11210.6
depthwise + pointwise2,33629,302,784—
3×3 conv, 32 → 64 channels, 112×112 output, stride 1, no bias

Reading the table

  1. 1Ratio 29,302,784/231,211,008=0.1267=1/64+1/929{,}302{,}784/231{,}211{,}008=0.1267=1/64+1/9: 7.9× fewer multiply-adds and 7.9× fewer parameters.
  2. 2The pointwise conv holds 87.7% of the separable layer's multiply-adds (25.69M/29.30M25.69\text{M}/29.30\text{M}). Deeper in the network, at 512 channels, the depthwise share falls to 1.7%.
  3. 3Arithmetic intensity: the depthwise conv does k2k^2 multiply-adds per output and reads and writes about one value per output, so in fp32 it gets ≈2k2/8=k2/4=2.25\approx 2k^2/8=k^2/4=2.25 FLOP per byte whatever the width. The dense conv reuses each input value k2Coutk^2C_{out} times and reaches 94.6.
  4. 4An A100-40GB needs about 12.5 FLOP/byte (FP32, 19.5 TFLOP/s over 1.555 TB/s) or 100 FLOP/byte (TF32 tensor cores) to be compute-bound. The depthwise conv is far below both, so its run time is set by memory traffic, not by its tiny FLOP count.
python
1import torch, torch.nn as nn, torch.nn.functional as F
2
3class SeparableConv2d(nn.Module):
4 """Depthwise k x k (groups = C_in) -> BN -> ReLU -> pointwise 1 x 1 -> BN -> ReLU."""
5 def __init__(self, cin, cout, k=3, stride=1):
6 super().__init__()
7 self.dw = nn.Conv2d(cin, cin, k, stride, padding=k // 2, groups=cin, bias=False)
8 self.bn1 = nn.BatchNorm2d(cin)
9 self.pw = nn.Conv2d(cin, cout, 1, bias=False)
10 self.bn2 = nn.BatchNorm2d(cout)
11
12 def forward(self, x):
13 x = F.relu(self.bn1(self.dw(x)))
14 return F.relu(self.bn2(self.pw(x)))
15
16count = lambda m: sum(p.numel() for p in m.parameters())
17std = nn.Conv2d(32, 64, 3, padding=1, bias=False)
18sep = SeparableConv2d(32, 64)
19print(count(std)) # 18432
20print(sep.dw.weight.shape) # torch.Size([32, 1, 3, 3]): one 3x3 kernel per channel
21print(count(sep.dw) + count(sep.pw)) # 288 + 2048 = 2336 (BN adds 2*(32+64) more)
22
23# Without BN/ReLU in between, dw -> pw equals ONE conv with kernel W[o, c] = P[o, c] * K_c
24dw = nn.Conv2d(32, 32, 3, padding=1, groups=32, bias=False)
25pw = nn.Conv2d(32, 64, 1, bias=False)
26W = pw.weight[:, :, 0, 0, None, None] * dw.weight[:, 0][None] # (64, 32, 3, 3)
27x = torch.randn(2, 32, 28, 28)
28with torch.no_grad():
29 print(torch.allclose(F.conv2d(x, W, padding=1), pw(dw(x)), atol=1e-5)) # True
A depthwise separable block, its parameter count, and the rank-1 equivalence

Use when

  • +Parameters and multiply-adds fall by 1/Cout+1/k21/C_{out}+1/k^2, about 8–9× for 3×33\times3
  • +Large kernels become affordable (5×5, 7×7 cost little more than 3×3)
  • +A big win on compute-bound hardware: mobile CPUs, DSPs, microcontrollers
  • +Drop-in: just a grouped conv plus a 1×1 conv in any framework

Avoid when

  • −Low arithmetic intensity (≈k2/4\approx k^2/4 FLOP/byte in fp32): memory-bound on GPUs and accelerators
  • −Two kernels and two passes over activations unless fused
  • −Less expressive than a full conv (rank-constrained kernel); slightly lower accuracy at equal depth and width
  • −Depthwise weights quantize poorly (few weights per channel, wide ranges between channels)

Check yourself

A 3×3 conv maps 256 → 256 channels. Roughly how many times fewer multiply-adds does its depthwise separable replacement need?

Check yourself

Why does a depthwise convolution often run far below a GPU's peak throughput?

Remember

  • ✓Depthwise: one k×kk\times k kernel per channel (groups=C_in). Pointwise: a 1×11\times1 conv that mixes channels.
  • ✓The pair equals a standard conv with rank-constrained kernel Wo,c=Po,cKcW_{o,c}=P_{o,c}K_c, and costs 1/Cout+1/k21/C_{out}+1/k^2 as much.
  • ✓About 8–9× fewer multiply-adds for 3×33\times3; almost all remaining work is in the 1×11\times1 conv.
  • ✓Depthwise convs are memory-bound (≈k2/4\approx k^2/4 FLOP/byte in fp32), so GPU speed-ups are smaller than FLOP counts suggest.
  • ✓It is the building block of Xception, MobileNet, ShuffleNet, EfficientNet and ConvNeXt.
← back to Deep Learning roadmap