# Depthwise separable convolution

> Split a k×k convolution into a per-channel k×k filter (depthwise) and a 1×1 channel mixer (pointwise). Cost drops by 1/C_out + 1/k², about 8–9× for 3×3, but the depthwise half does so little work per byte that GPUs rarely see the full speed-up.

Source: https://www.mlfoundry.in/learn/depthwise-separable-convolution  
Section: Deep Learning › Convolutional Neural Networks  
Level: core · 11 min read · updated 2026-10-11

**Depthwise separable convolution:** replaces one $k\times k$ convolution from $C_{in}$ to $C_{out}$ channels by two cheaper ones: a **depthwise** $k\times k$ convolution that filters every input channel separately (one kernel per channel, `groups = C_in`), followed by a **pointwise** $1\times1$ convolution that mixes the filtered channels into $C_{out}$ outputs. Spatial filtering and channel mixing are done by different layers.

A standard convolution does two jobs at once. Each of its $C_{out}$ filters looks at a $k\times k$ neighbourhood **and** at all $C_{in}$ channels, so it has $k^2C_{in}$ weights and every output needs $k^2C_{in}$ multiply-adds. The depthwise separable version assumes the two jobs can be done one after the other. That assumption costs some expressive power and saves most of the arithmetic.

Laurent Sifre introduced depthwise separable convolutions for image classification in his PhD thesis (Sifre, 2014, with Mallat) on rigid-motion scattering networks. Chollet's **Xception** (2017) built a full ImageNet network from them and framed them as an extreme Inception module. Howard et al.'s **MobileNet** (2017) made them the standard building block for mobile networks. They are also inside **ShuffleNet**, EfficientNet and **ConvNeXt** (which uses a $7\times7$ depthwise conv), and Kaiser, Gomez and Chollet (2017) used them for machine translation.

**At a glance**

- **Depthwise:** $C_{in}$ kernels of size $k\times k$, one per channel; params $k^2C_{in}$
- **Pointwise:** a $1\times1$ conv $C_{in}\to C_{out}$; params $C_{in}C_{out}$
- **Cost ratio:** $\dfrac{k^2C_{in}+C_{in}C_{out}}{k^2C_{in}C_{out}}=\dfrac1{C_{out}}+\dfrac1{k^2}$, for params and for multiply-adds
- **Typical saving:** 8–9× for $3\times3$ with hundreds of channels; 24× for $5\times5$ at 512 channels
- **In PyTorch:** `nn.Conv2d(C, C, k, padding=k//2, groups=C)` then `nn.Conv2d(C, C_out, 1)`
- **Not the same as:** spatially separable convs ($k\times1$ then $1\times k$), which factor the kernel in space, not across channels

## The two convolutions

$$
\underbrace{d_c(y,x)=\sum_{i,j}K_c(i,j)\,X_c(y+i,\,x+j)}_{\text{depthwise: one kernel per channel}}\qquad\qquad \underbrace{Y_o(y,x)=\sum_{c=1}^{C_{in}}P_{o,c}\;d_c(y,x)}_{\text{pointwise: }1\times1\text{ mixing}}
$$

- $X_c$: input channel $c$, $c=1,\dots,C_{in}$
- $K_c$: the $k\times k$ depthwise kernel of channel $c$; it never sees any other channel
- $d_c$: the filtered map of channel $c$; there are exactly $C_{in}$ of them
- $P_{o,c}$: the pointwise weight from filtered channel $c$ to output $o$, a $C_{out}\times C_{in}$ matrix applied at every pixel

Substitute the first formula into the second and you get an ordinary convolution with kernel $W_{o,c}(i,j)=P_{o,c}\,K_c(i,j)$. So a depthwise separable layer **is** a standard convolution whose kernel is constrained: for a fixed input channel $c$, all $C_{out}$ filters use the same spatial pattern $K_c$, scaled by $P_{o,c}$. A standard layer has $C_{in}C_{out}$ independent $k\times k$ patterns; the separable one has $C_{in}$ patterns and a mixing matrix. In practice BatchNorm and an activation usually sit between the two halves (MobileNet does this), which makes the pair slightly more expressive than one rank-constrained conv.

With a **channel multiplier** $m$ the depthwise step produces $m$ maps per input channel (`out_channels = m·C_in`, `groups = C_in`). $m=1$ is by far the most common choice.

## Gradients

Backprop runs through the two halves in reverse. Writing $\delta_o=\partial L/\partial Y_o$ for the incoming error map and $D_c=\partial L/\partial d_c$ for the error at the filtered maps (ignoring any BN or activation between them):

$$
\frac{\partial L}{\partial P_{o,c}}=\sum_{y,x}\delta_o(y,x)\,d_c(y,x),\qquad D_c=\sum_{o}P_{o,c}\,\delta_o,\qquad \frac{\partial L}{\partial K_c(i,j)}=\sum_{y,x}D_c(y,x)\,X_c(y+i,x+j)
$$

- $\partial L/\partial P$: a $1\times1$ conv's gradient: correlate each output error map with each filtered map
- $D_c$: error sent back to channel $c$: the transpose of the pointwise mixing
- $\partial L/\partial K_c$: depends only on channel $c$'s input and error; no cross-channel terms

The input gradient $\partial L/\partial X_c$ is the full convolution of $D_c$ with the flipped $K_c$, again channel by channel. Every rule is the ordinary conv rule restricted to one group, so autograd needs nothing special.

> Figure: Top: a 3-channel 8×8 image (vertical bar, horizontal bar, diagonal ramp) is filtered by three depthwise 3×3 kernels, one per channel (vertical edge, horizontal edge, box blur), giving three 6×6 maps; a 1×1 pointwise conv mixes them into two outputs. Click any output pixel to see the arithmetic. At the default pixel (2, 1) of output 2 the depthwise values are 12, −12 and 0.44, and y = 0.5·12 − 0.5·(−12) + 1·0.44 = 12.44. A standard 3×3 conv with kernel W[o,c] = P[o,c]·K_c gives the identical number at every pixel. This tiny layer uses 33 weights instead of 54. Bottom: a cost calculator. At the default 3×3, 512 → 512 channels on 14×14, the separable pair needs 52.3M instead of 462.4M multiply-adds (8.84× fewer, = 1/512 + 1/9). An idealized FP32 roofline on an A100 still gives about 8.2× less time. Switch to TF32 and the speed-up drops to 3.8×. The depthwise half runs at 2.2 FLOP/byte, far below the GPU's ridge point, so it is memory-bound. At 32 → 64 channels on 112×112, TF32 makes the separable pair slower than the dense conv (0.61×).

## How much it saves

$$
\frac{\text{separable}}{\text{standard}}=\frac{k^2C_{in}HW+C_{in}C_{out}HW}{k^2C_{in}C_{out}HW}=\frac1{C_{out}}+\frac1{k^2}
$$

- $H W$: output resolution; it cancels, so the ratio is the same for parameters and for multiply-adds (also at stride 2 if the depthwise conv takes the stride)
- $1/C_{out}$: the price of the depthwise filters; negligible once $C_{out}$ is in the hundreds
- $1/k^2$: the price of the pointwise mixing; this sets the ceiling, $k^2$ = 9× for $3\times3$

The saving is bounded by $k^2$ no matter how wide the layer is. Larger kernels save more: at 512 channels a $5\times5$ separable layer costs $1/512+1/25$ of the standard one, 23.8× less. That is why separable designs can afford big kernels. MobileNetV3 uses $5\times5$ and ConvNeXt uses $7\times7$ depthwise filters. Almost all the remaining cost is in the $1\times1$ convolution.

## Worked example: MobileNet's first separable layer

The first block of MobileNetV1 maps $32\to64$ channels at $112\times112$ with a $3\times3$ kernel. Numbers from the figure's library (checked in Node):

3×3 conv, 32 → 64 channels, 112×112 output, stride 1, no bias

| layer | parameters | multiply-adds | FLOP/byte (fp32, unfused) |
| --- | --- | --- | --- |
| standard 3×3 | $9\cdot32\cdot64=18{,}432$ | 231,211,008 | 94.6 |
| depthwise 3×3 | $9\cdot32=288$ | 3,612,672 | 2.25 |
| pointwise 1×1 | $32\cdot64=2{,}048$ | 25,690,112 | 10.6 |
| depthwise + pointwise | 2,336 | 29,302,784 | — |

**Reading the table**

1. Ratio $29{,}302{,}784/231{,}211{,}008=0.1267=1/64+1/9$: 7.9× fewer multiply-adds and 7.9× fewer parameters.
2. The pointwise conv holds 87.7% of the separable layer's multiply-adds ($25.69\text{M}/29.30\text{M}$). Deeper in the network, at 512 channels, the depthwise share falls to 1.7%.
3. Arithmetic intensity: the depthwise conv does $k^2$ multiply-adds per output and reads and writes about one value per output, so in fp32 it gets $\approx 2k^2/8=k^2/4=2.25$ FLOP per byte whatever the width. The dense conv reuses each input value $k^2C_{out}$ times and reaches 94.6.
4. An A100-40GB needs about 12.5 FLOP/byte (FP32, 19.5 TFLOP/s over 1.555 TB/s) or 100 FLOP/byte (TF32 tensor cores) to be compute-bound. The depthwise conv is far below both, so its run time is set by memory traffic, not by its tiny FLOP count.

> **FLOPs are not latency:** The $1/C_{out}+1/k^2$ saving is real for multiply-adds and parameters, but depthwise kernels are memory-bound and launch an extra pass over the activation tensor. On GPUs, especially with tensor cores, a separable layer is often only 2–4× faster than the dense one, and for thin layers on large maps it can be slower (the figure shows 0.61× for 32 → 64 channels at 112×112 under TF32). Mobile CPUs and DSPs, where compute is the bottleneck, see much more of the theoretical gain. **ShuffleNet** v2 turned this observation into design guidelines. Always benchmark on the target device.

```python
import torch, torch.nn as nn, torch.nn.functional as F

class SeparableConv2d(nn.Module):
    """Depthwise k x k (groups = C_in) -> BN -> ReLU -> pointwise 1 x 1 -> BN -> ReLU."""
    def __init__(self, cin, cout, k=3, stride=1):
        super().__init__()
        self.dw = nn.Conv2d(cin, cin, k, stride, padding=k // 2, groups=cin, bias=False)
        self.bn1 = nn.BatchNorm2d(cin)
        self.pw = nn.Conv2d(cin, cout, 1, bias=False)
        self.bn2 = nn.BatchNorm2d(cout)

    def forward(self, x):
        x = F.relu(self.bn1(self.dw(x)))
        return F.relu(self.bn2(self.pw(x)))

count = lambda m: sum(p.numel() for p in m.parameters())
std = nn.Conv2d(32, 64, 3, padding=1, bias=False)
sep = SeparableConv2d(32, 64)
print(count(std))                        # 18432
print(sep.dw.weight.shape)               # torch.Size([32, 1, 3, 3]): one 3x3 kernel per channel
print(count(sep.dw) + count(sep.pw))     # 288 + 2048 = 2336 (BN adds 2*(32+64) more)

# Without BN/ReLU in between, dw -> pw equals ONE conv with kernel W[o, c] = P[o, c] * K_c
dw = nn.Conv2d(32, 32, 3, padding=1, groups=32, bias=False)
pw = nn.Conv2d(32, 64, 1, bias=False)
W = pw.weight[:, :, 0, 0, None, None] * dw.weight[:, 0][None]   # (64, 32, 3, 3)
x = torch.randn(2, 32, 28, 28)
with torch.no_grad():
    print(torch.allclose(F.conv2d(x, W, padding=1), pw(dw(x)), atol=1e-5))   # True
```

A depthwise separable block, its parameter count, and the rank-1 equivalence

> **Why the factorization is a good bet:** Trained standard conv layers are highly redundant across channels: many filters share similar spatial patterns. The separable form keeps $C_{in}$ spatial patterns and lets a dense $1\times1$ layer recombine them, so most of the useful capacity survives. MobileNetV1 reports that its separable network loses only 1.1 points of ImageNet top-1 against the same network with full convolutions (70.6% vs 71.7%) while using about 8.5× fewer multiply-adds (569M vs 4866M) and 7× fewer parameters.

> **Order and nonlinearity:** MobileNet uses depthwise then pointwise, with BN + ReLU after each. Xception's version also goes depthwise first, but it puts no nonlinearity between the two halves, and Chollet found that adding one hurt. Inverted residual blocks (MobileNetV2) wrap the depthwise conv between two $1\times1$ convs. Grouped convolution with $1<g<C_{in}$ groups (**ResNeXt**) sits between the dense case ($g=1$) and the depthwise case ($g=C_{in}$).

**Strengths**

- Parameters and multiply-adds fall by $1/C_{out}+1/k^2$, about 8–9× for $3\times3$
- Large kernels become affordable (5×5, 7×7 cost little more than 3×3)
- A big win on compute-bound hardware: mobile CPUs, DSPs, microcontrollers
- Drop-in: just a grouped conv plus a 1×1 conv in any framework

**Limitations**

- Low arithmetic intensity ($\approx k^2/4$ FLOP/byte in fp32): memory-bound on GPUs and accelerators
- Two kernels and two passes over activations unless fused
- Less expressive than a full conv (rank-constrained kernel); slightly lower accuracy at equal depth and width
- Depthwise weights quantize poorly (few weights per channel, wide ranges between channels)

**Question:** A 3×3 conv maps 256 → 256 channels. Roughly how many times fewer multiply-adds does its depthwise separable replacement need?

- A. About 2×
- B. About 8.7×
- C. About 256×
- D. About 2304×

**Answer:** B. About 8.7× — The ratio is $1/C_{out}+1/k^2=1/256+1/9=0.1150$, so about $1/0.1150=8.7\times$ fewer. The bound is $k^2=9$: making the layer wider only moves the ratio closer to 1/9.

**Question:** Why does a depthwise convolution often run far below a GPU's peak throughput?

- A. Its kernels cannot be parallelized across channels
- B. It does only k² multiply-adds per output value it reads and writes, so it is limited by memory bandwidth
- C. It needs more parameters than a dense conv
- D. PyTorch implements it with a Python loop over channels

**Answer:** B. It does only k² multiply-adds per output value it reads and writes, so it is limited by memory bandwidth — Each output reads a $k\times k$ window of a single channel and nothing else, so each loaded value is reused only about $k^2$ times. That is $\approx 2.25$ FLOP/byte in fp32 for $3\times3$, well below the 12.5 (FP32) or 100 (TF32) FLOP/byte an A100 needs to be compute-bound.

**Key points**

- Depthwise: one $k\times k$ kernel per channel (`groups=C_in`). Pointwise: a $1\times1$ conv that mixes channels.
- The pair equals a standard conv with rank-constrained kernel $W_{o,c}=P_{o,c}K_c$, and costs $1/C_{out}+1/k^2$ as much.
- About 8–9× fewer multiply-adds for $3\times3$; almost all remaining work is in the $1\times1$ conv.
- Depthwise convs are memory-bound ($\approx k^2/4$ FLOP/byte in fp32), so GPU speed-ups are smaller than FLOP counts suggest.
- It is the building block of Xception, MobileNet, ShuffleNet, EfficientNet and ConvNeXt.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
