ShuffleNet

V1 makes the 1×1 convs cheap with group convolutions and a channel shuffle so groups still exchange information. V2 shows FLOPs mislead and gives four speed guidelines: equal widths, few groups, little fragmentation, few element-wise ops.

12 min read892 wordsadvanced0 viewsupdated 2026-10-11

Definition

ShuffleNet is a family of mobile CNNs from Megvii. ShuffleNet v1 (Zhang, Zhou, Lin and Sun, CVPR 2018) uses pointwise group convolutions for the 1×11\times1 layers and a channel shuffle that permutes channels between groups. ShuffleNet v2 (Ma, Zhang, Zheng and Sun, ECCV 2018) drops the group convolutions. It is built from channel split units designed with four guidelines for real-device speed.

In a MobileNet-style network the depthwise 3×33\times3 convs are cheap and the dense 1×11\times1 convs take over 90% of the multiply-adds. ShuffleNet v1 attacks that remaining cost by splitting the 1×11\times1 convs into gg groups, as ResNeXt does for its 3×33\times3 convs. That divides their cost by gg, and the saving can be spent on wider layers. The catch is that stacked group convolutions never mix information across groups. The channel shuffle fixes that for free.

At a glance

v1 unit
1×11\times1 GConv → BN, ReLU → shuffle → dw 3×33\times3 → BN → 1×11\times1 GConv → BN, add, ReLU; stride 2: avg-pool shortcut and concat
v2 unit
split channels in half; one half is identity, the other goes 1×11\times1 → dw 3×33\times3 → 1×11\times1 (equal widths, no groups); concat, then shuffle
v1 result
1×, g=3g=3: 137M multiply-adds, 32.6% top-1 error; 7.8 points lower error than MobileNet at about 40M multiply-adds; about 13× faster than AlexNet on an ARM phone
v2 result
1×: 146M multiply-adds; torchvision weights 2,278,604 params, 69.36% top-1
Note
both papers say "FLOPs" for multiply-adds
In PyTorch
nn.ChannelShuffle(g), torchvision.models.shufflenet_v2_x1_0 (v1 is not in torchvision)

Channel shuffle

With gg groups, a group convolution's output group GG depends only on input group GG. Stack two of them and the network splits into gg independent narrow towers. The shuffle sends one channel from every group into every group before the next layer. With C=gnC=gn channels it reshapes the channel axis to (g,n)(g,n), transposes to (n,g)(n,g) and flattens:

out[p]=in[ (p mod g)⋅n+⌊p/g⌋ ],p=0,…,C−1,∂L∂ in=π−1 ⁣(∂L∂ out)\text{out}[p]=\text{in}\big[\,(p\bmod g)\cdot n+\lfloor p/g\rfloor\,\big],\qquad p=0,\dots,C-1,\qquad \frac{\partial L}{\partial\,\text{in}}=\pi^{-1}\!\Big(\frac{\partial L}{\partial\,\text{out}}\Big)
g,  ng,\;n
number of groups and channels per group, C=gnC=gn
π−1\pi^{-1}
the backward pass applies the inverse permutation (reshape to (n,g)(n,g), transpose); no parameters, no arithmetic

For C=12C=12, g=3g=3 the new order is 0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11. Each group of the next convolution now reads one channel from each of the three previous groups. The operation is differentiable, has no parameters and costs only a memory copy, which is often fused into the next layer's read.

1×1 GConv cost=h w c1c2g⇒at a fixed budget, width can grow as g\text{1×1 GConv cost}=\frac{h\,w\,c_1c_2}{g}\quad\Rightarrow\quad\text{at a fixed budget, width can grow as }\sqrt g
c1,  c2c_1,\;c_2
input and output channels
gg
groups; v1 tried g∈{1,2,3,4,8}g\in\{1,2,3,4,8\} with stage widths 144 → 384 so all versions cost about 140M multiply-adds
12 channels, g = 3 groups of 4 · channel shuffle onGConv1 output (colour = the input group it was computed from)0123456789101104815926103711GConv2 group 1GConv2 group 2GConv2 group 3click a GConv2 group: it reads 3 of 3 input groupsGConv2 group ← input group9 / 9 linksevery group mixesreshape (3, 4) → transpose → flattenposition p ← channel (p mod 3)·4 + ⌊p/3⌋1×1 conv on 56×56 at fixed B = 51.38M mult-adds: memory access cost (MAC)G1: c₁ : c₂1:11:21:61:12G2: groups gg=1g=2g=4g=8c₁ 128, c₂ 128MAC 819.2k1.00× the boundc₁=c₂=128MAC 819.2k1.00× g=1- - lower bound 2√(hw·B) + B/hw = 819.2k, reached only at c₁ = c₂ with g = 1ShuffleNet v1 1×, g = 3 at 224×224paramsmult-addsconv1 3×3/2 + maxpool24×56²6968.13Mstage2240×28² ·4u39.6k32.18Mstage3480×14² ·8u307.7k63.79Mstage4960×7² ·4u556.8k32.41Mpool + FC1000×1²961.0k960.0ktotal 1,865,728 params · 137.5M mult-addspaper (Zhang et al. 2018): 137M mult-adds, 32.6% top-1 error
shuffle demo g
c₁:c₂groups
groups
fig.Top: twelve channels from a group conv with g = 3, coloured by the input group they came from, routed to the next group conv. With the shuffle on, every GConv2 group reads all 3 input groups (9 of 9 links in the reach matrix). Turn it off and each group sees only its own (3 of 9). Middle: the memory access cost (MAC) of a 1×1 conv on a 56×56 map at a fixed 51.4M multiply-adds. With equal widths 128:128 the MAC is 819.2k, exactly the lower bound. A 1:12 ratio raises it to 1.52M (1.86×). With 8 groups at the same FLOPs it reaches 2.29M (2.79×). Bottom: the whole network stage by stage. The default ShuffleNet v1 1×, g = 3 has 1,865,728 parameters (with BN) and 137.5M multiply-adds, against the paper's 137M. ShuffleNet v2 1.0× gives 2,278,604 parameters (torchvision's exact count) and 144.9M (paper 146M).

Worked example: one v1 unit (g = 3, stage 3, 480 channels on 14×14, stride 1)

  1. 1Bottleneck width is a quarter of the output: 480/4=120480/4=120.
  2. 21×11\times1 GConv 480→120480\to120 in 3 groups: 480⋅120/3=19,200480\cdot120/3=19{,}200 weights. Depthwise 3×33\times3: 9⋅120=1,0809\cdot120=1{,}080. 1×11\times1 GConv 120→480120\to480: 19,20019{,}200. That is 39,480 weights, plus 2⋅(120+120+480)=1,4402\cdot(120+120+480)=1{,}440 BN parameters.
  3. 3Multiply-adds 39,480⋅142=7,738,08039{,}480\cdot14^2=7{,}738{,}080. With dense 1×11\times1 convs (g=1g=1) the same widths would need 57,600+1,080+57,600=116,28057{,}600+1{,}080+57{,}600=116{,}280 weights, 2.95× more.
  4. 4Stage 3 is one stride-2 unit and seven of these: 307,680 parameters and 63.8M multiply-adds in the figure's library, almost half of the network's 137.5M.

v2: FLOPs are an indirect metric

Ma et al. measured networks with the same FLOPs and found very different speeds on GPU and ARM. FLOPs ignore memory access cost (MAC) and degree of parallelism. Their analysis of a 1×11\times1 conv with B=hwc1c2B=hwc_1c_2 multiply-adds gives the first guideline:

MAC=hw(c1+c2)+c1c2  ≥  2hw B+Bhw,equality iff c1=c2\text{MAC}=hw(c_1+c_2)+c_1c_2\;\ge\;2\sqrt{hw\,B}+\frac{B}{hw},\qquad \text{equality iff } c_1=c_2
hw(c1+c2)hw(c_1+c_2)
reading the input map and writing the output map
c1c2c_1c_2
reading the weights
≥\ge
AM–GM: c1+c2≥2c1c2=2B/hwc_1+c_2\ge2\sqrt{c_1c_2}=2\sqrt{B/hw}
settingc₁, c₂MACvs best
G1: 1:1128, 128819,2001.00× (= bound)
G1: 1:290.5, 181867,8991.06×
G1: 1:652.3, 313.51,163,5031.42×
G1: 1:1237, 443.41,522,7791.86×
G2: g=2g=2181, 1811,151,7371.41×
G2: g=4g=4256, 2561,622,0161.98×
G2: g=8g=8362, 3622,287,0912.79×
1×1 conv on 56×56 at fixed B = 51,380,224 multiply-adds (c = 128 at 1:1), from the figure's library

The four guidelines (Ma et al. 2018)

  1. 1G1, equal channel width minimizes MAC. For fixed FLOPs, a 1×11\times1 conv is cheapest in memory traffic when c1=c2c_1=c_2 (table, top rows).
  2. 2G2, excessive group convolution increases MAC. With gg groups, MAC=hwc1+Bg/c1+B/hw\text{MAC}=hwc_1+Bg/c_1+B/hw. At fixed FLOPs the wider layers that groups allow read and write more activations (table, bottom rows). Choose gg for the target platform, not by FLOPs alone.
  3. 3G3, network fragmentation reduces parallelism. Many small parallel branches (as in Inception cells or NAS cells) launch many small kernels and synchronize often. That is slow on GPUs even when the FLOPs are low.
  4. 4G4, element-wise operations are not negligible. ReLU, additions, bias and depthwise convs have few FLOPs but high MAC. Removing the ReLU and shortcut from a ResNet bottleneck made it about 20% faster in their measurements on both GPU and ARM.

ShuffleNet v1 breaks three of them: G2 with its group 1×11\times1 convs, G1 with its 4:1 bottleneck, and G4 with its residual add. The v2 unit obeys all four. Channel split sends c/2c/2 channels through an identity path and c/2c/2 through 1×1→1\times1\to dw 3×3→1×13\times3\to1\times1 with equal widths and no groups (G1, G2). There is a single branch of real work (G3), and concatenation replaces the add (G4). A channel shuffle then mixes the two halves. In the stride-2 unit both halves process the full input with stride 2, which doubles the channel count.

python
1import torch, torch.nn as nn
2from torchvision.models import shufflenet_v2_x1_0, ShuffleNet_V2_X1_0_Weights
3
4def channel_shuffle(x, g):
5 b, c, h, w = x.shape
6 return x.view(b, g, c // g, h, w).transpose(1, 2).reshape(b, c, h, w)
7
8x = torch.arange(12.).view(1, 12, 1, 1)
9print(channel_shuffle(x, 3).flatten().tolist()) # [0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11]
10print(torch.equal(nn.ChannelShuffle(3)(x), channel_shuffle(x, 3))) # True
11
12class ShuffleV2Unit(nn.Module): # stride 1
13 def __init__(self, c):
14 super().__init__()
15 b = c // 2
16 self.branch = nn.Sequential(
17 nn.Conv2d(b, b, 1, bias=False), nn.BatchNorm2d(b), nn.ReLU(inplace=True),
18 nn.Conv2d(b, b, 3, padding=1, groups=b, bias=False), nn.BatchNorm2d(b), # depthwise, no ReLU
19 nn.Conv2d(b, b, 1, bias=False), nn.BatchNorm2d(b), nn.ReLU(inplace=True))
20
21 def forward(self, x):
22 x1, x2 = x.chunk(2, dim=1) # channel split
23 return channel_shuffle(torch.cat([x1, self.branch(x2)], dim=1), 2)
24
25m = shufflenet_v2_x1_0(weights=ShuffleNet_V2_X1_0_Weights.IMAGENET1K_V1).eval()
26print(sum(p.numel() for p in m.parameters())) # 2278604
Channel shuffle and a ShuffleNet v2 unit

Use when

  • +Very low multiply-adds: v1 and v2 1× need about 140M
  • +Channel shuffle is free in parameters and FLOPs and restores cross-group flow
  • +v2 is fast on real hardware: equal widths, no groups, concat instead of add
  • +Simple width scaling (0.5× to 2×) and good torchvision weights for v2

Avoid when

  • −v1's group convs and channel shuffles are memory-bound on GPUs
  • −Shuffle and split are reshapes/copies that some runtimes and accelerators handle poorly
  • −Accuracy now below newer mobile nets (MobileNetV3, EfficientNet-Lite) at similar cost
  • −Guidelines are about a 1×1-conv model of cost; real kernels and fusion can change the picture

Check yourself

Two group convolutions with g = 4 are stacked, with no shuffle between them. Output group 2 of the second layer depends on which input groups?

Check yourself

A 1×1 conv on a 28×28 map must cost exactly 25M multiply-adds. Which channel choice gives the lowest memory access cost?

Remember

  • ✓v1: group 1×11\times1 convs cut the dominant cost by gg; the channel shuffle (reshape (g,n)(g,n) → transpose → flatten) lets groups exchange information.
  • ✓v1 1×, g=3g=3: about 137M multiply-adds, 32.6% top-1 error; the shuffle is worth 1.9 points at g=3g=3 and 5.2 at g=8g=8.
  • ✓v2: FLOPs are indirect. MAC of a 1×11\times1 conv is ≥2hwB+B/hw\ge 2\sqrt{hwB}+B/hw, minimized at c1=c2c_1=c_2.
  • ✓Four guidelines: equal widths, avoid excessive groups, limit fragmentation, count element-wise ops.
  • ✓v2 unit: channel split, a single equal-width branch, concat, shuffle; no group convs.
← back to Deep Learning roadmap