# ShuffleNet

> V1 makes the 1×1 convs cheap with group convolutions and a channel shuffle so groups still exchange information. V2 shows FLOPs mislead and gives four speed guidelines: equal widths, few groups, little fragmentation, few element-wise ops.

Source: https://www.mlfoundry.in/learn/shufflenet  
Section: Deep Learning › Convolutional Neural Networks  
Level: advanced · 12 min read · updated 2026-10-11

**ShuffleNet:** is a family of mobile CNNs from Megvii. **ShuffleNet v1** (Zhang, Zhou, Lin and Sun, CVPR 2018) uses **pointwise group convolutions** for the $1\times1$ layers and a **channel shuffle** that permutes channels between groups. **ShuffleNet v2** (Ma, Zhang, Zheng and Sun, ECCV 2018) drops the group convolutions. It is built from **channel split** units designed with four guidelines for real-device speed.

In a **MobileNet**-style network the depthwise $3\times3$ convs are cheap and the dense $1\times1$ convs take over 90% of the multiply-adds. ShuffleNet v1 attacks that remaining cost by splitting the $1\times1$ convs into $g$ groups, as **ResNeXt** does for its $3\times3$ convs. That divides their cost by $g$, and the saving can be spent on wider layers. The catch is that stacked group convolutions never mix information across groups. The channel shuffle fixes that for free.

**At a glance**

- **v1 unit:** $1\times1$ GConv → BN, ReLU → **shuffle** → dw $3\times3$ → BN → $1\times1$ GConv → BN, add, ReLU; stride 2: avg-pool shortcut and concat
- **v2 unit:** split channels in half; one half is identity, the other goes $1\times1$ → dw $3\times3$ → $1\times1$ (equal widths, no groups); concat, then shuffle
- **v1 result:** 1×, $g=3$: 137M multiply-adds, 32.6% top-1 error; 7.8 points lower error than MobileNet at about 40M multiply-adds; about 13× faster than AlexNet on an ARM phone
- **v2 result:** 1×: 146M multiply-adds; torchvision weights 2,278,604 params, 69.36% top-1
- **Note:** both papers say "FLOPs" for multiply-adds
- **In PyTorch:** `nn.ChannelShuffle(g)`, `torchvision.models.shufflenet_v2_x1_0` (v1 is not in torchvision)

## Channel shuffle

With $g$ groups, a group convolution's output group $G$ depends only on input group $G$. Stack two of them and the network splits into $g$ independent narrow towers. The shuffle sends one channel from every group into every group before the next layer. With $C=gn$ channels it reshapes the channel axis to $(g,n)$, transposes to $(n,g)$ and flattens:

$$
\text{out}[p]=\text{in}\big[\,(p\bmod g)\cdot n+\lfloor p/g\rfloor\,\big],\qquad p=0,\dots,C-1,\qquad \frac{\partial L}{\partial\,\text{in}}=\pi^{-1}\!\Big(\frac{\partial L}{\partial\,\text{out}}\Big)
$$

- $g,\;n$: number of groups and channels per group, $C=gn$
- $\pi^{-1}$: the backward pass applies the inverse permutation (reshape to $(n,g)$, transpose); no parameters, no arithmetic

For $C=12$, $g=3$ the new order is 0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11. Each group of the next convolution now reads one channel from each of the three previous groups. The operation is differentiable, has no parameters and costs only a memory copy, which is often fused into the next layer's read.

$$
\text{1×1 GConv cost}=\frac{h\,w\,c_1c_2}{g}\quad\Rightarrow\quad\text{at a fixed budget, width can grow as }\sqrt g
$$

- $c_1,\;c_2$: input and output channels
- $g$: groups; v1 tried $g\in\{1,2,3,4,8\}$ with stage widths 144 → 384 so all versions cost about 140M multiply-adds

> Figure: Top: twelve channels from a group conv with g = 3, coloured by the input group they came from, routed to the next group conv. With the shuffle on, every GConv2 group reads all 3 input groups (9 of 9 links in the reach matrix). Turn it off and each group sees only its own (3 of 9). Middle: the memory access cost (MAC) of a 1×1 conv on a 56×56 map at a fixed 51.4M multiply-adds. With equal widths 128:128 the MAC is 819.2k, exactly the lower bound. A 1:12 ratio raises it to 1.52M (1.86×). With 8 groups at the same FLOPs it reaches 2.29M (2.79×). Bottom: the whole network stage by stage. The default ShuffleNet v1 1×, g = 3 has 1,865,728 parameters (with BN) and 137.5M multiply-adds, against the paper's 137M. ShuffleNet v2 1.0× gives 2,278,604 parameters (torchvision's exact count) and 144.9M (paper 146M).

**Worked example: one v1 unit (g = 3, stage 3, 480 channels on 14×14, stride 1)**

1. Bottleneck width is a quarter of the output: $480/4=120$.
2. $1\times1$ GConv $480\to120$ in 3 groups: $480\cdot120/3=19{,}200$ weights. Depthwise $3\times3$: $9\cdot120=1{,}080$. $1\times1$ GConv $120\to480$: $19{,}200$. That is 39,480 weights, plus $2\cdot(120+120+480)=1{,}440$ BN parameters.
3. Multiply-adds $39{,}480\cdot14^2=7{,}738{,}080$. With dense $1\times1$ convs ($g=1$) the same widths would need $57{,}600+1{,}080+57{,}600=116{,}280$ weights, 2.95× more.
4. Stage 3 is one stride-2 unit and seven of these: 307,680 parameters and 63.8M multiply-adds in the figure's library, almost half of the network's 137.5M.

## v2: FLOPs are an indirect metric

Ma et al. measured networks with the same FLOPs and found very different speeds on GPU and ARM. FLOPs ignore **memory access cost** (MAC) and **degree of parallelism**. Their analysis of a $1\times1$ conv with $B=hwc_1c_2$ multiply-adds gives the first guideline:

$$
\text{MAC}=hw(c_1+c_2)+c_1c_2\;\ge\;2\sqrt{hw\,B}+\frac{B}{hw},\qquad \text{equality iff } c_1=c_2
$$

- $hw(c_1+c_2)$: reading the input map and writing the output map
- $c_1c_2$: reading the weights
- $\ge$: AM–GM: $c_1+c_2\ge2\sqrt{c_1c_2}=2\sqrt{B/hw}$

1×1 conv on 56×56 at fixed B = 51,380,224 multiply-adds (c = 128 at 1:1), from the figure's library

| setting | c₁, c₂ | MAC | vs best |
| --- | --- | --- | --- |
| G1: 1:1 | 128, 128 | 819,200 | 1.00× (= bound) |
| G1: 1:2 | 90.5, 181 | 867,899 | 1.06× |
| G1: 1:6 | 52.3, 313.5 | 1,163,503 | 1.42× |
| G1: 1:12 | 37, 443.4 | 1,522,779 | 1.86× |
| G2: $g=2$ | 181, 181 | 1,151,737 | 1.41× |
| G2: $g=4$ | 256, 256 | 1,622,016 | 1.98× |
| G2: $g=8$ | 362, 362 | 2,287,091 | 2.79× |

**The four guidelines (Ma et al. 2018)**

1. **G1, equal channel width minimizes MAC.** For fixed FLOPs, a $1\times1$ conv is cheapest in memory traffic when $c_1=c_2$ (table, top rows).
2. **G2, excessive group convolution increases MAC.** With $g$ groups, $\text{MAC}=hwc_1+Bg/c_1+B/hw$. At fixed FLOPs the wider layers that groups allow read and write more activations (table, bottom rows). Choose $g$ for the target platform, not by FLOPs alone.
3. **G3, network fragmentation reduces parallelism.** Many small parallel branches (as in Inception cells or NAS cells) launch many small kernels and synchronize often. That is slow on GPUs even when the FLOPs are low.
4. **G4, element-wise operations are not negligible.** ReLU, additions, bias and depthwise convs have few FLOPs but high MAC. Removing the ReLU and shortcut from a ResNet bottleneck made it about 20% faster in their measurements on both GPU and ARM.

ShuffleNet v1 breaks three of them: G2 with its group $1\times1$ convs, G1 with its 4:1 bottleneck, and G4 with its residual add. The v2 unit obeys all four. **Channel split** sends $c/2$ channels through an identity path and $c/2$ through $1\times1\to$ dw $3\times3\to1\times1$ with equal widths and no groups (G1, G2). There is a single branch of real work (G3), and **concatenation** replaces the add (G4). A channel shuffle then mixes the two halves. In the stride-2 unit both halves process the full input with stride 2, which doubles the channel count.

```python
import torch, torch.nn as nn
from torchvision.models import shufflenet_v2_x1_0, ShuffleNet_V2_X1_0_Weights

def channel_shuffle(x, g):
    b, c, h, w = x.shape
    return x.view(b, g, c // g, h, w).transpose(1, 2).reshape(b, c, h, w)

x = torch.arange(12.).view(1, 12, 1, 1)
print(channel_shuffle(x, 3).flatten().tolist())  # [0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11]
print(torch.equal(nn.ChannelShuffle(3)(x), channel_shuffle(x, 3)))   # True

class ShuffleV2Unit(nn.Module):                     # stride 1
    def __init__(self, c):
        super().__init__()
        b = c // 2
        self.branch = nn.Sequential(
            nn.Conv2d(b, b, 1, bias=False), nn.BatchNorm2d(b), nn.ReLU(inplace=True),
            nn.Conv2d(b, b, 3, padding=1, groups=b, bias=False), nn.BatchNorm2d(b),   # depthwise, no ReLU
            nn.Conv2d(b, b, 1, bias=False), nn.BatchNorm2d(b), nn.ReLU(inplace=True))

    def forward(self, x):
        x1, x2 = x.chunk(2, dim=1)                  # channel split
        return channel_shuffle(torch.cat([x1, self.branch(x2)], dim=1), 2)

m = shufflenet_v2_x1_0(weights=ShuffleNet_V2_X1_0_Weights.IMAGENET1K_V1).eval()
print(sum(p.numel() for p in m.parameters()))   # 2278604
```

Channel shuffle and a ShuffleNet v2 unit

> **Shuffle without groups does nothing:** The shuffle only matters when it sits between two **grouped** layers (v1) or between two halves that would otherwise never meet (v2). Shuffling before a dense $1\times1$ conv just permutes its weight rows, and the network can learn any permutation anyway. Full mixing in one step also needs at least as many channels per group as there are groups ($n\ge g$), which real networks easily satisfy (60 per group in a v1 unit with $g=3$). In the figure's 12-channel toy, $g=2$ and $g=3$ give a full reach matrix with the shuffle on, while $g=4$ and $g=6$ reach only 3 of 4 and 2 of 6 groups per step. Without the shuffle the matrix stays diagonal however many layers you stack.

> **Measure the metric you care about:** The lasting contribution of ShuffleNet v2 is methodological: compare architectures by measured speed on the target hardware, with batch size and precision stated, not by FLOPs. At equal FLOPs, layers differ a lot in memory traffic and parallelism. The same lesson motivated latency-aware search in MobileNetV3 and the GPU-friendly design spaces of RegNet.

> **Shuffle results:** In ShuffleNet v1's ablation (Table 3), removing the shuffle raised top-1 error of the 1× model from 32.6% to 34.5% at $g=3$ and from 32.4% to 37.6% at $g=8$. More groups make the isolation worse, so they make the shuffle matter more.

**Strengths**

- Very low multiply-adds: v1 and v2 1× need about 140M
- Channel shuffle is free in parameters and FLOPs and restores cross-group flow
- v2 is fast on real hardware: equal widths, no groups, concat instead of add
- Simple width scaling (0.5× to 2×) and good torchvision weights for v2

**Limitations**

- v1's group convs and channel shuffles are memory-bound on GPUs
- Shuffle and split are reshapes/copies that some runtimes and accelerators handle poorly
- Accuracy now below newer mobile nets (MobileNetV3, EfficientNet-Lite) at similar cost
- Guidelines are about a 1×1-conv model of cost; real kernels and fusion can change the picture

**Question:** Two group convolutions with g = 4 are stacked, with no shuffle between them. Output group 2 of the second layer depends on which input groups?

- A. Only group 2
- B. Groups 1 and 2
- C. All four groups
- D. None: it depends only on the bias

**Answer:** A. Only group 2 — Each group conv maps group $G$ to group $G$ only, so composing two of them still maps group 2 to group 2. The towers never meet. A channel shuffle between them would make output group 2 read one channel from each of the four groups.

**Question:** A 1×1 conv on a 28×28 map must cost exactly 25M multiply-adds. Which channel choice gives the lowest memory access cost?

- A. c₁ = 64, c₂ ≈ 498
- B. c₁ = c₂ ≈ 179
- C. c₁ ≈ 498, c₂ = 64
- D. It does not matter: same FLOPs means same MAC

**Answer:** B. c₁ = c₂ ≈ 179 — MAC $=hw(c_1+c_2)+c_1c_2$ with $c_1c_2$ fixed, and $c_1+c_2$ is smallest when $c_1=c_2=\sqrt{B/hw}=\sqrt{25\text{M}/784}\approx179$. This is guideline G1. Both 64/498 options have the same product but a much larger sum.

**Key points**

- v1: group $1\times1$ convs cut the dominant cost by $g$; the channel shuffle (reshape $(g,n)$ → transpose → flatten) lets groups exchange information.
- v1 1×, $g=3$: about 137M multiply-adds, 32.6% top-1 error; the shuffle is worth 1.9 points at $g=3$ and 5.2 at $g=8$.
- v2: FLOPs are indirect. MAC of a $1\times1$ conv is $\ge 2\sqrt{hwB}+B/hw$, minimized at $c_1=c_2$.
- Four guidelines: equal widths, avoid excessive groups, limit fragmentation, count element-wise ops.
- v2 unit: channel split, a single equal-width branch, concat, shuffle; no group convs.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
