ResNeXt

A ResNet whose bottleneck is split into C parallel narrow paths (a grouped 3×3 conv). At the same cost, raising this cardinality beats making the net deeper or wider: ResNeXt-50 32×4d has fewer parameters than ResNet-50 and 1.7 points lower top-1 error.

12 min read1176 wordsadvanced0 viewsupdated 2026-10-11

Definition

ResNeXt is a residual network whose blocks compute y=x+∑i=1CTi(x)y=x+\sum_{i=1}^{C}\mathcal T_i(x): the bottleneck branch is split into CC identical, narrow transformations whose outputs are summed. The number of paths CC is called the cardinality. In practice the CC paths are implemented as a single grouped convolution with CC groups.

Xie, Girshick, Dollár, Tu and He introduced ResNeXt in "Aggregated Residual Transformations for Deep Neural Networks" (CVPR 2017). The paper's claim is that cardinality is a third way to scale a network, alongside depth and width, and that at a fixed budget of parameters and FLOPs it is the most effective of the three. A ResNeXt entry took second place in the ILSVRC 2016 classification task.

The design takes two ideas and keeps both. From VGG and Residual neural network (ResNet) it keeps the repetition of one block type, so there are very few hyperparameters. From Inception it keeps split-transform-merge: project the input into several low-dimensional embeddings, transform each one, and merge the results. Inception's branches are hand-designed and all different. ResNeXt makes every branch the same, so the whole block is described by just two numbers, the cardinality CC and the bottleneck width per path dd.

At a glance

Paper
Xie, Girshick, Dollár, Tu, He, CVPR 2017
Key idea
cardinality CC: identical parallel paths, summed (a grouped 3×33\times3)
Template
ResNeXt-50 32×4d: C=32C=32 paths of d=4d=4 channels in stage conv2 (width Cd=128Cd=128)
Cost
25.0M params, 4.2G multiply-adds (ResNet-50: 25.6M, 4.1G)
Accuracy
22.2% vs 23.9% ImageNet top-1 error (paper, 224 single crop)
In PyTorch
nn.Conv2d(..., groups=C); torchvision.models.resnext50_32x4d

Aggregated transformations

A single neuron already does split-transform-merge: ∑iwixi\sum_i w_ix_i splits xx into coordinates, scales each one and adds them up. ResNeXt keeps that shape but replaces each scalar product with a small network Ti\mathcal T_i:

y=x+∑i=1CTi(x),Ti(x)=Wi(3) σ ⁣(Wi(2)∗σ(Wi(1)x))y=x+\sum_{i=1}^{C}\mathcal T_i(x),\qquad \mathcal T_i(x)=W^{(3)}_i\,\sigma\!\Big(W^{(2)}_i * \sigma\big(W^{(1)}_i x\big)\Big)
CC
cardinality: the number of parallel transformations (32 in ResNeXt-50 32×4d)
Wi(1)W^{(1)}_i
1×11\times1 conv projecting the 256-channel input to dd channels (BN + ReLU follow)
Wi(2)∗W^{(2)}_i *
3×33\times3 conv, d→dd\to d channels: the only spatial operation in the path
Wi(3)W^{(3)}_i
1×11\times1 conv back to 256 channels; the path outputs are summed
σ\sigma
BatchNorm then ReLU (the last BN comes before the addition, the last ReLU after it)

The network follows two rules taken from VGG and ResNet. Blocks that produce maps of the same size share the same CC and dd. Each time the map is downsampled by 2, the width of the block is doubled. With those rules, choosing the conv2 template, for example "C=32C=32, d=4d=4", fixes the entire network. That is what the name ResNeXt-50 (32×4d) means: ResNet-50's stage layout of 3, 4, 6, 3 blocks, cardinality 32, 4 channels per path in the first stage.

Three equivalent forms

The paper shows the block in three forms that compute the same function.

Forms (a), (b), (c) of one ResNeXt block (BN omitted)

  1. 1(a) Aggregated paths. CC separate paths, each 1×11\times1 (256→dd) → 3×33\times3 (dd→dd) → 1×11\times1 (dd→256), summed. This is the definition.
  2. 2(b) Early concatenation. Summing the outputs of CC separate 1×11\times1 convs is the same as concatenating their inputs and applying one 1×11\times1 conv whose weight matrix is the CC blocks side by side: ∑iWi(3)hi=[W1(3)⋯WC(3)] [h1;… ;hC]\sum_i W^{(3)}_ih_i=[W^{(3)}_1\cdots W^{(3)}_C]\,[h_1;\dots;h_C].
  3. 3(c) Grouped convolution. The CC first-layer 1×11\times1 convs also read the same input, so they stack into one 1×11\times1 conv 256→CdCd. The CC independent 3×33\times3 convs become one 3×33\times3 conv with groups=C\mathtt{groups}=C: output channel group ii sees only input channel group ii. That leaves 1×1 → grouped 3×3 → 1×1, a ResNet bottleneck with a wider middle and a sparser 3×33\times3.

Form (c) is what every library implements, because one grouped conv runs as a single kernel launch. Two caveats: the equivalence needs depth ≥ 3 inside the block (with only two layers, the grouping collapses into a wide dense layer), and the BN after the first two layers is per channel, so it splits across paths without changing anything.

form (c) · C = 32, d = 4 · conv2 template blockinput x · 256 channels1×1 conv, 256 → 1283×3 conv, 128 → 128, groups = 321×1 conv, 128 → 256+identityy = x + Σ Tᵢ(x) · 256 chtemplate conv weights: 70,144 (ResNet: 69,632)test block 16ch, 6×6: max|a−c| 1.3e−15, max|b−c| 0 (rms 0.88)stageoutputC·dparamsMACsstem64×56²–0.01M0.12Gconv2256×56²1280.21M0.63Gconv3512×28²2561.20M1.09Gconv41024×14²5127.02M1.53Gconv52048×7²102414.54M0.87Gfc1000–2.05M0.00Gtotal25.03M4.23GResNet-5025.56M4.09Gconv2: 3 blocks → 256×56×563×3 groups = 32, 4 ch each205,824 params · 634,224,640 MACsImageNet top-1 err (paper, 224 crop):22.2% vs 23.9% for 1×64d
C × d
formclick a stage row
fig.Pick a C × d setting from the paper's Table 2 and a form. The left panel draws the conv2 template block. The bottom line runs forms (a) and (b) against (c) with the same random weights on a 16-channel 6×6 test input: at 32×4d the largest difference is about 1e−15, which is float64 round-off, so the three are the same function. The right panel counts the whole ImageNet network from its real configuration: 32×4d gives 25.03M parameters and 4.23G multiply-adds, against 25.56M and 4.09G for ResNet-50 (1×64d). Click a stage row to see its shapes, groups and counts. The paper's top-1 error falls from 23.9% at 1×64d to 22.2% at 32×4d, with the template block held at roughly 70k weights.

Same budget, more paths

The conv weights of one template block, with 256 input and output channels, are C⋅(256d+9d2+256d)C\cdot(256d+9d^2+256d). Holding this at about 70k while increasing CC forces dd down, but the total bottleneck width CdCd grows, because the 3×33\times3 term 9Cd29Cd^2 shrinks as dd shrinks. The paper's Table 2 settings, with counts checked against the figure's code:

C × dwidth C·dtemplate weightsparamsMACstop-1 err.
1 × 64d (ResNet-50)6469,63225.56M4.09G23.9%
2 × 40d8069,76025.43M4.12G23.0%
4 × 24d9669,88825.29M4.16G22.6%
8 × 14d11271,45625.60M4.27G22.3%
32 × 4d12870,14425.03M4.23G22.2%
ResNeXt-50 settings at about the same complexity. Template = conv weights of one conv2 block. Whole-network params and multiply-adds counted from the torchvision layout (224×224 input). Top-1 error: Xie et al. 2017, Table 3, single 224 crop.

The gain is not overfitting relief: the paper reports lower training error for 32×4d as well, so the extra paths give a better-conditioned or more expressive block at the same cost. The same trend holds for ResNet-101 (22.0% → 21.2%). At twice the cost of ResNet-101, going deeper (ResNet-200, 21.7%) or wider (1×100d, 21.3%) helped less than going to 64×4d (20.4%).

Worked example: the 32×4d template block

Counting conv2 of ResNeXt-50 (input 256×56×56)

  1. 1First 1×11\times1: 256 → Cd=128Cd=128 channels, 256⋅128=32,768256\cdot128=32{,}768 weights.
  2. 2Grouped 3×33\times3, 128 → 128 with 32 groups: each output channel sees 128/32=4128/32=4 input channels, so 128⋅4⋅9=4,608128\cdot4\cdot9=4{,}608 weights. A dense 3×33\times3 at 128 channels would need 128⋅128⋅9=147,456128\cdot128\cdot9=147{,}456, 32× more.
  3. 3Last 1×11\times1: 128 → 256, another 32,76832{,}768. Total 70,14470{,}144 conv weights, versus 69,63269{,}632 for the ResNet-50 block, whose middle is a dense 64-channel 3×33\times3 (36,86436{,}864 weights).
  4. 4Multiply-adds at 56×5656\times56: 70,144⋅3136=220.070{,}144\cdot3136=220.0M per block. ResNeXt spends its budget on the two cheap 1×11\times1 layers and on a twice-as-wide (128 vs 64) representation, not on the 3×33\times3 mixing.
  5. 5Whole network: summing every block, the projections, the stem and the 2048→1000 classifier gives 25,028,904 parameters (BN included), exactly torchvision's count for resnext50_32x4d, and 4.23G multiply-adds.

Gradients through a grouped convolution

In form (c) the 3×33\times3 weight tensor is block-diagonal across groups. The Jacobian of the grouped layer with respect to its input is therefore block-diagonal too, and each group's weights receive gradient only from their own output channels:

∂L∂Wi(2)=∑p∂L∂zi,p  patchp(hi),∂L∂x=∂L∂y(I+∑i=1C∂Ti∂x)\frac{\partial\mathcal L}{\partial W^{(2)}_i}=\sum_{p}\frac{\partial\mathcal L}{\partial z_{i,p}}\;\mathrm{patch}_p\big(h_i\big),\qquad \frac{\partial\mathcal L}{\partial x}=\frac{\partial\mathcal L}{\partial y}\Big(I+\sum_{i=1}^{C}\frac{\partial\mathcal T_i}{\partial x}\Big)
zi,pz_{i,p}
output of group ii at spatial position pp
patchp(hi)\mathrm{patch}_p(h_i)
the 3×3×d3\times3\times d patch of group ii's input around pp; group j≠ij\ne i never appears
II
identity shortcut: as in ResNet, the gradient reaches the block input unchanged
∑i∂Ti/∂x\sum_i \partial\mathcal T_i/\partial x
the paths are summed forward, so their input gradients are summed backward

The paths only interact through the shared 1×11\times1 layers before and after them. Information mixes across all 128 channels at the start and end of each block, and the 3×33\times3 inside works on 32 independent 4-channel slices. Trained ResNeXts do learn different filters in different groups, an effect AlexNet first reported for its two GPU groups (one group learned colour-agnostic filters, the other colour-specific ones).

python
1import torch, torch.nn as nn
2from torchvision.models import resnext50_32x4d, ResNeXt50_32X4D_Weights
3
4class ResNeXtBlock(nn.Module):
5 """Form (c): 1x1 -> grouped 3x3 -> 1x1, plus the residual shortcut."""
6 def __init__(self, c_in, c_out, cardinality=32, d=4, stride=1):
7 super().__init__()
8 width = cardinality * d # 128 for the conv2 template
9 self.branch = nn.Sequential(
10 nn.Conv2d(c_in, width, 1, bias=False), nn.BatchNorm2d(width), nn.ReLU(inplace=True),
11 nn.Conv2d(width, width, 3, stride=stride, padding=1,
12 groups=cardinality, bias=False), # weight shape (128, 4, 3, 3)
13 nn.BatchNorm2d(width), nn.ReLU(inplace=True),
14 nn.Conv2d(width, c_out, 1, bias=False), nn.BatchNorm2d(c_out),
15 )
16 self.short = nn.Identity()
17 if stride != 1 or c_in != c_out:
18 self.short = nn.Sequential(nn.Conv2d(c_in, c_out, 1, stride=stride, bias=False),
19 nn.BatchNorm2d(c_out))
20
21 def forward(self, x):
22 return torch.relu(self.short(x) + self.branch(x))
23
24blk = ResNeXtBlock(256, 256)
25convs = sum(m.weight.numel() for m in blk.modules() if isinstance(m, nn.Conv2d))
26print(convs) # 70144
27
28model = resnext50_32x4d(weights=ResNeXt50_32X4D_Weights.IMAGENET1K_V2).eval()
29print(sum(p.numel() for p in model.parameters())) # 25028904
30print(model.layer1[0].conv2) # Conv2d(128, 128, kernel_size=(3, 3), ..., groups=32, bias=False)
A ResNeXt block in form (c), the torchvision model, and a check of the parameter count
ModelParamsGFLOPsacc@1 V1acc@1 V2
resnet5025.6M4.0976.13%80.86%
resnext50_32x4d25.0M4.2377.62%81.20%
resnext101_32x8d88.8M16.4179.31%82.83%
resnext101_64x4d83.5M15.46–83.25%
torchvision ImageNet weights (documented acc@1 at 224 crop; V1 = original recipe, V2 = torchvision's 2021 improved training recipe)

Use when

  • +Better accuracy than ResNet at equal parameters and FLOPs (22.2% vs 23.9% top-1 for the 50-layer models)
  • +Only two new hyperparameters, CC and dd; the rest is inherited from ResNet
  • +Drop-in: same stage layout, same shortcuts, a groups argument on one conv
  • +Strong, widely available pretrained backbones for detection and segmentation

Avoid when

  • −Grouped convs run slower than their FLOP count suggests on GPUs
  • −The bottleneck is twice as wide, so it uses more activation memory than ResNet-50
  • −No mixing across groups inside the 3×3; only the 1×1 layers mix
  • −Superseded on accuracy per FLOP by EfficientNet, RegNet and ConvNeXt

Check yourself

A 3×3 convolution maps 128 channels to 128 channels with groups = 32. How many weights does it have (no bias)?

Check yourself

Why is form (a), C separate 1×1 → 3×3 → 1×1 paths summed, the same function as form (c) with a grouped 3×3?

Remember

  • ✓ResNeXt block: y=x+∑i=1CTi(x)y=x+\sum_{i=1}^{C}\mathcal T_i(x) with CC identical bottleneck paths; CC is the cardinality.
  • ✓Implemented as 1×1 → grouped 3×3 (groups = C) → 1×1; the split-sum, concatenate and grouped forms are mathematically identical.
  • ✓ResNeXt-50 32×4d: bottleneck width 128 with 4 channels per group, 25.0M params and 4.2 GFLOPs, 22.2% vs 23.9% top-1 for ResNet-50.
  • ✓At a fixed budget, raising cardinality helped more than going deeper or wider in the paper's experiments.
  • ✓Grouped convs save FLOPs but not proportionally time; check real throughput.
← back to Deep Learning roadmap