Definition
ResNeXt is a residual network whose blocks compute : the bottleneck branch is split into identical, narrow transformations whose outputs are summed. The number of paths is called the cardinality. In practice the paths are implemented as a single grouped convolution with groups.
Xie, Girshick, Dollár, Tu and He introduced ResNeXt in "Aggregated Residual Transformations for Deep Neural Networks" (CVPR 2017). The paper's claim is that cardinality is a third way to scale a network, alongside depth and width, and that at a fixed budget of parameters and FLOPs it is the most effective of the three. A ResNeXt entry took second place in the ILSVRC 2016 classification task.
The design takes two ideas and keeps both. From VGG and Residual neural network (ResNet) it keeps the repetition of one block type, so there are very few hyperparameters. From Inception it keeps split-transform-merge: project the input into several low-dimensional embeddings, transform each one, and merge the results. Inception's branches are hand-designed and all different. ResNeXt makes every branch the same, so the whole block is described by just two numbers, the cardinality and the bottleneck width per path .
At a glance
- Paper
- Xie, Girshick, Dollár, Tu, He, CVPR 2017
- Key idea
- cardinality : identical parallel paths, summed (a grouped )
- Template
- ResNeXt-50 32×4d: paths of channels in stage conv2 (width )
- Cost
- 25.0M params, 4.2G multiply-adds (ResNet-50: 25.6M, 4.1G)
- Accuracy
- 22.2% vs 23.9% ImageNet top-1 error (paper, 224 single crop)
- In PyTorch
nn.Conv2d(..., groups=C);torchvision.models.resnext50_32x4d
Aggregated transformations
A single neuron already does split-transform-merge: splits into coordinates, scales each one and adds them up. ResNeXt keeps that shape but replaces each scalar product with a small network :
- cardinality: the number of parallel transformations (32 in ResNeXt-50 32×4d)
- conv projecting the 256-channel input to channels (BN + ReLU follow)
- conv, channels: the only spatial operation in the path
- conv back to 256 channels; the path outputs are summed
- BatchNorm then ReLU (the last BN comes before the addition, the last ReLU after it)
The network follows two rules taken from VGG and ResNet. Blocks that produce maps of the same size share the same and . Each time the map is downsampled by 2, the width of the block is doubled. With those rules, choosing the conv2 template, for example ", ", fixes the entire network. That is what the name ResNeXt-50 (32×4d) means: ResNet-50's stage layout of 3, 4, 6, 3 blocks, cardinality 32, 4 channels per path in the first stage.
Three equivalent forms
The paper shows the block in three forms that compute the same function.
Forms (a), (b), (c) of one ResNeXt block (BN omitted)
- 1(a) Aggregated paths. separate paths, each (256→) → (→) → (→256), summed. This is the definition.
- 2(b) Early concatenation. Summing the outputs of separate convs is the same as concatenating their inputs and applying one conv whose weight matrix is the blocks side by side: .
- 3(c) Grouped convolution. The first-layer convs also read the same input, so they stack into one conv 256→. The independent convs become one conv with : output channel group sees only input channel group . That leaves 1×1 → grouped 3×3 → 1×1, a ResNet bottleneck with a wider middle and a sparser .
Form (c) is what every library implements, because one grouped conv runs as a single kernel launch. Two caveats: the equivalence needs depth ≥ 3 inside the block (with only two layers, the grouping collapses into a wide dense layer), and the BN after the first two layers is per channel, so it splits across paths without changing anything.
Same budget, more paths
The conv weights of one template block, with 256 input and output channels, are . Holding this at about 70k while increasing forces down, but the total bottleneck width grows, because the term shrinks as shrinks. The paper's Table 2 settings, with counts checked against the figure's code:
| C × d | width C·d | template weights | params | MACs | top-1 err. |
|---|---|---|---|---|---|
| 1 × 64d (ResNet-50) | 64 | 69,632 | 25.56M | 4.09G | 23.9% |
| 2 × 40d | 80 | 69,760 | 25.43M | 4.12G | 23.0% |
| 4 × 24d | 96 | 69,888 | 25.29M | 4.16G | 22.6% |
| 8 × 14d | 112 | 71,456 | 25.60M | 4.27G | 22.3% |
| 32 × 4d | 128 | 70,144 | 25.03M | 4.23G | 22.2% |
The gain is not overfitting relief: the paper reports lower training error for 32×4d as well, so the extra paths give a better-conditioned or more expressive block at the same cost. The same trend holds for ResNet-101 (22.0% → 21.2%). At twice the cost of ResNet-101, going deeper (ResNet-200, 21.7%) or wider (1×100d, 21.3%) helped less than going to 64×4d (20.4%).
Worked example: the 32×4d template block
Counting conv2 of ResNeXt-50 (input 256×56×56)
- 1First : 256 → channels, weights.
- 2Grouped , 128 → 128 with 32 groups: each output channel sees input channels, so weights. A dense at 128 channels would need , 32× more.
- 3Last : 128 → 256, another . Total conv weights, versus for the ResNet-50 block, whose middle is a dense 64-channel ( weights).
- 4Multiply-adds at : M per block. ResNeXt spends its budget on the two cheap layers and on a twice-as-wide (128 vs 64) representation, not on the mixing.
- 5Whole network: summing every block, the projections, the stem and the 2048→1000 classifier gives 25,028,904 parameters (BN included), exactly torchvision's count for
resnext50_32x4d, and 4.23G multiply-adds.
Gradients through a grouped convolution
In form (c) the weight tensor is block-diagonal across groups. The Jacobian of the grouped layer with respect to its input is therefore block-diagonal too, and each group's weights receive gradient only from their own output channels:
- output of group at spatial position
- the patch of group 's input around ; group never appears
- identity shortcut: as in ResNet, the gradient reaches the block input unchanged
- the paths are summed forward, so their input gradients are summed backward
The paths only interact through the shared layers before and after them. Information mixes across all 128 channels at the start and end of each block, and the inside works on 32 independent 4-channel slices. Trained ResNeXts do learn different filters in different groups, an effect AlexNet first reported for its two GPU groups (one group learned colour-agnostic filters, the other colour-specific ones).
1import torch, torch.nn as nn2from torchvision.models import resnext50_32x4d, ResNeXt50_32X4D_Weights34class ResNeXtBlock(nn.Module):5 """Form (c): 1x1 -> grouped 3x3 -> 1x1, plus the residual shortcut."""6 def __init__(self, c_in, c_out, cardinality=32, d=4, stride=1):7 super().__init__()8 width = cardinality * d # 128 for the conv2 template9 self.branch = nn.Sequential(10 nn.Conv2d(c_in, width, 1, bias=False), nn.BatchNorm2d(width), nn.ReLU(inplace=True),11 nn.Conv2d(width, width, 3, stride=stride, padding=1,12 groups=cardinality, bias=False), # weight shape (128, 4, 3, 3)13 nn.BatchNorm2d(width), nn.ReLU(inplace=True),14 nn.Conv2d(width, c_out, 1, bias=False), nn.BatchNorm2d(c_out),15 )16 self.short = nn.Identity()17 if stride != 1 or c_in != c_out:18 self.short = nn.Sequential(nn.Conv2d(c_in, c_out, 1, stride=stride, bias=False),19 nn.BatchNorm2d(c_out))2021 def forward(self, x):22 return torch.relu(self.short(x) + self.branch(x))2324blk = ResNeXtBlock(256, 256)25convs = sum(m.weight.numel() for m in blk.modules() if isinstance(m, nn.Conv2d))26print(convs) # 701442728model = resnext50_32x4d(weights=ResNeXt50_32X4D_Weights.IMAGENET1K_V2).eval()29print(sum(p.numel() for p in model.parameters())) # 2502890430print(model.layer1[0].conv2) # Conv2d(128, 128, kernel_size=(3, 3), ..., groups=32, bias=False)| Model | Params | GFLOPs | acc@1 V1 | acc@1 V2 |
|---|---|---|---|---|
| resnet50 | 25.6M | 4.09 | 76.13% | 80.86% |
| resnext50_32x4d | 25.0M | 4.23 | 77.62% | 81.20% |
| resnext101_32x8d | 88.8M | 16.41 | 79.31% | 82.83% |
| resnext101_64x4d | 83.5M | 15.46 | – | 83.25% |
Use when
- +Better accuracy than ResNet at equal parameters and FLOPs (22.2% vs 23.9% top-1 for the 50-layer models)
- +Only two new hyperparameters, and ; the rest is inherited from ResNet
- +Drop-in: same stage layout, same shortcuts, a groups argument on one conv
- +Strong, widely available pretrained backbones for detection and segmentation
Avoid when
- −Grouped convs run slower than their FLOP count suggests on GPUs
- −The bottleneck is twice as wide, so it uses more activation memory than ResNet-50
- −No mixing across groups inside the 3×3; only the 1×1 layers mix
- −Superseded on accuracy per FLOP by EfficientNet, RegNet and ConvNeXt
Check yourself
A 3×3 convolution maps 128 channels to 128 channels with groups = 32. How many weights does it have (no bias)?
Check yourself
Why is form (a), C separate 1×1 → 3×3 → 1×1 paths summed, the same function as form (c) with a grouped 3×3?
Remember
- ✓ResNeXt block: with identical bottleneck paths; is the cardinality.
- ✓Implemented as 1×1 → grouped 3×3 (groups = C) → 1×1; the split-sum, concatenate and grouped forms are mathematically identical.
- ✓ResNeXt-50 32×4d: bottleneck width 128 with 4 channels per group, 25.0M params and 4.2 GFLOPs, 22.2% vs 23.9% top-1 for ResNet-50.
- ✓At a fixed budget, raising cardinality helped more than going deeper or wider in the paper's experiments.
- ✓Grouped convs save FLOPs but not proportionally time; check real throughput.