# ResNeXt

> A ResNet whose bottleneck is split into C parallel narrow paths (a grouped 3×3 conv). At the same cost, raising this cardinality beats making the net deeper or wider: ResNeXt-50 32×4d has fewer parameters than ResNet-50 and 1.7 points lower top-1 error.

Source: https://www.mlfoundry.in/learn/resnext  
Section: Deep Learning › Convolutional Neural Networks  
Level: advanced · 12 min read · updated 2026-10-11

**ResNeXt:** is a residual network whose blocks compute $y=x+\sum_{i=1}^{C}\mathcal T_i(x)$: the bottleneck branch is split into $C$ identical, narrow transformations whose outputs are summed. The number of paths $C$ is called the **cardinality**. In practice the $C$ paths are implemented as a single **grouped convolution** with $C$ groups.

Xie, Girshick, Dollár, Tu and He introduced ResNeXt in "Aggregated Residual Transformations for Deep Neural Networks" (CVPR 2017). The paper's claim is that cardinality is a third way to scale a network, alongside depth and width, and that at a fixed budget of parameters and FLOPs it is the most effective of the three. A ResNeXt entry took second place in the ILSVRC 2016 classification task.

The design takes two ideas and keeps both. From VGG and **Residual neural network (ResNet)** it keeps the repetition of one block type, so there are very few hyperparameters. From Inception it keeps **split-transform-merge**: project the input into several low-dimensional embeddings, transform each one, and merge the results. Inception's branches are hand-designed and all different. ResNeXt makes every branch the same, so the whole block is described by just two numbers, the cardinality $C$ and the bottleneck width per path $d$.

**At a glance**

- **Paper:** Xie, Girshick, Dollár, Tu, He, CVPR 2017
- **Key idea:** cardinality $C$: identical parallel paths, summed (a grouped $3\times3$)
- **Template:** ResNeXt-50 **32×4d**: $C=32$ paths of $d=4$ channels in stage conv2 (width $Cd=128$)
- **Cost:** 25.0M params, 4.2G multiply-adds (ResNet-50: 25.6M, 4.1G)
- **Accuracy:** 22.2% vs 23.9% ImageNet top-1 error (paper, 224 single crop)
- **In PyTorch:** `nn.Conv2d(..., groups=C)`; `torchvision.models.resnext50_32x4d`

## Aggregated transformations

A single neuron already does split-transform-merge: $\sum_i w_ix_i$ splits $x$ into coordinates, scales each one and adds them up. ResNeXt keeps that shape but replaces each scalar product with a small network $\mathcal T_i$:

$$
y=x+\sum_{i=1}^{C}\mathcal T_i(x),\qquad \mathcal T_i(x)=W^{(3)}_i\,\sigma\!\Big(W^{(2)}_i * \sigma\big(W^{(1)}_i x\big)\Big)
$$

- $C$: cardinality: the number of parallel transformations (32 in ResNeXt-50 32×4d)
- $W^{(1)}_i$: $1\times1$ conv projecting the 256-channel input to $d$ channels (BN + ReLU follow)
- $W^{(2)}_i *$: $3\times3$ conv, $d\to d$ channels: the only spatial operation in the path
- $W^{(3)}_i$: $1\times1$ conv back to 256 channels; the path outputs are summed
- $\sigma$: BatchNorm then ReLU (the last BN comes before the addition, the last ReLU after it)

The network follows two rules taken from VGG and ResNet. Blocks that produce maps of the same size share the same $C$ and $d$. Each time the map is downsampled by 2, the width of the block is doubled. With those rules, choosing the conv2 template, for example "$C=32$, $d=4$", fixes the entire network. That is what the name **ResNeXt-50 (32×4d)** means: ResNet-50's stage layout of 3, 4, 6, 3 blocks, cardinality 32, 4 channels per path in the first stage.

## Three equivalent forms

The paper shows the block in three forms that compute the same function.

**Forms (a), (b), (c) of one ResNeXt block (BN omitted)**

1. **(a) Aggregated paths.** $C$ separate paths, each $1\times1$ (256→$d$) → $3\times3$ ($d$→$d$) → $1\times1$ ($d$→256), summed. This is the definition.
2. **(b) Early concatenation.** Summing the outputs of $C$ separate $1\times1$ convs is the same as concatenating their inputs and applying one $1\times1$ conv whose weight matrix is the $C$ blocks side by side: $\sum_i W^{(3)}_ih_i=[W^{(3)}_1\cdots W^{(3)}_C]\,[h_1;\dots;h_C]$.
3. **(c) Grouped convolution.** The $C$ first-layer $1\times1$ convs also read the same input, so they stack into one $1\times1$ conv 256→$Cd$. The $C$ independent $3\times3$ convs become one $3\times3$ conv with $\mathtt{groups}=C$: output channel group $i$ sees only input channel group $i$. That leaves 1×1 → grouped 3×3 → 1×1, a ResNet bottleneck with a wider middle and a sparser $3\times3$.

Form (c) is what every library implements, because one grouped conv runs as a single kernel launch. Two caveats: the equivalence needs depth ≥ 3 inside the block (with only two layers, the grouping collapses into a wide dense layer), and the BN after the first two layers is per channel, so it splits across paths without changing anything.

> Figure: Pick a C × d setting from the paper's Table 2 and a form. The left panel draws the conv2 template block. The bottom line runs forms (a) and (b) against (c) with the same random weights on a 16-channel 6×6 test input: at 32×4d the largest difference is about 1e−15, which is float64 round-off, so the three are the same function. The right panel counts the whole ImageNet network from its real configuration: 32×4d gives 25.03M parameters and 4.23G multiply-adds, against 25.56M and 4.09G for ResNet-50 (1×64d). Click a stage row to see its shapes, groups and counts. The paper's top-1 error falls from 23.9% at 1×64d to 22.2% at 32×4d, with the template block held at roughly 70k weights.

## Same budget, more paths

The conv weights of one template block, with 256 input and output channels, are $C\cdot(256d+9d^2+256d)$. Holding this at about 70k while increasing $C$ forces $d$ down, but the total bottleneck width $Cd$ grows, because the $3\times3$ term $9Cd^2$ shrinks as $d$ shrinks. The paper's Table 2 settings, with counts checked against the figure's code:

ResNeXt-50 settings at about the same complexity. Template = conv weights of one conv2 block. Whole-network params and multiply-adds counted from the torchvision layout (224×224 input). Top-1 error: Xie et al. 2017, Table 3, single 224 crop.

| C × d | width C·d | template weights | params | MACs | top-1 err. |
| --- | --- | --- | --- | --- | --- |
| 1 × 64d (ResNet-50) | 64 | 69,632 | 25.56M | 4.09G | 23.9% |
| 2 × 40d | 80 | 69,760 | 25.43M | 4.12G | 23.0% |
| 4 × 24d | 96 | 69,888 | 25.29M | 4.16G | 22.6% |
| 8 × 14d | 112 | 71,456 | 25.60M | 4.27G | 22.3% |
| 32 × 4d | 128 | 70,144 | 25.03M | 4.23G | 22.2% |

The gain is not overfitting relief: the paper reports lower **training** error for 32×4d as well, so the extra paths give a better-conditioned or more expressive block at the same cost. The same trend holds for ResNet-101 (22.0% → 21.2%). At twice the cost of ResNet-101, going deeper (ResNet-200, 21.7%) or wider (1×100d, 21.3%) helped less than going to 64×4d (20.4%).

> **4.1 vs 3.8 GFLOPs for ResNet-50:** The ResNet paper lists ResNet-50 at 3.8 GFLOPs; the ResNeXt paper and torchvision list about 4.1. The difference is where the stride sits in the first block of each stage. The original ResNet put it in the first $1\times1$ conv, while ResNeXt (and torchvision's "ResNet v1.5") put it in the $3\times3$, which runs the first $1\times1$ at full resolution. Counting both layouts gives 3.86G and 4.09G multiply-adds.

## Worked example: the 32×4d template block

**Counting conv2 of ResNeXt-50 (input 256×56×56)**

1. First $1\times1$: 256 → $Cd=128$ channels, $256\cdot128=32{,}768$ weights.
2. Grouped $3\times3$, 128 → 128 with 32 groups: each output channel sees $128/32=4$ input channels, so $128\cdot4\cdot9=4{,}608$ weights. A dense $3\times3$ at 128 channels would need $128\cdot128\cdot9=147{,}456$, 32× more.
3. Last $1\times1$: 128 → 256, another $32{,}768$. Total $70{,}144$ conv weights, versus $69{,}632$ for the ResNet-50 block, whose middle is a dense 64-channel $3\times3$ ($36{,}864$ weights).
4. Multiply-adds at $56\times56$: $70{,}144\cdot3136=220.0$M per block. ResNeXt spends its budget on the two cheap $1\times1$ layers and on a twice-as-wide (128 vs 64) representation, not on the $3\times3$ mixing.
5. Whole network: summing every block, the projections, the stem and the 2048→1000 classifier gives **25,028,904** parameters (BN included), exactly torchvision's count for `resnext50_32x4d`, and 4.23G multiply-adds.

## Gradients through a grouped convolution

In form (c) the $3\times3$ weight tensor is block-diagonal across groups. The Jacobian of the grouped layer with respect to its input is therefore block-diagonal too, and each group's weights receive gradient only from their own output channels:

$$
\frac{\partial\mathcal L}{\partial W^{(2)}_i}=\sum_{p}\frac{\partial\mathcal L}{\partial z_{i,p}}\;\mathrm{patch}_p\big(h_i\big),\qquad \frac{\partial\mathcal L}{\partial x}=\frac{\partial\mathcal L}{\partial y}\Big(I+\sum_{i=1}^{C}\frac{\partial\mathcal T_i}{\partial x}\Big)
$$

- $z_{i,p}$: output of group $i$ at spatial position $p$
- $\mathrm{patch}_p(h_i)$: the $3\times3\times d$ patch of group $i$'s input around $p$; group $j\ne i$ never appears
- $I$: identity shortcut: as in ResNet, the gradient reaches the block input unchanged
- $\sum_i \partial\mathcal T_i/\partial x$: the paths are summed forward, so their input gradients are summed backward

The paths only interact through the shared $1\times1$ layers before and after them. Information mixes across all 128 channels at the start and end of each block, and the $3\times3$ inside works on 32 independent 4-channel slices. Trained ResNeXts do learn different filters in different groups, an effect AlexNet first reported for its two GPU groups (one group learned colour-agnostic filters, the other colour-specific ones).

```python
import torch, torch.nn as nn
from torchvision.models import resnext50_32x4d, ResNeXt50_32X4D_Weights

class ResNeXtBlock(nn.Module):
    """Form (c): 1x1 -> grouped 3x3 -> 1x1, plus the residual shortcut."""
    def __init__(self, c_in, c_out, cardinality=32, d=4, stride=1):
        super().__init__()
        width = cardinality * d                      # 128 for the conv2 template
        self.branch = nn.Sequential(
            nn.Conv2d(c_in, width, 1, bias=False), nn.BatchNorm2d(width), nn.ReLU(inplace=True),
            nn.Conv2d(width, width, 3, stride=stride, padding=1,
                      groups=cardinality, bias=False),   # weight shape (128, 4, 3, 3)
            nn.BatchNorm2d(width), nn.ReLU(inplace=True),
            nn.Conv2d(width, c_out, 1, bias=False), nn.BatchNorm2d(c_out),
        )
        self.short = nn.Identity()
        if stride != 1 or c_in != c_out:
            self.short = nn.Sequential(nn.Conv2d(c_in, c_out, 1, stride=stride, bias=False),
                                       nn.BatchNorm2d(c_out))

    def forward(self, x):
        return torch.relu(self.short(x) + self.branch(x))

blk = ResNeXtBlock(256, 256)
convs = sum(m.weight.numel() for m in blk.modules() if isinstance(m, nn.Conv2d))
print(convs)                                     # 70144

model = resnext50_32x4d(weights=ResNeXt50_32X4D_Weights.IMAGENET1K_V2).eval()
print(sum(p.numel() for p in model.parameters()))  # 25028904
print(model.layer1[0].conv2)                     # Conv2d(128, 128, kernel_size=(3, 3), ..., groups=32, bias=False)
```

A ResNeXt block in form (c), the torchvision model, and a check of the parameter count

torchvision ImageNet weights (documented acc@1 at 224 crop; V1 = original recipe, V2 = torchvision's 2021 improved training recipe)

| Model | Params | GFLOPs | acc@1 V1 | acc@1 V2 |
| --- | --- | --- | --- | --- |
| resnet50 | 25.6M | 4.09 | 76.13% | 80.86% |
| resnext50_32x4d | 25.0M | 4.23 | 77.62% | 81.20% |
| resnext101_32x8d | 88.8M | 16.41 | 79.31% | 82.83% |
| resnext101_64x4d | 83.5M | 15.46 | – | 83.25% |

> **FLOPs are not latency:** A grouped $3\times3$ with 4 channels per group has very low arithmetic intensity: each weight is reused little, and the work splits into many small matrix products. On GPUs ResNeXt-50 typically runs noticeably slower than ResNet-50 despite almost equal FLOPs, and uses more activation memory because its bottleneck is twice as wide. Check measured throughput on your hardware before choosing it for speed. RegNet's later analysis found that group widths of 8 to 48 channels work better than 4 in practice.

> **Groups between dense and depthwise:** Groups interpolate between two familiar extremes. `groups=1` is an ordinary dense convolution. `groups=channels`, one channel per group, is a **depthwise convolution**, the spatial half of Xception's and MobileNet's **depthwise separable convolution**. ResNeXt sits in between and showed that the middle ground pays off. ShuffleNet then added a channel shuffle so that information can cross groups between grouped layers.

> **Where you meet it:** ResNeXt backbones were standard in detection (Mask R-CNN with ResNeXt-101 FPN) through the late 2010s. Mahajan et al. (2018) pretrained ResNeXt-101 32×48d on 3.5 billion Instagram images and reported 85.4% ImageNet top-1. RegNet treats the group width as a design-space parameter, and ConvNeXt starts its modernization of ResNet by "ResNeXt-ifying" it with depthwise convs.

**Strengths**

- Better accuracy than ResNet at equal parameters and FLOPs (22.2% vs 23.9% top-1 for the 50-layer models)
- Only two new hyperparameters, $C$ and $d$; the rest is inherited from ResNet
- Drop-in: same stage layout, same shortcuts, a groups argument on one conv
- Strong, widely available pretrained backbones for detection and segmentation

**Limitations**

- Grouped convs run slower than their FLOP count suggests on GPUs
- The bottleneck is twice as wide, so it uses more activation memory than ResNet-50
- No mixing across groups inside the 3×3; only the 1×1 layers mix
- Superseded on accuracy per FLOP by EfficientNet, RegNet and ConvNeXt

**Question:** A 3×3 convolution maps 128 channels to 128 channels with groups = 32. How many weights does it have (no bias)?

- A. 147,456
- B. 4,608
- C. 36,864
- D. 1,152

**Answer:** B. 4,608 — Each of the 128 output channels sees only its own group's $128/32=4$ input channels: $128\cdot4\cdot3\cdot3=4{,}608$. The dense version has $128\cdot128\cdot9=147{,}456$, exactly 32 times more. In general a grouped conv divides the weights and FLOPs by the number of groups.

**Question:** Why is form (a), C separate 1×1 → 3×3 → 1×1 paths summed, the same function as form (c) with a grouped 3×3?

- A. Because BatchNorm makes all paths identical
- B. The C first 1×1 convs read the same input so they stack into one; the C independent 3×3s are a grouped conv; summing C final 1×1s equals one 1×1 over their concatenation
- C. They are only approximately equal after training
- D. Because ReLU is linear on positive inputs

**Answer:** B. The C first 1×1 convs read the same input so they stack into one; the C independent 3×3s are a grouped conv; summing C final 1×1s equals one 1×1 over their concatenation — Each step is an exact rewrite: stacking weight matrices that share an input, block-diagonal (grouped) convolution for independent paths, and $\sum_iW_ih_i=[W_1\cdots W_C][h_1;\dots;h_C]$ for the final sum. The nonlinearities are elementwise, so they commute with the regrouping. The figure confirms it numerically to about $10^{-15}$.

**Key points**

- ResNeXt block: $y=x+\sum_{i=1}^{C}\mathcal T_i(x)$ with $C$ identical bottleneck paths; $C$ is the cardinality.
- Implemented as 1×1 → grouped 3×3 (groups = C) → 1×1; the split-sum, concatenate and grouped forms are mathematically identical.
- ResNeXt-50 32×4d: bottleneck width 128 with 4 channels per group, 25.0M params and 4.2 GFLOPs, 22.2% vs 23.9% top-1 for ResNet-50.
- At a fixed budget, raising cardinality helped more than going deeper or wider in the paper's experiments.
- Grouped convs save FLOPs but not proportionally time; check real throughput.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
