# ConvNeXt

> A pure ConvNet built by modernizing ResNet-50 one Transformer-inspired step at a time: new training recipe, 1:1:3:1 stages, patchify stem, depthwise 7×7, inverted bottleneck, GELU, LayerNorm, fewer norms. ConvNeXt-T: 28M params, 4.5 GFLOPs, 82.1%.

Source: https://www.mlfoundry.in/learn/convnext  
Section: Deep Learning › Convolutional Neural Networks  
Level: advanced · 14 min read · updated 2026-10-11

**ConvNeXt:** is a family of pure convolutional networks whose block is $y=x+\gamma\odot W_2\,\mathrm{GELU}\big(W_1\,\mathrm{LN}(\mathrm{DWConv}_{7\times7}(x))\big)$: a $7\times7$ depthwise convolution for spatial mixing, then a LayerNorm and an inverted-bottleneck MLP ($d\to4d\to d$, as $1\times1$ convs) for channel mixing, with a residual connection. Stages are separated by LayerNorm + $2\times2$ stride-2 convolutions, after a $4\times4$ stride-4 "patchify" stem.

Liu, Mao, Wu, Feichtenhofer, Darrell and Xie (Meta AI and UC Berkeley) introduced ConvNeXt in "A ConvNet for the 2020s" (CVPR 2022). By 2021 hierarchical Transformers such as **Swin Transformer** had overtaken ResNets on ImageNet and especially on detection and segmentation, and the common explanation was self-attention. The paper tested that explanation directly. It started from a standard ResNet-50, changed one design decision at a time toward Swin-T, and measured ImageNet accuracy after each change. The end point contains no attention, yet it matches or beats Swin at the same cost.

The answer it gives is that much of the Transformer advantage came from the **training recipe** and from **macro design choices** (stage ratios, stem, large kernels, fewer normalizations and activations) that are not specific to attention. ConvNeXt-T reached 82.1% ImageNet-1K top-1 at 4.5 GFLOPs, against 81.3% for Swin-T, and the larger models kept pace with Swin on COCO and ADE20K.

## The block

$$
y=x+\gamma\odot\Big(W_2\,\mathrm{GELU}\big(W_1\,\mathrm{LN}(\mathrm{DW}_{7\times7}(x))+b_1\big)+b_2\Big)
$$

- $\mathrm{DW}_{7\times7}$: depthwise conv, one $7\times7$ filter per channel ($\mathtt{groups}=d$): the only spatial operation, like attention in a Transformer block
- $\mathrm{LN}$: **Layer normalization** over the $d$ channels at each position (the only norm in the block)
- $W_1\in\mathbb R^{4d\times d},\;W_2\in\mathbb R^{d\times4d}$: pointwise ($1\times1$) layers forming the inverted bottleneck, like the Transformer MLP
- $\mathrm{GELU}$: the only activation in the block
- $\gamma\in\mathbb R^{d}$: layer scale, initialized to $10^{-6}$, so every block starts as almost the identity

Backward, the block Jacobian is $\partial y/\partial x=I+\operatorname{diag}(\gamma)\,J_{\text{branch}}$. With $\gamma=10^{-6}$ at initialization the second term is negligible, so the gradient reaches early layers through the identity path almost unchanged, and $\partial L/\partial\gamma_c=\sum_p g_{p,c}\,u_{p,c}$ (the upstream gradient times the branch output, summed over positions) lets each channel grow its branch only as fast as it helps. This, together with **Stochastic depth**, is what allows the 36-block ConvNeXt-B/L to train stably with **AdamW**.

## The modernization ladder

Every number below is from the paper (Section 2, Figure 2): ImageNet-1K top-1 at $224^2$ and GFLOPs, after each change, applied cumulatively. Changes that did not help (for example, ReLU to GELU) were kept anyway when they made the design simpler or closer to the Transformer.

ResNet-50 → ConvNeXt-T, one design change at a time (Liu et al. 2022, Figure 2 and Section 2)

| step | change | top-1 | GFLOPs |
| --- | --- | --- | --- |
| 0 | ResNet-50, original 90-epoch training | 76.1% | 4.1 |
| 1 | training recipe: 300 epochs, AdamW, Mixup, CutMix, RandAugment, Random Erasing, Stochastic depth, Label smoothing | 78.8% | 4.1 |
| 2 | stage compute ratio (3,4,6,3) → (3,3,9,3), Swin's 1:1:3:1 | 79.4% | 4.5 |
| 3 | patchify stem: 7×7 s2 conv + max-pool → 4×4 conv, stride 4 | 79.5% | 4.4 |
| 4 | ResNeXt-ify: depthwise 3×3, width 64 → 96 | 80.5% | 5.3 |
| 5 | inverted bottleneck (hidden = 4× block width) | 80.6% | 4.6 |
| 6 | move the depthwise conv to the top of the block | 79.9% | 4.1 |
| 7 | kernel 3×3 → 7×7 | 80.6% | 4.2 |
| 8 | ReLU → GELU | 80.6% | 4.2 |
| 9 | one activation per block | 81.3% | 4.2 |
| 10 | one normalization per block | 81.4% | 4.2 |
| 11 | BatchNorm → LayerNorm | 81.5% | 4.2 |
| 12 | separate downsampling layers (LN + 2×2 conv s2) | 82.0% | 4.5 |

> Figure: Step through the ladder or click a step: the violet line is top-1 accuracy and the bars GFLOPs, both as reported in the paper, and the right panel redraws the residual block at that step. The two largest single gains are the training recipe (+2.7 points, step 1) and the switch to one activation per block (+0.7, step 9); moving the depthwise conv up costs 0.7 points until the 7×7 kernel wins it back. The bottom lines are exact counts from the figure's code: ResNet-50 25.56M / 4.09 GMACs, ConvNeXt-T 28.59M / 4.46 GMACs. Drag the kernel slider: going from 7×7 to 11×11 adds only 0.48M parameters, and the depthwise conv stays a small share of the multiply-adds (0.4% at 3×3, 2.4% at 7×7, 5.6% at 11×11).

## Why each step helps

**The reasoning behind the main changes**

1. **Recipe first.** The modern recipe (borrowed from DeiT and Swin) lifts the *unchanged* ResNet-50 from 76.1% to 78.8%. Any CNN-vs-Transformer comparison that does not equalize the recipe is mostly a comparison of recipes.
2. **Stage ratio and stem.** Swin-T puts most of its blocks in the $14\times14$ stage (1:1:3:1). Moving ResNet-50 to (3,3,9,3) does the same. A non-overlapping $4\times4$ stride-4 conv replaces the $7\times7$ stem and max-pool, exactly like a ViT/Swin patch embedding with $P=4$.
3. **Depthwise conv ≈ attention's mixing.** A depthwise conv mixes only spatially, per channel; the $1\times1$ convs mix only across channels. This is the same split as attention (spatial) and MLP (channel) in a Transformer. Being cheap, it lets the width grow to Swin's 96 channels.
4. **Inverted bottleneck and large kernels.** Transformer MLPs are 4× wider than the block; so is the new hidden layer. Moving the depthwise conv before the expansion (as attention comes before the MLP) makes the large kernel affordable, and $7\times7$ matches Swin's $7\times7$ windows. The paper saw gains up to $7\times7$ and saturation beyond.
5. **Fewer activations and norms.** A Transformer block has one activation (in the MLP) and two norms; a ResNet block has three ReLUs and three BNs. Keeping only one GELU between the $1\times1$ layers gave +0.7 points; one norm per block, then LN instead of BN, another +0.2.
6. **Separate downsampling.** ResNet downsamples with a stride-2 conv inside the first block of a stage. ConvNeXt uses a dedicated $2\times2$ stride-2 conv between stages, like Swin's patch merging; training diverged until LN layers were added before each downsampling, after the stem and after the final pooling.

The final ConvNeXt-T, trained with the full recipe, is reported at **82.1%** in the paper's main results table (the ladder's last step shows 82.0%). torchvision's `convnext_tiny` weights, trained with torchvision's own recipe, list 82.52% acc@1, 28,589,128 parameters and 4.46 GFLOPS.

## Worked example: counting ConvNeXt-T

ConvNeXt-T at 224², dims (96, 192, 384, 768), depths (3, 3, 9, 3); counted by the figure's code, total matches torchvision's 28,589,128

| stage | map | C | blocks | params | MACs |
| --- | --- | --- | --- | --- | --- |
| stem 4×4 s4 + LN | 56×56 | 96 | – | 4,896 | 14.5M |
| 1 | 56×56 | 96 | 3 | 237,888 | 737.9M |
| 2 (LN + 2×2 s2) | 28×28 | 192 | 3 | 992,256 | 773.6M |
| 3 (LN + 2×2 s2) | 14×14 | 384 | 9 | 11,112,960 | 2,171.9M |
| 4 (LN + 2×2 s2) | 7×7 | 768 | 3 | 15,470,592 | 757.0M |
| LN + head | 1 | 768 | – | 770,536 | 0.8M |

**One stage-3 block (d = 384, 14×14 map)**

1. Depthwise $7\times7$: $49\cdot384+384=19{,}200$ parameters; $196\cdot49\cdot384=3.69$M multiply-adds.
2. LayerNorm $2\cdot384=768$; $W_1$: $384\cdot1536+1536=591{,}360$; $W_2$: $1536\cdot384+384=590{,}208$; layer scale $384$. Block total **1,201,920** parameters.
3. Pointwise multiply-adds: $196\cdot8\cdot384^2=231.2$M, 63× the depthwise conv. The $7\times7$ kernel is almost free; the cost is in the $1\times1$ layers, as in a Transformer's MLP.
4. Whole network: 28.59M parameters, 4.46G multiply-adds, of which the depthwise convs are 105M (2.4%). For comparison ResNet-50 (torchvision layout) is 25.56M and 4.09G.

```python
import torch, torch.nn as nn
from torchvision.models import convnext_tiny, ConvNeXt_Tiny_Weights

class Block(nn.Module):
    def __init__(self, d, layer_scale=1e-6):
        super().__init__()
        self.dw = nn.Conv2d(d, d, kernel_size=7, padding=3, groups=d)  # depthwise 7x7
        self.norm = nn.LayerNorm(d)              # over channels, channels-last
        self.pw1 = nn.Linear(d, 4 * d)           # 1x1 conv == Linear on channels-last
        self.act = nn.GELU()
        self.pw2 = nn.Linear(4 * d, d)
        self.gamma = nn.Parameter(layer_scale * torch.ones(d))

    def forward(self, x):                        # x: (B, d, H, W)
        u = self.dw(x).permute(0, 2, 3, 1)       # (B, H, W, d)
        u = self.pw2(self.act(self.pw1(self.norm(u))))
        return x + (self.gamma * u).permute(0, 3, 1, 2)

blk = Block(384)
print(sum(p.numel() for p in blk.parameters()))  # 1201920
print(blk(torch.randn(1, 384, 14, 14)).shape)    # torch.Size([1, 384, 14, 14])

model = convnext_tiny(weights=ConvNeXt_Tiny_Weights.IMAGENET1K_V1).eval()
print(sum(p.numel() for p in model.parameters()))  # 28589128
print(model.features[0])   # Conv2dNormActivation: Conv2d(3, 96, kernel_size=(4, 4), stride=(4, 4)) + LayerNorm2d
```

A ConvNeXt block from scratch and torchvision's convnext_tiny

> **LayerNorm in a ConvNet needs care:** ConvNeXt's LN normalizes the $C$ channels at each pixel, not over $(C,H,W)$. `nn.LayerNorm(C)` does this only on channels-last tensors; on an NCHW tensor you need a permute (as above) or a custom "LayerNorm2d" (torchvision and timm both have one). Swapping BN for LN in a plain ResNet hurts; it only worked here after the other modernizations. And with no BatchNorm, ConvNeXt depends on the recipe (AdamW, layer scale, stochastic depth): trained with an old SGD recipe it loses much of its advantage.

> **What the ladder shows:** Almost half (2.7) of the 5.9-point gain from 76.1% to 82.0% comes from the training recipe alone, before any architecture change; the remaining 3.2 points are spread over eleven small design steps. The study is a reminder to compare architectures under the same recipe and compute. It also kept convolution's practical advantages: no position embeddings, any input size, and plain dense ops that run fast on every accelerator.

> **Variants:** ConvNeXt-T/S/B/L/XL use dims (96, 192, 384, 768) up to (256, 512, 1024, 2048) with depths (3,3,9,3) for T and (3,3,27,3) for the rest. The paper reports ImageNet-1K top-1 of 82.1 / 83.1 / 83.8 / 84.3% for T/S/B/L at $224^2$, and up to 87.8% for ConvNeXt-XL pre-trained on ImageNet-22K at $384^2$. **ConvNeXt V2** adds masked-autoencoder pre-training and Global Response Normalization.

**Strengths**

- Matches Swin at equal FLOPs on ImageNet, COCO and ADE20K with only standard conv layers
- No attention, no position embeddings, no windows or masks: works at any resolution
- Dense ops (depthwise + 1×1) are well supported by every framework and accelerator
- Simple, uniform design; a strong default backbone

**Limitations**

- Depends on the modern recipe (AdamW, heavy augmentation, stochastic depth, layer scale, 300 epochs)
- Depthwise 7×7 has low arithmetic intensity; it can be memory-bound and slower than its FLOPs suggest
- Channels-last LN and permutes add overhead in some frameworks
- Local receptive field: global interactions only through depth and downsampling

**Question:** In the ConvNeXt modernization ladder, which single change gave the largest ImageNet gain?

- A. The 7×7 depthwise kernel
- B. BatchNorm → LayerNorm
- C. The modern training recipe applied to the unchanged ResNet-50
- D. The patchify stem

**Answer:** C. The modern training recipe applied to the unchanged ResNet-50 — 76.1% → 78.8% (+2.7 points) with no architecture change: 300 epochs of AdamW with Mixup, CutMix, RandAugment, Random Erasing, stochastic depth and label smoothing. The next largest steps were ResNeXt-ification (+1.0, with more FLOPs) and one activation per block (+0.7).

**Question:** A ConvNeXt block at width d = 384 on a 14×14 map: what share of its multiply-adds is the 7×7 depthwise conv?

- A. About 1.6%
- B. About 25%
- C. About 50%
- D. About 90%

**Answer:** A. About 1.6% — Depthwise: $196\cdot49\cdot384=3.69$M. The two $1\times1$ layers: $196\cdot8\cdot384^2=231.2$M. So $3.69/(3.69+231.2)\approx1.6\%$. A depthwise conv costs $k^2$ per channel, a dense $1\times1$ costs $C$ per channel, and $C=384\gg49$.

**Key points**

- Block: depthwise $7\times7$ → LN → $1\times1$ ($d\to4d$) → GELU → $1\times1$ ($4d\to d$) → layer scale → + residual.
- Built by modernizing ResNet-50 step by step toward Swin-T; every step's accuracy and GFLOPs are reported.
- Recipe (+2.7) is the biggest step; stage ratio 1:1:3:1, patchify stem, depthwise conv, inverted bottleneck, large kernel, fewer activations/norms, LN and separate downsampling do the rest.
- ConvNeXt-T: 28.6M params, 4.46 GMACs, 82.1% top-1 (paper), beating Swin-T's 81.3% with no attention.
- The 7×7 kernel is cheap: depthwise convs are 2.4% of ConvNeXt-T's multiply-adds, the 1×1 layers about 93%.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
