Definition
ConvNeXt is a family of pure convolutional networks whose block is : a depthwise convolution for spatial mixing, then a LayerNorm and an inverted-bottleneck MLP (, as convs) for channel mixing, with a residual connection. Stages are separated by LayerNorm + stride-2 convolutions, after a stride-4 "patchify" stem.
Liu, Mao, Wu, Feichtenhofer, Darrell and Xie (Meta AI and UC Berkeley) introduced ConvNeXt in "A ConvNet for the 2020s" (CVPR 2022). By 2021 hierarchical Transformers such as Swin Transformer had overtaken ResNets on ImageNet and especially on detection and segmentation, and the common explanation was self-attention. The paper tested that explanation directly. It started from a standard ResNet-50, changed one design decision at a time toward Swin-T, and measured ImageNet accuracy after each change. The end point contains no attention, yet it matches or beats Swin at the same cost.
The answer it gives is that much of the Transformer advantage came from the training recipe and from macro design choices (stage ratios, stem, large kernels, fewer normalizations and activations) that are not specific to attention. ConvNeXt-T reached 82.1% ImageNet-1K top-1 at 4.5 GFLOPs, against 81.3% for Swin-T, and the larger models kept pace with Swin on COCO and ADE20K.
The block
- depthwise conv, one filter per channel (): the only spatial operation, like attention in a Transformer block
- Layer normalization over the channels at each position (the only norm in the block)
- pointwise () layers forming the inverted bottleneck, like the Transformer MLP
- the only activation in the block
- layer scale, initialized to , so every block starts as almost the identity
Backward, the block Jacobian is . With at initialization the second term is negligible, so the gradient reaches early layers through the identity path almost unchanged, and (the upstream gradient times the branch output, summed over positions) lets each channel grow its branch only as fast as it helps. This, together with Stochastic depth, is what allows the 36-block ConvNeXt-B/L to train stably with AdamW.
The modernization ladder
Every number below is from the paper (Section 2, Figure 2): ImageNet-1K top-1 at and GFLOPs, after each change, applied cumulatively. Changes that did not help (for example, ReLU to GELU) were kept anyway when they made the design simpler or closer to the Transformer.
| step | change | top-1 | GFLOPs |
|---|---|---|---|
| 0 | ResNet-50, original 90-epoch training | 76.1% | 4.1 |
| 1 | training recipe: 300 epochs, AdamW, Mixup, CutMix, RandAugment, Random Erasing, Stochastic depth, Label smoothing | 78.8% | 4.1 |
| 2 | stage compute ratio (3,4,6,3) → (3,3,9,3), Swin's 1:1:3:1 | 79.4% | 4.5 |
| 3 | patchify stem: 7×7 s2 conv + max-pool → 4×4 conv, stride 4 | 79.5% | 4.4 |
| 4 | ResNeXt-ify: depthwise 3×3, width 64 → 96 | 80.5% | 5.3 |
| 5 | inverted bottleneck (hidden = 4× block width) | 80.6% | 4.6 |
| 6 | move the depthwise conv to the top of the block | 79.9% | 4.1 |
| 7 | kernel 3×3 → 7×7 | 80.6% | 4.2 |
| 8 | ReLU → GELU | 80.6% | 4.2 |
| 9 | one activation per block | 81.3% | 4.2 |
| 10 | one normalization per block | 81.4% | 4.2 |
| 11 | BatchNorm → LayerNorm | 81.5% | 4.2 |
| 12 | separate downsampling layers (LN + 2×2 conv s2) | 82.0% | 4.5 |
Why each step helps
The reasoning behind the main changes
- 1Recipe first. The modern recipe (borrowed from DeiT and Swin) lifts the unchanged ResNet-50 from 76.1% to 78.8%. Any CNN-vs-Transformer comparison that does not equalize the recipe is mostly a comparison of recipes.
- 2Stage ratio and stem. Swin-T puts most of its blocks in the stage (1:1:3:1). Moving ResNet-50 to (3,3,9,3) does the same. A non-overlapping stride-4 conv replaces the stem and max-pool, exactly like a ViT/Swin patch embedding with .
- 3Depthwise conv ≈ attention's mixing. A depthwise conv mixes only spatially, per channel; the convs mix only across channels. This is the same split as attention (spatial) and MLP (channel) in a Transformer. Being cheap, it lets the width grow to Swin's 96 channels.
- 4Inverted bottleneck and large kernels. Transformer MLPs are 4× wider than the block; so is the new hidden layer. Moving the depthwise conv before the expansion (as attention comes before the MLP) makes the large kernel affordable, and matches Swin's windows. The paper saw gains up to and saturation beyond.
- 5Fewer activations and norms. A Transformer block has one activation (in the MLP) and two norms; a ResNet block has three ReLUs and three BNs. Keeping only one GELU between the layers gave +0.7 points; one norm per block, then LN instead of BN, another +0.2.
- 6Separate downsampling. ResNet downsamples with a stride-2 conv inside the first block of a stage. ConvNeXt uses a dedicated stride-2 conv between stages, like Swin's patch merging; training diverged until LN layers were added before each downsampling, after the stem and after the final pooling.
The final ConvNeXt-T, trained with the full recipe, is reported at 82.1% in the paper's main results table (the ladder's last step shows 82.0%). torchvision's convnext_tiny weights, trained with torchvision's own recipe, list 82.52% acc@1, 28,589,128 parameters and 4.46 GFLOPS.
Worked example: counting ConvNeXt-T
| stage | map | C | blocks | params | MACs |
|---|---|---|---|---|---|
| stem 4×4 s4 + LN | 56×56 | 96 | – | 4,896 | 14.5M |
| 1 | 56×56 | 96 | 3 | 237,888 | 737.9M |
| 2 (LN + 2×2 s2) | 28×28 | 192 | 3 | 992,256 | 773.6M |
| 3 (LN + 2×2 s2) | 14×14 | 384 | 9 | 11,112,960 | 2,171.9M |
| 4 (LN + 2×2 s2) | 7×7 | 768 | 3 | 15,470,592 | 757.0M |
| LN + head | 1 | 768 | – | 770,536 | 0.8M |
One stage-3 block (d = 384, 14×14 map)
- 1Depthwise : parameters; M multiply-adds.
- 2LayerNorm ; : ; : ; layer scale . Block total 1,201,920 parameters.
- 3Pointwise multiply-adds: M, 63× the depthwise conv. The kernel is almost free; the cost is in the layers, as in a Transformer's MLP.
- 4Whole network: 28.59M parameters, 4.46G multiply-adds, of which the depthwise convs are 105M (2.4%). For comparison ResNet-50 (torchvision layout) is 25.56M and 4.09G.
1import torch, torch.nn as nn2from torchvision.models import convnext_tiny, ConvNeXt_Tiny_Weights34class Block(nn.Module):5 def __init__(self, d, layer_scale=1e-6):6 super().__init__()7 self.dw = nn.Conv2d(d, d, kernel_size=7, padding=3, groups=d) # depthwise 7x78 self.norm = nn.LayerNorm(d) # over channels, channels-last9 self.pw1 = nn.Linear(d, 4 * d) # 1x1 conv == Linear on channels-last10 self.act = nn.GELU()11 self.pw2 = nn.Linear(4 * d, d)12 self.gamma = nn.Parameter(layer_scale * torch.ones(d))1314 def forward(self, x): # x: (B, d, H, W)15 u = self.dw(x).permute(0, 2, 3, 1) # (B, H, W, d)16 u = self.pw2(self.act(self.pw1(self.norm(u))))17 return x + (self.gamma * u).permute(0, 3, 1, 2)1819blk = Block(384)20print(sum(p.numel() for p in blk.parameters())) # 120192021print(blk(torch.randn(1, 384, 14, 14)).shape) # torch.Size([1, 384, 14, 14])2223model = convnext_tiny(weights=ConvNeXt_Tiny_Weights.IMAGENET1K_V1).eval()24print(sum(p.numel() for p in model.parameters())) # 2858912825print(model.features[0]) # Conv2dNormActivation: Conv2d(3, 96, kernel_size=(4, 4), stride=(4, 4)) + LayerNorm2dUse when
- +Matches Swin at equal FLOPs on ImageNet, COCO and ADE20K with only standard conv layers
- +No attention, no position embeddings, no windows or masks: works at any resolution
- +Dense ops (depthwise + 1×1) are well supported by every framework and accelerator
- +Simple, uniform design; a strong default backbone
Avoid when
- −Depends on the modern recipe (AdamW, heavy augmentation, stochastic depth, layer scale, 300 epochs)
- −Depthwise 7×7 has low arithmetic intensity; it can be memory-bound and slower than its FLOPs suggest
- −Channels-last LN and permutes add overhead in some frameworks
- −Local receptive field: global interactions only through depth and downsampling
Check yourself
In the ConvNeXt modernization ladder, which single change gave the largest ImageNet gain?
Check yourself
A ConvNeXt block at width d = 384 on a 14×14 map: what share of its multiply-adds is the 7×7 depthwise conv?
Remember
- ✓Block: depthwise → LN → () → GELU → () → layer scale → + residual.
- ✓Built by modernizing ResNet-50 step by step toward Swin-T; every step's accuracy and GFLOPs are reported.
- ✓Recipe (+2.7) is the biggest step; stage ratio 1:1:3:1, patchify stem, depthwise conv, inverted bottleneck, large kernel, fewer activations/norms, LN and separate downsampling do the rest.
- ✓ConvNeXt-T: 28.6M params, 4.46 GMACs, 82.1% top-1 (paper), beating Swin-T's 81.3% with no attention.
- ✓The 7×7 kernel is cheap: depthwise convs are 2.4% of ConvNeXt-T's multiply-adds, the 1×1 layers about 93%.