ConvNeXt

A pure ConvNet built by modernizing ResNet-50 one Transformer-inspired step at a time: new training recipe, 1:1:3:1 stages, patchify stem, depthwise 7×7, inverted bottleneck, GELU, LayerNorm, fewer norms. ConvNeXt-T: 28M params, 4.5 GFLOPs, 82.1%.

14 min read996 wordsadvanced0 viewsupdated 2026-10-11

Definition

ConvNeXt is a family of pure convolutional networks whose block is y=x+γ⊙W2 GELU(W1 LN(DWConv7×7(x)))y=x+\gamma\odot W_2\,\mathrm{GELU}\big(W_1\,\mathrm{LN}(\mathrm{DWConv}_{7\times7}(x))\big): a 7×77\times7 depthwise convolution for spatial mixing, then a LayerNorm and an inverted-bottleneck MLP (d→4d→dd\to4d\to d, as 1×11\times1 convs) for channel mixing, with a residual connection. Stages are separated by LayerNorm + 2×22\times2 stride-2 convolutions, after a 4×44\times4 stride-4 "patchify" stem.

Liu, Mao, Wu, Feichtenhofer, Darrell and Xie (Meta AI and UC Berkeley) introduced ConvNeXt in "A ConvNet for the 2020s" (CVPR 2022). By 2021 hierarchical Transformers such as Swin Transformer had overtaken ResNets on ImageNet and especially on detection and segmentation, and the common explanation was self-attention. The paper tested that explanation directly. It started from a standard ResNet-50, changed one design decision at a time toward Swin-T, and measured ImageNet accuracy after each change. The end point contains no attention, yet it matches or beats Swin at the same cost.

The answer it gives is that much of the Transformer advantage came from the training recipe and from macro design choices (stage ratios, stem, large kernels, fewer normalizations and activations) that are not specific to attention. ConvNeXt-T reached 82.1% ImageNet-1K top-1 at 4.5 GFLOPs, against 81.3% for Swin-T, and the larger models kept pace with Swin on COCO and ADE20K.

The block

y=x+γ⊙(W2 GELU(W1 LN(DW7×7(x))+b1)+b2)y=x+\gamma\odot\Big(W_2\,\mathrm{GELU}\big(W_1\,\mathrm{LN}(\mathrm{DW}_{7\times7}(x))+b_1\big)+b_2\Big)
DW7×7\mathrm{DW}_{7\times7}
depthwise conv, one 7×77\times7 filter per channel (groups=d\mathtt{groups}=d): the only spatial operation, like attention in a Transformer block
LN\mathrm{LN}
Layer normalization over the dd channels at each position (the only norm in the block)
W1∈R4d×d,  W2∈Rd×4dW_1\in\mathbb R^{4d\times d},\;W_2\in\mathbb R^{d\times4d}
pointwise (1×11\times1) layers forming the inverted bottleneck, like the Transformer MLP
GELU\mathrm{GELU}
the only activation in the block
γ∈Rd\gamma\in\mathbb R^{d}
layer scale, initialized to 10−610^{-6}, so every block starts as almost the identity

Backward, the block Jacobian is ∂y/∂x=I+diag⁡(γ) Jbranch\partial y/\partial x=I+\operatorname{diag}(\gamma)\,J_{\text{branch}}. With γ=10−6\gamma=10^{-6} at initialization the second term is negligible, so the gradient reaches early layers through the identity path almost unchanged, and ∂L/∂γc=∑pgp,c up,c\partial L/\partial\gamma_c=\sum_p g_{p,c}\,u_{p,c} (the upstream gradient times the branch output, summed over positions) lets each channel grow its branch only as fast as it helps. This, together with Stochastic depth, is what allows the 36-block ConvNeXt-B/L to train stably with AdamW.

The modernization ladder

Every number below is from the paper (Section 2, Figure 2): ImageNet-1K top-1 at 2242224^2 and GFLOPs, after each change, applied cumulatively. Changes that did not help (for example, ReLU to GELU) were kept anyway when they made the design simpler or closer to the Transformer.

stepchangetop-1GFLOPs
0ResNet-50, original 90-epoch training76.1%4.1
1training recipe: 300 epochs, AdamW, Mixup, CutMix, RandAugment, Random Erasing, Stochastic depth, Label smoothing78.8%4.1
2stage compute ratio (3,4,6,3) → (3,3,9,3), Swin's 1:1:3:179.4%4.5
3patchify stem: 7×7 s2 conv + max-pool → 4×4 conv, stride 479.5%4.4
4ResNeXt-ify: depthwise 3×3, width 64 → 9680.5%5.3
5inverted bottleneck (hidden = 4× block width)80.6%4.6
6move the depthwise conv to the top of the block79.9%4.1
7kernel 3×3 → 7×780.6%4.2
8ReLU → GELU80.6%4.2
9one activation per block81.3%4.2
10one normalization per block81.4%4.2
11BatchNorm → LayerNorm81.5%4.2
12separate downsampling layers (LN + 2×2 conv s2)82.0%4.5
ResNet-50 → ConvNeXt-T, one design change at a time (Liu et al. 2022, Figure 2 and Section 2)
ImageNet-1K top-1 (%) after each change, values from Liu et al. 202276788082Swin-T 81.3% · 4.5G4.104.114.524.435.344.654.164.274.284.294.2104.21182.04.512GFLOPs12. separate downsampling: 82.0% (+0.5), 4.5 GFLOPs
2×2 s2 conv between stages, with LN before it, after the stem and before pooling
block at step 127×7 depthwise dLN1×1 conv d→4dGELU1×1 conv 4d→dlayer scale γ(+ shortcut)
stem: 4×4 conv s4 (patchify) + LN
stages: 3 : 3 : 9 : 3, d₁ = 96
downsample: separate LN + 2×2 conv s2 between stages
exact counts (this page's code):ResNet-50 25.56M · 4.09 GMACsConvNeXt-T k=7: 28.59M · 4.46G7×7 depthwise = 2.4% of MACs
click a step on the chart
fig.Step through the ladder or click a step: the violet line is top-1 accuracy and the bars GFLOPs, both as reported in the paper, and the right panel redraws the residual block at that step. The two largest single gains are the training recipe (+2.7 points, step 1) and the switch to one activation per block (+0.7, step 9); moving the depthwise conv up costs 0.7 points until the 7×7 kernel wins it back. The bottom lines are exact counts from the figure's code: ResNet-50 25.56M / 4.09 GMACs, ConvNeXt-T 28.59M / 4.46 GMACs. Drag the kernel slider: going from 7×7 to 11×11 adds only 0.48M parameters, and the depthwise conv stays a small share of the multiply-adds (0.4% at 3×3, 2.4% at 7×7, 5.6% at 11×11).

Why each step helps

The reasoning behind the main changes

  1. 1Recipe first. The modern recipe (borrowed from DeiT and Swin) lifts the unchanged ResNet-50 from 76.1% to 78.8%. Any CNN-vs-Transformer comparison that does not equalize the recipe is mostly a comparison of recipes.
  2. 2Stage ratio and stem. Swin-T puts most of its blocks in the 14×1414\times14 stage (1:1:3:1). Moving ResNet-50 to (3,3,9,3) does the same. A non-overlapping 4×44\times4 stride-4 conv replaces the 7×77\times7 stem and max-pool, exactly like a ViT/Swin patch embedding with P=4P=4.
  3. 3Depthwise conv ≈ attention's mixing. A depthwise conv mixes only spatially, per channel; the 1×11\times1 convs mix only across channels. This is the same split as attention (spatial) and MLP (channel) in a Transformer. Being cheap, it lets the width grow to Swin's 96 channels.
  4. 4Inverted bottleneck and large kernels. Transformer MLPs are 4× wider than the block; so is the new hidden layer. Moving the depthwise conv before the expansion (as attention comes before the MLP) makes the large kernel affordable, and 7×77\times7 matches Swin's 7×77\times7 windows. The paper saw gains up to 7×77\times7 and saturation beyond.
  5. 5Fewer activations and norms. A Transformer block has one activation (in the MLP) and two norms; a ResNet block has three ReLUs and three BNs. Keeping only one GELU between the 1×11\times1 layers gave +0.7 points; one norm per block, then LN instead of BN, another +0.2.
  6. 6Separate downsampling. ResNet downsamples with a stride-2 conv inside the first block of a stage. ConvNeXt uses a dedicated 2×22\times2 stride-2 conv between stages, like Swin's patch merging; training diverged until LN layers were added before each downsampling, after the stem and after the final pooling.

The final ConvNeXt-T, trained with the full recipe, is reported at 82.1% in the paper's main results table (the ladder's last step shows 82.0%). torchvision's convnext_tiny weights, trained with torchvision's own recipe, list 82.52% acc@1, 28,589,128 parameters and 4.46 GFLOPS.

Worked example: counting ConvNeXt-T

stagemapCblocksparamsMACs
stem 4×4 s4 + LN56×5696–4,89614.5M
156×56963237,888737.9M
2 (LN + 2×2 s2)28×281923992,256773.6M
3 (LN + 2×2 s2)14×14384911,112,9602,171.9M
4 (LN + 2×2 s2)7×7768315,470,592757.0M
LN + head1768–770,5360.8M
ConvNeXt-T at 224², dims (96, 192, 384, 768), depths (3, 3, 9, 3); counted by the figure's code, total matches torchvision's 28,589,128

One stage-3 block (d = 384, 14×14 map)

  1. 1Depthwise 7×77\times7: 49⋅384+384=19,20049\cdot384+384=19{,}200 parameters; 196⋅49⋅384=3.69196\cdot49\cdot384=3.69M multiply-adds.
  2. 2LayerNorm 2⋅384=7682\cdot384=768; W1W_1: 384⋅1536+1536=591,360384\cdot1536+1536=591{,}360; W2W_2: 1536⋅384+384=590,2081536\cdot384+384=590{,}208; layer scale 384384. Block total 1,201,920 parameters.
  3. 3Pointwise multiply-adds: 196⋅8⋅3842=231.2196\cdot8\cdot384^2=231.2M, 63× the depthwise conv. The 7×77\times7 kernel is almost free; the cost is in the 1×11\times1 layers, as in a Transformer's MLP.
  4. 4Whole network: 28.59M parameters, 4.46G multiply-adds, of which the depthwise convs are 105M (2.4%). For comparison ResNet-50 (torchvision layout) is 25.56M and 4.09G.
python
1import torch, torch.nn as nn
2from torchvision.models import convnext_tiny, ConvNeXt_Tiny_Weights
3
4class Block(nn.Module):
5 def __init__(self, d, layer_scale=1e-6):
6 super().__init__()
7 self.dw = nn.Conv2d(d, d, kernel_size=7, padding=3, groups=d) # depthwise 7x7
8 self.norm = nn.LayerNorm(d) # over channels, channels-last
9 self.pw1 = nn.Linear(d, 4 * d) # 1x1 conv == Linear on channels-last
10 self.act = nn.GELU()
11 self.pw2 = nn.Linear(4 * d, d)
12 self.gamma = nn.Parameter(layer_scale * torch.ones(d))
13
14 def forward(self, x): # x: (B, d, H, W)
15 u = self.dw(x).permute(0, 2, 3, 1) # (B, H, W, d)
16 u = self.pw2(self.act(self.pw1(self.norm(u))))
17 return x + (self.gamma * u).permute(0, 3, 1, 2)
18
19blk = Block(384)
20print(sum(p.numel() for p in blk.parameters())) # 1201920
21print(blk(torch.randn(1, 384, 14, 14)).shape) # torch.Size([1, 384, 14, 14])
22
23model = convnext_tiny(weights=ConvNeXt_Tiny_Weights.IMAGENET1K_V1).eval()
24print(sum(p.numel() for p in model.parameters())) # 28589128
25print(model.features[0]) # Conv2dNormActivation: Conv2d(3, 96, kernel_size=(4, 4), stride=(4, 4)) + LayerNorm2d
A ConvNeXt block from scratch and torchvision's convnext_tiny

Use when

  • +Matches Swin at equal FLOPs on ImageNet, COCO and ADE20K with only standard conv layers
  • +No attention, no position embeddings, no windows or masks: works at any resolution
  • +Dense ops (depthwise + 1×1) are well supported by every framework and accelerator
  • +Simple, uniform design; a strong default backbone

Avoid when

  • −Depends on the modern recipe (AdamW, heavy augmentation, stochastic depth, layer scale, 300 epochs)
  • −Depthwise 7×7 has low arithmetic intensity; it can be memory-bound and slower than its FLOPs suggest
  • −Channels-last LN and permutes add overhead in some frameworks
  • −Local receptive field: global interactions only through depth and downsampling

Check yourself

In the ConvNeXt modernization ladder, which single change gave the largest ImageNet gain?

Check yourself

A ConvNeXt block at width d = 384 on a 14×14 map: what share of its multiply-adds is the 7×7 depthwise conv?

Remember

  • ✓Block: depthwise 7×77\times7 → LN → 1×11\times1 (d→4dd\to4d) → GELU → 1×11\times1 (4d→d4d\to d) → layer scale → + residual.
  • ✓Built by modernizing ResNet-50 step by step toward Swin-T; every step's accuracy and GFLOPs are reported.
  • ✓Recipe (+2.7) is the biggest step; stage ratio 1:1:3:1, patchify stem, depthwise conv, inverted bottleneck, large kernel, fewer activations/norms, LN and separate downsampling do the rest.
  • ✓ConvNeXt-T: 28.6M params, 4.46 GMACs, 82.1% top-1 (paper), beating Swin-T's 81.3% with no attention.
  • ✓The 7×7 kernel is cheap: depthwise convs are 2.4% of ConvNeXt-T's multiply-adds, the 1×1 layers about 93%.
← back to Deep Learning roadmap