ConvNeXt V2

ConvNeXt co-designed with masked-autoencoder pre-training: FCMAE masks 60% of the image and encodes only the visible part with sparse convolutions, and Global Response Normalization makes channels compete, curing the feature collapse that held V1 back. Atto 3.7M → Huge 660M.

13 min read944 wordsadvanced0 viewsupdated 2026-10-11

Definition

ConvNeXt V2 is the ConvNeXt architecture with one new layer, Global Response Normalization (GRN), placed after the GELU in every block (layer scale is removed), pre-trained as a fully convolutional masked autoencoder (FCMAE): random 60% of the image is masked, the encoder sees only the visible part through sparse convolutions, and a light decoder reconstructs the masked pixels. After pre-training the model is fine-tuned with labels as usual.

Woo, Debnath, Hu, Chen, Liu, Kweon and Xie introduced it in "ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders" (CVPR 2023). Masked image modelling, especially He et al.'s MAE (2022), had become the strongest self-supervised pre-training for Vision Transformers (ViT). A ViT drops masked tokens trivially: it simply does not feed them to the encoder. A ConvNet cannot, because its sliding windows have no notion of "absent" inputs, and simply zero-filling masked patches lets every convolution see the mask boundaries and leak information.

The paper's point is that the architecture and the pre-training must be co-designed. FCMAE alone barely helped ConvNeXt V1: inspecting the trained V1 features, the authors found feature collapse, many channels in the wide MLP layer were dead, saturated or redundant copies of each other. GRN fixes that, and with both changes FCMAE pre-training improves every model size. ConvNeXt V2-Huge reached 88.9% ImageNet top-1 at 5122512^2 using only public data (ImageNet-22K intermediate fine-tuning), and the 3.7M-parameter Atto reached 76.7%.

Global Response Normalization

Gi=∥Xi∥2=(∑pXi,p2)1/2,Ni=Gi1C∑j=1CGj+ϵ,Yi=γi Xi Ni+βi+XiG_i=\lVert X_i\rVert_2=\Big(\sum_{p}X_{i,p}^2\Big)^{1/2},\qquad N_i=\frac{G_i}{\frac1C\sum_{j=1}^{C}G_j+\epsilon},\qquad Y_i=\gamma_i\,X_i\,N_i+\beta_i+X_i
XiX_i
channel ii of the block's 4d4d-wide hidden map (after GELU), over all spatial positions pp
GiG_i
global aggregation: the L2 energy of channel ii over the whole map, one number per channel
NiN_i
divisive normalization: energy relative to the channel mean, so ∑iNi=C\sum_iN_i=C; channels above average get Ni>1N_i>1
γi,βi\gamma_i,\beta_i
learned per-channel scale and shift, initialized to 0, so GRN starts as the identity
+Xi+X_i
residual inside the layer, which makes the identity initialization possible
ϵ\epsilon
10−610^{-6}

Each channel is rescaled by how strong it is relative to the other channels of the same image. This is the classic idea of lateral inhibition or divisive normalization: channels compete for a fixed total of response, ∑iNi=C\sum_iN_i=C. The paper argues that this competition is what counters collapse: a channel that merely duplicates the others gains nothing, and the learned γi\gamma_i (which can be negative) decide per channel whether to amplify or damp the contrast. It adds only 2⋅4d2\cdot4d parameters per block and negligible compute, and once GRN is present, layer scale is no longer needed.

∂Gi∂Xi,p=Xi,pGi,∂Nj∂Gi=δijGˉ−GjC Gˉ2,∂L∂γi=Ni∑pgi,pXi,p,∂L∂βi=∑pgi,p\frac{\partial G_i}{\partial X_{i,p}}=\frac{X_{i,p}}{G_i},\qquad \frac{\partial N_j}{\partial G_i}=\frac{\delta_{ij}}{\bar G}-\frac{G_j}{C\,\bar G^{2}},\qquad \frac{\partial L}{\partial\gamma_i}=N_i\sum_p g_{i,p}X_{i,p},\qquad \frac{\partial L}{\partial\beta_i}=\sum_p g_{i,p}
Gˉ\bar G
1C∑jGj\frac1C\sum_jG_j (ignoring ϵ\epsilon)
gi,pg_{i,p}
upstream gradient ∂L/∂Yi,p\partial L/\partial Y_{i,p}
δij\delta_{ij}
1 if i=ji=j, else 0: raising one channel's energy raises its own NiN_i and lowers every other NjN_j

At initialization γ=0\gamma=0, so ∂Y/∂X=I\partial Y/\partial X=I and the network trains like V1. The first updates to γi\gamma_i are proportional to NiN_i: channels that are already strong relative to the others start with the largest calibration gradients.

FCMAE: masking a ConvNet

FCMAE masks a random 60% of the 32×3232\times32 patches of a 2242224^2 image. 32 is the total stride of ConvNeXt, so each masked patch is one position of the 7×77\times7 last-stage map, and the same mask, upsampled, applies at every stage. The encoder is the ConvNeXt V2 backbone with sparse convolutions (the paper uses the MinkowskiEngine library): it stores and computes only the visible sites, and its kernels read only visible inputs, so the masked area is never touched. The decoder is a single ConvNeXt block of width 512; the loss is the mean squared error between the reconstructed and target pixels, on the masked patches only. After pre-training the sparse layers are converted back to ordinary dense convolutions.

Sparse convolution is equivalent to a dense convolution with the mask applied to its input and output, which is how it can be emulated in plain PyTorch. Without that, a dense 7×77\times7 depthwise conv on the zero-filled image spreads visible values 3 positions into the masked area per layer, and lets visible positions near the border "see" the zeros, so the encoder learns the mask pattern instead of the image.

FCMAE mask on the 56×56 stage-1 map29/49 32px patches masked (59.2%)dense 7×7 dw conv ×1, zeros inmasked sites now ≠ 0: 1062 / 1856visible sites seeing zeros: 918sparse MACs = 40.8% of dense■ visible ■ leakedGRN on 12 channels: Y = γ·X·N + β + X, γ = 1.00, β = 00.111010010002.652.251.631.451.050.930.750.530.390.230.090.05Nᵢ■ Gᵢ = ‖Xᵢ‖ ■ ‖Yᵢ‖ (Nᵢ ≥ 1) ■ ‖Yᵢ‖ (Nᵢ < 1), logXYeach map scaled to its own row maximummax/min channel norm: 58.2 → 202.9energy in top 3 channels: 74.1% → 87.6%ΣNᵢ = 12.00 = C: N only redistributes
conv layers
fig.Left: the FCMAE mask on the 56×56 stage-1 map. With the first mask at ratio 0.60, 29 of the 49 patches (59.2%) are masked. One dense 7×7 depthwise conv on the zero-filled input makes 1,062 of the 1,856 masked sites nonzero (57.2%) and lets 918 of the 1,280 visible sites read masked zeros; two layers leak into 1,640 masked sites and three into all of them. Switch to sparse conv: nothing leaks, and the encoder computes only 40.8% of the dense multiply-adds. Right: GRN on a 12-channel, 8×8 synthetic map, with one shared γ for all channels (in the model γ is learned per channel). The labels under the bars are Nᵢ; they always sum to 12. At γ = 1 the strongest/weakest channel-norm ratio rises from 58.2 to 202.9 and the top-3 channels' share of energy from 74.1% to 87.6%; a negative γ = −0.5 does the opposite (ratio 19.2, share 59.1%).

Worked example: GRN on 3 channels of a 2×2 map, γ = 1, β = 0

  1. 1Channels X1=(3,4,0,0)X_1=(3,4,0,0), X2=(1,0,0,0)X_2=(1,0,0,0), X3=(2,2,1,0)X_3=(2,2,1,0). Energies G=(25,1,9)=(5, 1, 3)G=(\sqrt{25},\sqrt1,\sqrt9)=(5,\,1,\,3).
  2. 2Mean energy Gˉ=3\bar G=3, so N=(5/3, 1/3, 1)=(1.667, 0.333, 1.000)N=(5/3,\,1/3,\,1)=(1.667,\,0.333,\,1.000), summing to C=3C=3.
  3. 3Yi=Xi(1+γNi)Y_i=X_i(1+\gamma N_i): channel 1 is scaled by 2.667, channel 2 by 1.333, channel 3 by 2. So Y1=(8, 10.667, 0, 0)Y_1=(8,\,10.667,\,0,\,0), Y2=(1.333, 0, 0, 0)Y_2=(1.333,\,0,\,0,\,0), Y3=(4, 4, 2, 0)Y_3=(4,\,4,\,2,\,0).
  4. 4Output norms (13.33, 1.33, 6.00)(13.33,\,1.33,\,6.00): the strong-to-weak ratio went from 5 to 10. With γ=0.5\gamma=0.5 the scales are (1.833, 1.167, 1.5)(1.833,\,1.167,\,1.5) and the norms (9.17, 1.17, 4.50)(9.17,\,1.17,\,4.50). Values checked with the figure's grn function.

Model family

modeldimsdepthsparamsMACstop-1
Atto40-80-160-3202-2-6-23.7M0.55G76.7%
Femto48-96-192-3842-2-6-25.2M0.78G78.5%
Pico64-128-256-5122-2-6-29.1M1.37G80.3%
Nano80-160-320-6402-2-8-215.6M2.45G81.9%
Tiny96-192-384-7683-3-9-328.6M4.46G83.0%
Base128-256-512-10243-3-27-388.7M15.35G84.9%
Large192-384-768-15363-3-27-3198.0M34.36G85.8%
Huge352-704-1408-28163-3-27-3660.3M114.9G86.3%
ConvNeXt V2 sizes. Params and MACs at 224² counted by this module's code (GRN, no layer scale); they match the paper's Table (3.7M/0.55G … 660M/115G). Top-1: official repository, FCMAE pre-training + ImageNet-1K fine-tuning, 224².

A V2 block has 7d7d more parameters than a V1 block (2⋅4d2\cdot4d for GRN minus dd for layer scale): ConvNeXt V2-Tiny has 28,635,496 parameters against 28,589,128 for V1-Tiny. The multiply-add count is unchanged except for GRN's elementwise work, which this count (convolutions and linear layers only) omits; the paper lists Tiny at 4.47G.

python
1import torch, torch.nn as nn, torch.nn.functional as F
2import timm
3
4class GRN(nn.Module):
5 """Global Response Normalization; input (B, H, W, C)."""
6 def __init__(self, dim):
7 super().__init__()
8 self.gamma = nn.Parameter(torch.zeros(1, 1, 1, dim))
9 self.beta = nn.Parameter(torch.zeros(1, 1, 1, dim))
10
11 def forward(self, x):
12 Gx = torch.norm(x, p=2, dim=(1, 2), keepdim=True) # (B, 1, 1, C)
13 Nx = Gx / (Gx.mean(dim=-1, keepdim=True) + 1e-6) # relative energy
14 return self.gamma * (x * Nx) + self.beta + x
15
16x = torch.tensor([[3., 4, 0, 0], [1, 0, 0, 0], [2, 2, 1, 0]]).T.reshape(1, 2, 2, 3)
17grn = GRN(3); nn.init.ones_(grn.gamma)
18y = grn(x)
19print(y.flatten(1, 2).norm(dim=1).detach()) # tensor([[13.3333, 1.3333, 6.0000]])
20
21# FCMAE masking: 60% of the 7x7 grid of 32x32 patches, upsampled to the 56x56 stage-1 map
22B = 1
23keep = (torch.rand(B, 49).argsort(1).argsort(1) >= int(0.6 * 49)).float() # 1 = visible
24m = keep.view(B, 1, 7, 7)
25m = F.interpolate(m, scale_factor=8, mode="nearest") # (B, 1, 56, 56)
26
27dw = nn.Conv2d(96, 96, 7, padding=3, groups=96)
28h = torch.randn(B, 96, 56, 56) * m
29h_sparse = dw(h * m) * m # dense conv emulating a sparse one: masked sites stay 0
30
31# Pretrained models (FCMAE pre-trained, ImageNet-1K fine-tuned)
32model = timm.create_model("convnextv2_tiny.fcmae_ft_in1k", pretrained=True).eval()
33print(sum(p.numel() for p in model.parameters())) # 28635496
GRN (channels-last, as in the official code), FCMAE-style masked convolution, and the pretrained models in timm

Use when

  • +Self-supervised pre-training that works for ConvNets, improving every size from Atto to Huge
  • +GRN is tiny (2 vectors per block), negligible in compute, and starts as the identity
  • +Sparse encoder computes only on the visible 40% during pre-training
  • +One architecture covers 3.7M to 660M parameters; 88.9% top-1 at the top with public data

Avoid when

  • −Sparse convolution needs a special library (MinkowskiEngine) or masked dense emulation, which saves no compute
  • −Pre-training (FCMAE) plus fine-tuning is far more expensive than supervised training alone
  • −GRN's global pooling couples all positions of an image; it behaves differently on crops vs full images
  • −Gains over V1 come mainly with FCMAE; supervised-only, the improvement is small

Check yourself

In GRN, a channel's energy Gᵢ equals the mean energy of all channels. With γᵢ = 1 and βᵢ = 0, how is that channel changed?

Check yourself

Why does FCMAE use sparse convolution in the encoder instead of simply zeroing the masked patches?

Remember

  • ✓GRN: Gi=∥Xi∥2G_i=\lVert X_i\rVert_2, Ni=Gi/GˉN_i=G_i/\bar G, Yi=γiXiNi+βi+XiY_i=\gamma_iX_iN_i+\beta_i+X_i, with γ=β=0\gamma=\beta=0 at init.
  • ✓GRN makes channels compete for a fixed total response and counters the feature collapse seen in V1 under masked pre-training; layer scale is dropped.
  • ✓FCMAE: mask 60% of 32×3232\times32 patches, sparse-conv encoder on the visible part, one-block decoder, MSE on masked patches.
  • ✓Dense convs on zero-filled inputs leak across mask borders; sparse convs (or input+output masking) prevent it.
  • ✓Sizes Atto (3.7M, 76.7%) to Huge (660M, 88.9% at 512² with ImageNet-22K).
← back to Deep Learning roadmap