Definition
ConvNeXt V2 is the ConvNeXt architecture with one new layer, Global Response Normalization (GRN), placed after the GELU in every block (layer scale is removed), pre-trained as a fully convolutional masked autoencoder (FCMAE): random 60% of the image is masked, the encoder sees only the visible part through sparse convolutions, and a light decoder reconstructs the masked pixels. After pre-training the model is fine-tuned with labels as usual.
Woo, Debnath, Hu, Chen, Liu, Kweon and Xie introduced it in "ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders" (CVPR 2023). Masked image modelling, especially He et al.'s MAE (2022), had become the strongest self-supervised pre-training for Vision Transformers (ViT). A ViT drops masked tokens trivially: it simply does not feed them to the encoder. A ConvNet cannot, because its sliding windows have no notion of "absent" inputs, and simply zero-filling masked patches lets every convolution see the mask boundaries and leak information.
The paper's point is that the architecture and the pre-training must be co-designed. FCMAE alone barely helped ConvNeXt V1: inspecting the trained V1 features, the authors found feature collapse, many channels in the wide MLP layer were dead, saturated or redundant copies of each other. GRN fixes that, and with both changes FCMAE pre-training improves every model size. ConvNeXt V2-Huge reached 88.9% ImageNet top-1 at using only public data (ImageNet-22K intermediate fine-tuning), and the 3.7M-parameter Atto reached 76.7%.
Global Response Normalization
- channel of the block's -wide hidden map (after GELU), over all spatial positions
- global aggregation: the L2 energy of channel over the whole map, one number per channel
- divisive normalization: energy relative to the channel mean, so ; channels above average get
- learned per-channel scale and shift, initialized to 0, so GRN starts as the identity
- residual inside the layer, which makes the identity initialization possible
Each channel is rescaled by how strong it is relative to the other channels of the same image. This is the classic idea of lateral inhibition or divisive normalization: channels compete for a fixed total of response, . The paper argues that this competition is what counters collapse: a channel that merely duplicates the others gains nothing, and the learned (which can be negative) decide per channel whether to amplify or damp the contrast. It adds only parameters per block and negligible compute, and once GRN is present, layer scale is no longer needed.
- (ignoring )
- upstream gradient
- 1 if , else 0: raising one channel's energy raises its own and lowers every other
At initialization , so and the network trains like V1. The first updates to are proportional to : channels that are already strong relative to the others start with the largest calibration gradients.
FCMAE: masking a ConvNet
FCMAE masks a random 60% of the patches of a image. 32 is the total stride of ConvNeXt, so each masked patch is one position of the last-stage map, and the same mask, upsampled, applies at every stage. The encoder is the ConvNeXt V2 backbone with sparse convolutions (the paper uses the MinkowskiEngine library): it stores and computes only the visible sites, and its kernels read only visible inputs, so the masked area is never touched. The decoder is a single ConvNeXt block of width 512; the loss is the mean squared error between the reconstructed and target pixels, on the masked patches only. After pre-training the sparse layers are converted back to ordinary dense convolutions.
Sparse convolution is equivalent to a dense convolution with the mask applied to its input and output, which is how it can be emulated in plain PyTorch. Without that, a dense depthwise conv on the zero-filled image spreads visible values 3 positions into the masked area per layer, and lets visible positions near the border "see" the zeros, so the encoder learns the mask pattern instead of the image.
Worked example: GRN on 3 channels of a 2×2 map, γ = 1, β = 0
- 1Channels , , . Energies .
- 2Mean energy , so , summing to .
- 3: channel 1 is scaled by 2.667, channel 2 by 1.333, channel 3 by 2. So , , .
- 4Output norms : the strong-to-weak ratio went from 5 to 10. With the scales are and the norms . Values checked with the figure's
grnfunction.
Model family
| model | dims | depths | params | MACs | top-1 |
|---|---|---|---|---|---|
| Atto | 40-80-160-320 | 2-2-6-2 | 3.7M | 0.55G | 76.7% |
| Femto | 48-96-192-384 | 2-2-6-2 | 5.2M | 0.78G | 78.5% |
| Pico | 64-128-256-512 | 2-2-6-2 | 9.1M | 1.37G | 80.3% |
| Nano | 80-160-320-640 | 2-2-8-2 | 15.6M | 2.45G | 81.9% |
| Tiny | 96-192-384-768 | 3-3-9-3 | 28.6M | 4.46G | 83.0% |
| Base | 128-256-512-1024 | 3-3-27-3 | 88.7M | 15.35G | 84.9% |
| Large | 192-384-768-1536 | 3-3-27-3 | 198.0M | 34.36G | 85.8% |
| Huge | 352-704-1408-2816 | 3-3-27-3 | 660.3M | 114.9G | 86.3% |
A V2 block has more parameters than a V1 block ( for GRN minus for layer scale): ConvNeXt V2-Tiny has 28,635,496 parameters against 28,589,128 for V1-Tiny. The multiply-add count is unchanged except for GRN's elementwise work, which this count (convolutions and linear layers only) omits; the paper lists Tiny at 4.47G.
1import torch, torch.nn as nn, torch.nn.functional as F2import timm34class GRN(nn.Module):5 """Global Response Normalization; input (B, H, W, C)."""6 def __init__(self, dim):7 super().__init__()8 self.gamma = nn.Parameter(torch.zeros(1, 1, 1, dim))9 self.beta = nn.Parameter(torch.zeros(1, 1, 1, dim))1011 def forward(self, x):12 Gx = torch.norm(x, p=2, dim=(1, 2), keepdim=True) # (B, 1, 1, C)13 Nx = Gx / (Gx.mean(dim=-1, keepdim=True) + 1e-6) # relative energy14 return self.gamma * (x * Nx) + self.beta + x1516x = torch.tensor([[3., 4, 0, 0], [1, 0, 0, 0], [2, 2, 1, 0]]).T.reshape(1, 2, 2, 3)17grn = GRN(3); nn.init.ones_(grn.gamma)18y = grn(x)19print(y.flatten(1, 2).norm(dim=1).detach()) # tensor([[13.3333, 1.3333, 6.0000]])2021# FCMAE masking: 60% of the 7x7 grid of 32x32 patches, upsampled to the 56x56 stage-1 map22B = 123keep = (torch.rand(B, 49).argsort(1).argsort(1) >= int(0.6 * 49)).float() # 1 = visible24m = keep.view(B, 1, 7, 7)25m = F.interpolate(m, scale_factor=8, mode="nearest") # (B, 1, 56, 56)2627dw = nn.Conv2d(96, 96, 7, padding=3, groups=96)28h = torch.randn(B, 96, 56, 56) * m29h_sparse = dw(h * m) * m # dense conv emulating a sparse one: masked sites stay 03031# Pretrained models (FCMAE pre-trained, ImageNet-1K fine-tuned)32model = timm.create_model("convnextv2_tiny.fcmae_ft_in1k", pretrained=True).eval()33print(sum(p.numel() for p in model.parameters())) # 28635496Use when
- +Self-supervised pre-training that works for ConvNets, improving every size from Atto to Huge
- +GRN is tiny (2 vectors per block), negligible in compute, and starts as the identity
- +Sparse encoder computes only on the visible 40% during pre-training
- +One architecture covers 3.7M to 660M parameters; 88.9% top-1 at the top with public data
Avoid when
- −Sparse convolution needs a special library (MinkowskiEngine) or masked dense emulation, which saves no compute
- −Pre-training (FCMAE) plus fine-tuning is far more expensive than supervised training alone
- −GRN's global pooling couples all positions of an image; it behaves differently on crops vs full images
- −Gains over V1 come mainly with FCMAE; supervised-only, the improvement is small
Check yourself
In GRN, a channel's energy Gᵢ equals the mean energy of all channels. With γᵢ = 1 and βᵢ = 0, how is that channel changed?
Check yourself
Why does FCMAE use sparse convolution in the encoder instead of simply zeroing the masked patches?
Remember
- ✓GRN: , , , with at init.
- ✓GRN makes channels compete for a fixed total response and counters the feature collapse seen in V1 under masked pre-training; layer scale is dropped.
- ✓FCMAE: mask 60% of patches, sparse-conv encoder on the visible part, one-block decoder, MSE on masked patches.
- ✓Dense convs on zero-filled inputs leak across mask borders; sparse convs (or input+output masking) prevent it.
- ✓Sizes Atto (3.7M, 76.7%) to Huge (660M, 88.9% at 512² with ImageNet-22K).