# ConvNeXt V2

> ConvNeXt co-designed with masked-autoencoder pre-training: FCMAE masks 60% of the image and encodes only the visible part with sparse convolutions, and Global Response Normalization makes channels compete, curing the feature collapse that held V1 back. Atto 3.7M → Huge 660M.

Source: https://www.mlfoundry.in/learn/convnext-v2  
Section: Deep Learning › Convolutional Neural Networks  
Level: advanced · 13 min read · updated 2026-10-11

**ConvNeXt V2:** is the **ConvNeXt** architecture with one new layer, **Global Response Normalization (GRN)**, placed after the GELU in every block (layer scale is removed), pre-trained as a **fully convolutional masked autoencoder (FCMAE)**: random 60% of the image is masked, the encoder sees only the visible part through sparse convolutions, and a light decoder reconstructs the masked pixels. After pre-training the model is fine-tuned with labels as usual.

Woo, Debnath, Hu, Chen, Liu, Kweon and Xie introduced it in "ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders" (CVPR 2023). Masked image modelling, especially He et al.'s MAE (2022), had become the strongest self-supervised pre-training for **Vision Transformers (ViT)**. A ViT drops masked tokens trivially: it simply does not feed them to the encoder. A ConvNet cannot, because its sliding windows have no notion of "absent" inputs, and simply zero-filling masked patches lets every convolution see the mask boundaries and leak information.

The paper's point is that the architecture and the pre-training must be **co-designed**. FCMAE alone barely helped ConvNeXt V1: inspecting the trained V1 features, the authors found **feature collapse**, many channels in the wide MLP layer were dead, saturated or redundant copies of each other. GRN fixes that, and with both changes FCMAE pre-training improves every model size. ConvNeXt V2-Huge reached 88.9% ImageNet top-1 at $512^2$ using only public data (ImageNet-22K intermediate fine-tuning), and the 3.7M-parameter Atto reached 76.7%.

## Global Response Normalization

$$
G_i=\lVert X_i\rVert_2=\Big(\sum_{p}X_{i,p}^2\Big)^{1/2},\qquad N_i=\frac{G_i}{\frac1C\sum_{j=1}^{C}G_j+\epsilon},\qquad Y_i=\gamma_i\,X_i\,N_i+\beta_i+X_i
$$

- $X_i$: channel $i$ of the block's $4d$-wide hidden map (after GELU), over all spatial positions $p$
- $G_i$: **global aggregation**: the L2 energy of channel $i$ over the whole map, one number per channel
- $N_i$: **divisive normalization**: energy relative to the channel mean, so $\sum_iN_i=C$; channels above average get $N_i>1$
- $\gamma_i,\beta_i$: learned per-channel scale and shift, **initialized to 0**, so GRN starts as the identity
- $+X_i$: residual inside the layer, which makes the identity initialization possible
- $\epsilon$: $10^{-6}$

Each channel is rescaled by how strong it is **relative to the other channels** of the same image. This is the classic idea of lateral inhibition or divisive normalization: channels compete for a fixed total of response, $\sum_iN_i=C$. The paper argues that this competition is what counters collapse: a channel that merely duplicates the others gains nothing, and the learned $\gamma_i$ (which can be negative) decide per channel whether to amplify or damp the contrast. It adds only $2\cdot4d$ parameters per block and negligible compute, and once GRN is present, layer scale is no longer needed.

$$
\frac{\partial G_i}{\partial X_{i,p}}=\frac{X_{i,p}}{G_i},\qquad \frac{\partial N_j}{\partial G_i}=\frac{\delta_{ij}}{\bar G}-\frac{G_j}{C\,\bar G^{2}},\qquad \frac{\partial L}{\partial\gamma_i}=N_i\sum_p g_{i,p}X_{i,p},\qquad \frac{\partial L}{\partial\beta_i}=\sum_p g_{i,p}
$$

- $\bar G$: $\frac1C\sum_jG_j$ (ignoring $\epsilon$)
- $g_{i,p}$: upstream gradient $\partial L/\partial Y_{i,p}$
- $\delta_{ij}$: 1 if $i=j$, else 0: raising one channel's energy raises its own $N_i$ and lowers every other $N_j$

At initialization $\gamma=0$, so $\partial Y/\partial X=I$ and the network trains like V1. The first updates to $\gamma_i$ are proportional to $N_i$: channels that are already strong relative to the others start with the largest calibration gradients.

## FCMAE: masking a ConvNet

FCMAE masks a random 60% of the $32\times32$ patches of a $224^2$ image. 32 is the total stride of ConvNeXt, so each masked patch is one position of the $7\times7$ last-stage map, and the same mask, upsampled, applies at every stage. The encoder is the ConvNeXt V2 backbone with **sparse convolutions** (the paper uses the MinkowskiEngine library): it stores and computes only the visible sites, and its kernels read only visible inputs, so the masked area is never touched. The decoder is a single ConvNeXt block of width 512; the loss is the mean squared error between the reconstructed and target pixels, on the masked patches only. After pre-training the sparse layers are converted back to ordinary dense convolutions.

Sparse convolution is equivalent to a dense convolution with the mask applied to its input and output, which is how it can be emulated in plain PyTorch. Without that, a dense $7\times7$ depthwise conv on the zero-filled image spreads visible values 3 positions into the masked area per layer, and lets visible positions near the border "see" the zeros, so the encoder learns the mask pattern instead of the image.

> Figure: Left: the FCMAE mask on the 56×56 stage-1 map. With the first mask at ratio 0.60, 29 of the 49 patches (59.2%) are masked. One dense 7×7 depthwise conv on the zero-filled input makes 1,062 of the 1,856 masked sites nonzero (57.2%) and lets 918 of the 1,280 visible sites read masked zeros; two layers leak into 1,640 masked sites and three into all of them. Switch to sparse conv: nothing leaks, and the encoder computes only 40.8% of the dense multiply-adds. Right: GRN on a 12-channel, 8×8 synthetic map, with one shared γ for all channels (in the model γ is learned per channel). The labels under the bars are Nᵢ; they always sum to 12. At γ = 1 the strongest/weakest channel-norm ratio rises from 58.2 to 202.9 and the top-3 channels' share of energy from 74.1% to 87.6%; a negative γ = −0.5 does the opposite (ratio 19.2, share 59.1%).

**Worked example: GRN on 3 channels of a 2×2 map, γ = 1, β = 0**

1. Channels $X_1=(3,4,0,0)$, $X_2=(1,0,0,0)$, $X_3=(2,2,1,0)$. Energies $G=(\sqrt{25},\sqrt1,\sqrt9)=(5,\,1,\,3)$.
2. Mean energy $\bar G=3$, so $N=(5/3,\,1/3,\,1)=(1.667,\,0.333,\,1.000)$, summing to $C=3$.
3. $Y_i=X_i(1+\gamma N_i)$: channel 1 is scaled by 2.667, channel 2 by 1.333, channel 3 by 2. So $Y_1=(8,\,10.667,\,0,\,0)$, $Y_2=(1.333,\,0,\,0,\,0)$, $Y_3=(4,\,4,\,2,\,0)$.
4. Output norms $(13.33,\,1.33,\,6.00)$: the strong-to-weak ratio went from 5 to 10. With $\gamma=0.5$ the scales are $(1.833,\,1.167,\,1.5)$ and the norms $(9.17,\,1.17,\,4.50)$. Values checked with the figure's `grn` function.

## Model family

ConvNeXt V2 sizes. Params and MACs at 224² counted by this module's code (GRN, no layer scale); they match the paper's Table (3.7M/0.55G … 660M/115G). Top-1: official repository, FCMAE pre-training + ImageNet-1K fine-tuning, 224².

| model | dims | depths | params | MACs | top-1 |
| --- | --- | --- | --- | --- | --- |
| Atto | 40-80-160-320 | 2-2-6-2 | 3.7M | 0.55G | 76.7% |
| Femto | 48-96-192-384 | 2-2-6-2 | 5.2M | 0.78G | 78.5% |
| Pico | 64-128-256-512 | 2-2-6-2 | 9.1M | 1.37G | 80.3% |
| Nano | 80-160-320-640 | 2-2-8-2 | 15.6M | 2.45G | 81.9% |
| Tiny | 96-192-384-768 | 3-3-9-3 | 28.6M | 4.46G | 83.0% |
| Base | 128-256-512-1024 | 3-3-27-3 | 88.7M | 15.35G | 84.9% |
| Large | 192-384-768-1536 | 3-3-27-3 | 198.0M | 34.36G | 85.8% |
| Huge | 352-704-1408-2816 | 3-3-27-3 | 660.3M | 114.9G | 86.3% |

A V2 block has $7d$ more parameters than a V1 block ($2\cdot4d$ for GRN minus $d$ for layer scale): ConvNeXt V2-Tiny has 28,635,496 parameters against 28,589,128 for V1-Tiny. The multiply-add count is unchanged except for GRN's elementwise work, which this count (convolutions and linear layers only) omits; the paper lists Tiny at 4.47G.

```python
import torch, torch.nn as nn, torch.nn.functional as F
import timm

class GRN(nn.Module):
    """Global Response Normalization; input (B, H, W, C)."""
    def __init__(self, dim):
        super().__init__()
        self.gamma = nn.Parameter(torch.zeros(1, 1, 1, dim))
        self.beta = nn.Parameter(torch.zeros(1, 1, 1, dim))

    def forward(self, x):
        Gx = torch.norm(x, p=2, dim=(1, 2), keepdim=True)        # (B, 1, 1, C)
        Nx = Gx / (Gx.mean(dim=-1, keepdim=True) + 1e-6)          # relative energy
        return self.gamma * (x * Nx) + self.beta + x

x = torch.tensor([[3., 4, 0, 0], [1, 0, 0, 0], [2, 2, 1, 0]]).T.reshape(1, 2, 2, 3)
grn = GRN(3); nn.init.ones_(grn.gamma)
y = grn(x)
print(y.flatten(1, 2).norm(dim=1).detach())   # tensor([[13.3333,  1.3333,  6.0000]])

# FCMAE masking: 60% of the 7x7 grid of 32x32 patches, upsampled to the 56x56 stage-1 map
B = 1
keep = (torch.rand(B, 49).argsort(1).argsort(1) >= int(0.6 * 49)).float()   # 1 = visible
m = keep.view(B, 1, 7, 7)
m = F.interpolate(m, scale_factor=8, mode="nearest")                         # (B, 1, 56, 56)

dw = nn.Conv2d(96, 96, 7, padding=3, groups=96)
h = torch.randn(B, 96, 56, 56) * m
h_sparse = dw(h * m) * m          # dense conv emulating a sparse one: masked sites stay 0

# Pretrained models (FCMAE pre-trained, ImageNet-1K fine-tuned)
model = timm.create_model("convnextv2_tiny.fcmae_ft_in1k", pretrained=True).eval()
print(sum(p.numel() for p in model.parameters()))   # 28635496
```

GRN (channels-last, as in the official code), FCMAE-style masked convolution, and the pretrained models in timm

> **Masking a ConvNet is not masking a ViT:** Zero-filling masked patches and running a dense ConvNet is the obvious approach, and it is what the figure's dense mode shows: after one $7\times7$ layer, more than half the masked area already holds values computed from visible pixels, and the visible sites near borders are computed from zeros. The pre-text task degrades and the features no longer match what the network sees at fine-tuning time, when nothing is masked. Use sparse convolution, or mask both the input and the output of every spatial layer.

> **Why a normalization layer helps self-supervision:** **Batch normalization** normalizes each channel over the batch and **Layer normalization** each position over channels; neither compares the *global* response of one channel with the others. GRN does exactly that, per image, and so penalizes channels that are always off, always on, or redundant. The paper shows that V1 trained with FCMAE has many such collapsed channels and V2 has far more diverse ones (larger pairwise cosine distance between feature maps), which is where the fine-tuning gain comes from.

> **Where you meet it:** timm ships every size, from `convnextv2_atto` to `convnextv2_huge`, with FCMAE-only, ImageNet-1K and ImageNet-22K fine-tuned weights. The small Atto–Nano models are strong mobile backbones; the FCMAE encoders are used as initializations for detection and segmentation.

**Strengths**

- Self-supervised pre-training that works for ConvNets, improving every size from Atto to Huge
- GRN is tiny (2 vectors per block), negligible in compute, and starts as the identity
- Sparse encoder computes only on the visible 40% during pre-training
- One architecture covers 3.7M to 660M parameters; 88.9% top-1 at the top with public data

**Limitations**

- Sparse convolution needs a special library (MinkowskiEngine) or masked dense emulation, which saves no compute
- Pre-training (FCMAE) plus fine-tuning is far more expensive than supervised training alone
- GRN's global pooling couples all positions of an image; it behaves differently on crops vs full images
- Gains over V1 come mainly with FCMAE; supervised-only, the improvement is small

**Question:** In GRN, a channel's energy Gᵢ equals the mean energy of all channels. With γᵢ = 1 and βᵢ = 0, how is that channel changed?

- A. It is left unchanged
- B. It is doubled
- C. It is set to zero
- D. It is divided by C

**Answer:** B. It is doubled — $N_i=G_i/\bar G=1$, so $Y_i=\gamma_iX_iN_i+X_i=2X_i$. Channels above the mean are scaled by more than 2, below the mean by less. With $\gamma_i=0$ (the initialization) every channel is unchanged.

**Question:** Why does FCMAE use sparse convolution in the encoder instead of simply zeroing the masked patches?

- A. Sparse convolution is more accurate in floating point
- B. A dense convolution spreads visible values into masked regions and lets visible sites see the mask, so the encoder learns the mask pattern instead of the image
- C. Zeroing patches would change the number of parameters
- D. Sparse convolution enlarges the receptive field

**Answer:** B. A dense convolution spreads visible values into masked regions and lets visible sites see the mask, so the encoder learns the mask pattern instead of the image — Each dense $7\times7$ layer leaks 3 positions across every mask boundary (57% of the masked sites after one layer in the figure, all of them after three). Sparse convolution computes only at visible sites from visible inputs, which is what a ViT gets for free by dropping masked tokens.

**Key points**

- GRN: $G_i=\lVert X_i\rVert_2$, $N_i=G_i/\bar G$, $Y_i=\gamma_iX_iN_i+\beta_i+X_i$, with $\gamma=\beta=0$ at init.
- GRN makes channels compete for a fixed total response and counters the feature collapse seen in V1 under masked pre-training; layer scale is dropped.
- FCMAE: mask 60% of $32\times32$ patches, sparse-conv encoder on the visible part, one-block decoder, MSE on masked patches.
- Dense convs on zero-filled inputs leak across mask borders; sparse convs (or input+output masking) prevent it.
- Sizes Atto (3.7M, 76.7%) to Huge (660M, 88.9% at 512² with ImageNet-22K).

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
