Mask2Former

Mask2Former (Cheng et al., CVPR 2022) predicts N (class, mask) pairs, each mask a query–pixel dot product, with masked cross-attention limited to the previous mask; one model reached 57.8 PQ, 50.1 AP and 57.7 mIoU.

8 min read897 wordsadvanced0 viewsupdated 2026-10-15

Definition

Mask2Former (Masked-attention Mask Transformer; Cheng, Misra, Schwing, Kirillov and Girdhar, "Masked-attention Mask Transformer for Universal Image Segmentation", CVPR 2022) is a segmentation model that represents any output as a set of NN binary masks, each with a class label. Semantic, instance and panoptic segmentation then share one architecture and one loss. Its key change is masked attention: each query's cross-attention is restricted to the foreground of the mask it predicted at the previous layer.

The three segmentation tasks used to need different designs (see Image segmentation). Semantic segmentation was per-pixel classification, as in FCN and DeepLab: a KK-way softmax at every pixel. Instance segmentation was detect-then-segment, as in Mask R-CNN. Panoptic segmentation combined the two. A per-pixel classifier cannot separate two touching dogs, and a box-based model handles amorphous "stuff" such as sky or road badly.

Mask classification handles all three. Predict NN pairs (pi,mi)(p_i, m_i), where pip_i is a distribution over KK classes plus "no object" ∅\varnothing and mi∈[0,1]H×Wm_i\in[0,1]^{H\times W} is a mask. A thing class gets one pair per instance, and a stuff class one pair per region. MaskFormer (Cheng, Schwing and Kirillov, NeurIPS 2021) introduced this with a DETR-style decoder. Mask2Former keeps the formulation and fixes what made MaskFormer slow to train and weak on small objects.

Architecture

Three parts. A backbone (ResNet or Swin Transformer) gives features at strides 4 to 32. A pixel decoder, by default 6 multi-scale deformable-attention layers as in Deformable DETR, refines the 1/32, 1/16 and 1/8 maps and upsamples them with a lateral connection to per-pixel embeddings F∈RC×H4×W4\mathcal F\in\mathbb R^{C\times\frac H4\times\frac W4}. A transformer decoder of 9 layers updates N=100N=100 query features. Successive layers read the 1/32, 1/16 and 1/8 maps in round robin, three times over. After every layer, each query predicts its class with a linear layer and its mask with a dot product:

mi[h,w]=σ(εi⊤ F[:,h,w]),εi=MLP⁡(qi)∈RCm_i[h,w]=\sigma\big(\varepsilon_i^{\top}\,\mathcal F[:,h,w]\big),\qquad \varepsilon_i=\operatorname{MLP}(q_i)\in\mathbb R^{C}
qiq_i
output feature of query ii after the current decoder layer
εi\varepsilon_i
mask embedding of query ii, from a small MLP; C=256C=256
F\mathcal F
per-pixel embeddings at 1/4 resolution from the pixel decoder, shared by all queries
Mask2Former: 100 queries → (class, mask) pairs; one architecture for every segmentation taskimagebackboneR50 / SwinC2 1/4C3 1/8C4 1/16C5 1/32pixel decoder6 MS-deform attnlayers on 1/8–1/32(as in DeformableDETR's encoder)+ upsample to 1/4:F ∈ ℝ^(256×H/4×W/4)1/321/161/8transformer decoder1/32masked X-attn→self→FFN1/16masked X-attn→self→FFN1/8masked X-attn→self→FFN× 3 rounds = 9 layersqueries100 × 256learnedper query i, after every decoder layerclass: linear → p_i over K + ∅mask: m_i = σ(MLP(q_i) · F), H/4 × W/4Fm_i > 0.5 → maskq_imasked cross-attention, layer l:X_l = softmax(M_(l−1) + Q_l K_lᵀ) V_l + X_(l−1)M_(l−1)(x, y) = 0 inside mask, −∞ outsideoutputs: 100 (class, mask) pairs; 4 confident showndog 0.96cat 0.91sky 0.88grass 0.85inference, same outputs for every task:semantic: argmax_c Σ_i p_i(c) · m_iinstance: score_i = p_i · mean mask probpanoptic: pixel → argmax_i p_i · m_i,drop low-score and occluded queriesTraining: Hungarian matching on class + mask cost; mask loss = BCE + Dice on 12,544 sampled points(112 × 112) instead of every pixel. Cross-attention comes before self-attention in each layer.Thing classes (dog, cat) get one query per instance; stuff classes (sky, grass) one per region.
fig.Mask2Former. The pixel decoder (multi-scale deformable attention) refines 1/8–1/32 features and builds 1/4 per-pixel embeddings F. Nine decoder layers update 100 queries, reading 1/32, 1/16 and 1/8 features in turn. After each layer every query predicts a class and a mask σ(MLP(q)·F). The mask, resized and thresholded at 0.5, limits where that query may look in the next layer's cross-attention. The same 100 (class, mask) pairs are merged into a semantic, instance or panoptic output.

Masked attention

In a DETR-style decoder, each query's cross-attention spans the whole feature map. MaskFormer needed long training partly because a query has to learn to ignore the background around its object. Mask2Former simply forbids that background. Each query attends only inside the mask it predicted at the previous layer:

Xl=softmax⁡(Ml−1+QlKl⊤)Vl+Xl−1,Ml−1(x,y)={0if σ(ml−1(x,y))>0.5−∞otherwiseX_l=\operatorname{softmax}\big(\mathcal M_{l-1}+Q_lK_l^{\top}\big)V_l+X_{l-1},\qquad \mathcal M_{l-1}(x,y)=\begin{cases}0 & \text{if } \sigma(m_{l-1}(x,y))>0.5\\-\infty & \text{otherwise}\end{cases}
XlX_l
the N×CN\times C query features after layer ll; Ql=fQ(Xl−1)Q_l=f_Q(X_{l-1})
Kl, VlK_l,\,V_l
keys and values from the image features of the scale assigned to layer ll
Ml−1\mathcal M_{l-1}
attention mask from the previous layer's mask prediction, resized to layer ll's resolution; M0\mathcal M_0 comes from the learnable query features before the first layer

The masks therefore refine themselves: a better mask focuses the next layer's attention, which gives a better query, which gives a better mask. If a query's mask is empty, the official code lets that query attend everywhere, so it is never left with an empty softmax. Three smaller changes help as well. Each layer runs cross-attention before self-attention. Query features are learned and supervised directly. Dropout is removed.

layer 2 · masked cross-attention · query attends to 74 of 384 pixelscross-attention weights of the dog querymask m = σ(γ(q·F − τ)) from this querygrey = masked out (mask ≤ 0.5) · black = dogIoU (m > 0.5 vs dog) = 0.803sky 0.0%dog 88.9%cat 0.0%grass 11.1%← share of this layer's attention per regionlayer012345standarddog attention · mask IoU34.3%0.31331.9%0.25528.3%0.20222.6%0.15515.7%0.0809.9%0.034maskeddog attention · mask IoU63.1%0.31377.7%0.57388.9%0.80394.9%0.88798.8%0.96999.4%0.984toy: 4-d unit pixel embeddings, no learned weights; per layer q ← unit(q + Σ a·F) · dog = 63 of 384 pixels
cross-attention
fig.A toy dog query refined over decoder layers, with no learned weights: unit 4-d pixel embeddings, masks from σ(γ(q·F − τ)), and the update q ← unit(q + attention read-out). The layer-0 mask is poor in both modes (IoU 0.313). At the defaults (layer 2, grass ≈ dog 0.5), masked attention looks at 74 of 384 pixels and puts 88.9% of its weight on the dog, so mask IoU reaches 0.803. Standard attention puts 28.3% there, and IoU falls to 0.202. By layer 5 it is 99.4% / 0.984 against 9.9% / 0.034. Raise the grass similarity to 0.6 and the first mask holds more grass (32 pixels) than dog (26). Masked attention then locks in the error and ends at IoU 0.034.

Training and inference

Training follows DETR. Predictions are Hungarian-matched to ground-truth segments with a class and mask cost, and unmatched predictions are trained toward ∅\varnothing (weight 0.1). The loss is L=λclsLcls+λceLBCE+λdiceLDice\mathcal L=\lambda_{\text{cls}}\mathcal L_{\text{cls}}+\lambda_{\text{ce}}\mathcal L_{\text{BCE}}+\lambda_{\text{dice}}\mathcal L_{\text{Dice}}, with λcls=2\lambda_{\text{cls}}=2 and λce=λdice=5\lambda_{\text{ce}}=\lambda_{\text{dice}}=5 (see Dice loss), applied after every decoder layer. The mask terms are not computed on all pixels. They use K=12,544K=12{,}544 sampled points (112×112112\times112): uniform points for the matching cost, and PointRend-style importance sampling for the loss. That cuts training memory about threefold, from 18 GB to 6 GB per image.

At inference the NN pairs are merged according to the task. Semantic: drop ∅\varnothing and take arg⁡max⁡c∑ipi(c) mi[h,w]\arg\max_c\sum_i p_i(c)\,m_i[h,w]. Instance: each query is one instance, scored by its class probability times its mean mask probability inside the mask. Panoptic: assign each pixel to arg⁡max⁡ipi(ci) mi[h,w]\arg\max_i p_i(c_i)\,m_i[h,w], then drop low-confidence queries and those left with little visible area.

Results reported by Cheng et al. (2022)

Headline (Swin-L based)
57.8 PQ on COCO panoptic, 50.1 AP on COCO instance, 57.7 mIoU on ADE20K semantic
ResNet-50, COCO
51.9 PQ panoptic, 43.7 AP instance (50 epochs)
Defaults
100 queries, 9 decoder layers (3 rounds over 1/32, 1/16, 1/8), 6-layer MSDeformAttn pixel decoder, C = 256
Optimisation
AdamW, lr 1e-4, weight decay 0.05, backbone lr × 0.1; mask loss on 12,544 points per mask

Worked example: semantic inference from mask classification

Three queries, four pixels A–D. Class probabilities over (dog, cat, grass, ∅\varnothing): p1=(0.90,0.05,0.02,0.03)p_1=(0.90,0.05,0.02,0.03), p2=(0.10,0.80,0.05,0.05)p_2=(0.10,0.80,0.05,0.05), p3=(0.02,0.03,0.90,0.05)p_3=(0.02,0.03,0.90,0.05). Masks on A–D: m1=(0.95,0.30,0.10,0.05)m_1=(0.95,0.30,0.10,0.05), m2=(0.20,0.85,0.05,0.02)m_2=(0.20,0.85,0.05,0.02), m3=(0.05,0.30,0.90,0.60)m_3=(0.05,0.30,0.90,0.60). The score of class cc at a pixel is ∑ipi(c) mi\sum_i p_i(c)\,m_i:

PixeldogcatgrassLabel
A0.90·0.95 + 0.10·0.20 + 0.02·0.05 = 0.8760.2090.074dog
B0.3610.05·0.30 + 0.80·0.85 + 0.03·0.30 = 0.7040.3185cat
C0.1130.0720.8145grass
D0.0590.03650.02·0.05 + 0.05·0.02 + 0.90·0.60 = 0.542grass
Σ_i p_i(c)·m_i per pixel (computed with numpy); ∅ is dropped before merging

At pixel B the dog query is confident about its class (0.90) but its mask there is low (0.30). The cat query wins with 0.80⋅0.850.80\cdot0.85. The class and the location come from different factors, and both must be high. For panoptic output, the same pixels go to arg⁡max⁡ipi(ci)mi\arg\max_i p_i(c_i)m_i: A→query 1 (0.855), B→query 2 (0.680), C and D→query 3 (0.810, 0.540).

python
1import torch
2from PIL import Image
3from transformers import AutoImageProcessor, Mask2FormerForUniversalSegmentation
4
5ckpt = "facebook/mask2former-swin-large-ade-semantic"
6proc = AutoImageProcessor.from_pretrained(ckpt)
7model = Mask2FormerForUniversalSegmentation.from_pretrained(ckpt).eval()
8image = Image.open("street.jpg").convert("RGB")
9with torch.no_grad():
10 out = model(**proc(images=image, return_tensors="pt"))
11
12logits = out.class_queries_logits # (1, N, K + 1), last column = no object
13masks = out.masks_queries_logits # (1, N, H/4, W/4) mask logits
14probs = logits.softmax(-1)[..., :-1] # drop no-object
15semseg = torch.einsum("bqc,bqhw->bchw", probs, masks.sigmoid()).argmax(1) # (1, H/4, W/4)
16
17# same rule, plus resizing to the input resolution
18seg = proc.post_process_semantic_segmentation(out, target_sizes=[image.size[::-1]])[0]
19print(semseg.shape, seg.shape, [model.config.id2label[i] for i in seg.unique().tolist()][:5])
Mask2Former with Hugging Face transformers: semantic output by the Σ p·m rule, and by the built-in post-processor (not run here: needs torch and transformers)

Check yourself

How does a Mask2Former query produce its mask?

Check yourself

What exactly does masked attention change compared with DETR/MaskFormer cross-attention?

Remember

  • ✓Mask classification: NN (class, mask) pairs, masks mi=σ(MLP⁡(qi)⊤F)m_i=\sigma(\operatorname{MLP}(q_i)^{\top}\mathcal F) on 1/4-resolution per-pixel embeddings; one architecture for semantic, instance and panoptic.
  • ✓Masked attention: softmax⁡(Ml−1+QK⊤)V\operatorname{softmax}(\mathcal M_{l-1}+QK^{\top})V with −∞-\infty outside the previous layer's mask (threshold 0.5), so the masks refine themselves.
  • ✓Pixel decoder = 6 multi-scale deformable-attention layers; 9 decoder layers read 1/32, 1/16 and 1/8 features in round robin; cross-attention before self-attention.
  • ✓Hungarian matching; loss 2·CE + 5·BCE + 5·Dice, with mask terms on 12,544 sampled points (about 3× less memory).
  • ✓Swin-L: 57.8 PQ (COCO panoptic), 50.1 AP (COCO instance), 57.7 mIoU (ADE20K).
← back to Deep Learning roadmap