# Mask2Former

> Mask2Former (Cheng et al., CVPR 2022) predicts N (class, mask) pairs, each mask a query–pixel dot product, with masked cross-attention limited to the previous mask; one model reached 57.8 PQ, 50.1 AP and 57.7 mIoU.

Source: https://www.mlfoundry.in/learn/mask2former  
Section: Deep Learning › Deep Learning for Computer Vision  
Level: advanced · 8 min read · updated 2026-10-15

**Mask2Former:** (Masked-attention Mask Transformer; Cheng, Misra, Schwing, Kirillov and Girdhar, "Masked-attention Mask Transformer for Universal Image Segmentation", CVPR 2022) is a segmentation model that represents any output as a set of $N$ binary masks, each with a class label. Semantic, instance and panoptic segmentation then share one architecture and one loss. Its key change is **masked attention**: each query's cross-attention is restricted to the foreground of the mask it predicted at the previous layer.

The three segmentation tasks used to need different designs (see **Image segmentation**). Semantic segmentation was **per-pixel classification**, as in **FCN** and **DeepLab**: a $K$-way softmax at every pixel. Instance segmentation was detect-then-segment, as in **Mask R-CNN**. Panoptic segmentation combined the two. A per-pixel classifier cannot separate two touching dogs, and a box-based model handles amorphous "stuff" such as sky or road badly.

**Mask classification** handles all three. Predict $N$ pairs $(p_i, m_i)$, where $p_i$ is a distribution over $K$ classes plus "no object" $\varnothing$ and $m_i\in[0,1]^{H\times W}$ is a mask. A thing class gets one pair per instance, and a stuff class one pair per region. MaskFormer (Cheng, Schwing and Kirillov, NeurIPS 2021) introduced this with a **DETR**-style decoder. Mask2Former keeps the formulation and fixes what made MaskFormer slow to train and weak on small objects.

## Architecture

Three parts. A **backbone** (ResNet or **Swin Transformer**) gives features at strides 4 to 32. A **pixel decoder**, by default 6 multi-scale deformable-attention layers as in **Deformable DETR**, refines the 1/32, 1/16 and 1/8 maps and upsamples them with a lateral connection to per-pixel embeddings $\mathcal F\in\mathbb R^{C\times\frac H4\times\frac W4}$. A **transformer decoder** of 9 layers updates $N=100$ query features. Successive layers read the 1/32, 1/16 and 1/8 maps in round robin, three times over. After every layer, each query predicts its class with a linear layer and its mask with a dot product:

$$
m_i[h,w]=\sigma\big(\varepsilon_i^{\top}\,\mathcal F[:,h,w]\big),\qquad \varepsilon_i=\operatorname{MLP}(q_i)\in\mathbb R^{C}
$$

- $q_i$: output feature of query $i$ after the current decoder layer
- $\varepsilon_i$: mask embedding of query $i$, from a small MLP; $C=256$
- $\mathcal F$: per-pixel embeddings at 1/4 resolution from the pixel decoder, shared by all queries

> Figure: Mask2Former. The pixel decoder (multi-scale deformable attention) refines 1/8–1/32 features and builds 1/4 per-pixel embeddings F. Nine decoder layers update 100 queries, reading 1/32, 1/16 and 1/8 features in turn. After each layer every query predicts a class and a mask σ(MLP(q)·F). The mask, resized and thresholded at 0.5, limits where that query may look in the next layer's cross-attention. The same 100 (class, mask) pairs are merged into a semantic, instance or panoptic output.

## Masked attention

In a DETR-style decoder, each query's cross-attention spans the whole feature map. MaskFormer needed long training partly because a query has to learn to ignore the background around its object. Mask2Former simply forbids that background. Each query attends only inside the mask it predicted at the previous layer:

$$
X_l=\operatorname{softmax}\big(\mathcal M_{l-1}+Q_lK_l^{\top}\big)V_l+X_{l-1},\qquad \mathcal M_{l-1}(x,y)=\begin{cases}0 & \text{if } \sigma(m_{l-1}(x,y))>0.5\\-\infty & \text{otherwise}\end{cases}
$$

- $X_l$: the $N\times C$ query features after layer $l$; $Q_l=f_Q(X_{l-1})$
- $K_l,\,V_l$: keys and values from the image features of the scale assigned to layer $l$
- $\mathcal M_{l-1}$: attention mask from the previous layer's mask prediction, resized to layer $l$'s resolution; $\mathcal M_0$ comes from the learnable query features before the first layer

The masks therefore refine themselves: a better mask focuses the next layer's attention, which gives a better query, which gives a better mask. If a query's mask is empty, the official code lets that query attend everywhere, so it is never left with an empty softmax. Three smaller changes help as well. Each layer runs cross-attention *before* self-attention. Query features are learned and supervised directly. Dropout is removed.

> Figure: A toy dog query refined over decoder layers, with no learned weights: unit 4-d pixel embeddings, masks from σ(γ(q·F − τ)), and the update q ← unit(q + attention read-out). The layer-0 mask is poor in both modes (IoU 0.313). At the defaults (layer 2, grass ≈ dog 0.5), masked attention looks at 74 of 384 pixels and puts 88.9% of its weight on the dog, so mask IoU reaches 0.803. Standard attention puts 28.3% there, and IoU falls to 0.202. By layer 5 it is 99.4% / 0.984 against 9.9% / 0.034. Raise the grass similarity to 0.6 and the first mask holds more grass (32 pixels) than dog (26). Masked attention then locks in the error and ends at IoU 0.034.

## Training and inference

Training follows DETR. Predictions are Hungarian-matched to ground-truth segments with a class and mask cost, and unmatched predictions are trained toward $\varnothing$ (weight 0.1). The loss is $\mathcal L=\lambda_{\text{cls}}\mathcal L_{\text{cls}}+\lambda_{\text{ce}}\mathcal L_{\text{BCE}}+\lambda_{\text{dice}}\mathcal L_{\text{Dice}}$, with $\lambda_{\text{cls}}=2$ and $\lambda_{\text{ce}}=\lambda_{\text{dice}}=5$ (see **Dice loss**), applied after every decoder layer. The mask terms are not computed on all pixels. They use $K=12{,}544$ sampled points ($112\times112$): uniform points for the matching cost, and PointRend-style importance sampling for the loss. That cuts training memory about threefold, from 18 GB to 6 GB per image.

At inference the $N$ pairs are merged according to the task. **Semantic**: drop $\varnothing$ and take $\arg\max_c\sum_i p_i(c)\,m_i[h,w]$. **Instance**: each query is one instance, scored by its class probability times its mean mask probability inside the mask. **Panoptic**: assign each pixel to $\arg\max_i p_i(c_i)\,m_i[h,w]$, then drop low-confidence queries and those left with little visible area.

**Results reported by Cheng et al. (2022)**

- **Headline (Swin-L based):** 57.8 PQ on COCO panoptic, 50.1 AP on COCO instance, 57.7 mIoU on ADE20K semantic
- **ResNet-50, COCO:** 51.9 PQ panoptic, 43.7 AP instance (50 epochs)
- **Defaults:** 100 queries, 9 decoder layers (3 rounds over 1/32, 1/16, 1/8), 6-layer MSDeformAttn pixel decoder, C = 256
- **Optimisation:** AdamW, lr 1e-4, weight decay 0.05, backbone lr × 0.1; mask loss on 12,544 points per mask

## Worked example: semantic inference from mask classification

Three queries, four pixels A–D. Class probabilities over (dog, cat, grass, $\varnothing$): $p_1=(0.90,0.05,0.02,0.03)$, $p_2=(0.10,0.80,0.05,0.05)$, $p_3=(0.02,0.03,0.90,0.05)$. Masks on A–D: $m_1=(0.95,0.30,0.10,0.05)$, $m_2=(0.20,0.85,0.05,0.02)$, $m_3=(0.05,0.30,0.90,0.60)$. The score of class $c$ at a pixel is $\sum_i p_i(c)\,m_i$:

Σ_i p_i(c)·m_i per pixel (computed with numpy); ∅ is dropped before merging

| Pixel | dog | cat | grass | Label |
| --- | --- | --- | --- | --- |
| A | 0.90·0.95 + 0.10·0.20 + 0.02·0.05 = 0.876 | 0.209 | 0.074 | dog |
| B | 0.361 | 0.05·0.30 + 0.80·0.85 + 0.03·0.30 = 0.704 | 0.3185 | cat |
| C | 0.113 | 0.072 | 0.8145 | grass |
| D | 0.059 | 0.0365 | 0.02·0.05 + 0.05·0.02 + 0.90·0.60 = 0.542 | grass |

At pixel B the dog query is confident about its class (0.90) but its mask there is low (0.30). The cat query wins with $0.80\cdot0.85$. The class and the location come from different factors, and both must be high. For panoptic output, the same pixels go to $\arg\max_i p_i(c_i)m_i$: A→query 1 (0.855), B→query 2 (0.680), C and D→query 3 (0.810, 0.540).

```python
import torch
from PIL import Image
from transformers import AutoImageProcessor, Mask2FormerForUniversalSegmentation

ckpt = "facebook/mask2former-swin-large-ade-semantic"
proc = AutoImageProcessor.from_pretrained(ckpt)
model = Mask2FormerForUniversalSegmentation.from_pretrained(ckpt).eval()
image = Image.open("street.jpg").convert("RGB")
with torch.no_grad():
    out = model(**proc(images=image, return_tensors="pt"))

logits = out.class_queries_logits            # (1, N, K + 1), last column = no object
masks = out.masks_queries_logits             # (1, N, H/4, W/4) mask logits
probs = logits.softmax(-1)[..., :-1]         # drop no-object
semseg = torch.einsum("bqc,bqhw->bchw", probs, masks.sigmoid()).argmax(1)   # (1, H/4, W/4)

# same rule, plus resizing to the input resolution
seg = proc.post_process_semantic_segmentation(out, target_sizes=[image.size[::-1]])[0]
print(semseg.shape, seg.shape, [model.config.id2label[i] for i in seg.unique().tolist()][:5])
```

Mask2Former with Hugging Face transformers: semantic output by the Σ p·m rule, and by the built-in post-processor (not run here: needs torch and transformers)

> **Why one model fits three tasks:** A per-pixel classifier ties a pixel's label to one of $K$ fixed channels, so two touching instances of a class cannot be told apart. Mask classification separates "which segment" (the query) from "which class" (its $p_i$). An instance is just a segment of a thing class, and a stuff region is a segment of a stuff class. Only the training targets and the merge step change between tasks.

> **Masked attention needs a reasonable first mask:** The restriction is hard: pixels outside the previous mask get $-\infty$ and no gradient through attention. A query whose early mask misses part of the object can still recover, because its mask is predicted against all pixels of $\mathcal F$ at every layer. If the early mask lands on the wrong region, though, attention keeps reading that region, as the figure shows at grass similarity 0.6. Deep supervision of every layer's mask, including $\mathcal M_0$ from the learned queries, is what keeps early masks useful.

**Question:** How does a Mask2Former query produce its mask?

- A. A separate FCN head per query, run on the box crop
- B. A sigmoid of the dot product between the query's mask embedding (MLP of the query) and the 1/4-resolution per-pixel embeddings
- C. Bilinear upsampling of its cross-attention map
- D. A softmax over the K classes at every pixel

**Answer:** B. A sigmoid of the dot product between the query's mask embedding (MLP of the query) and the 1/4-resolution per-pixel embeddings — $m_i=\sigma(\varepsilon_i^{\top}\mathcal F)$ with $\varepsilon_i=\operatorname{MLP}(q_i)$. All queries share the same $\mathcal F$, so each extra query costs one $C$-dimensional dot product per pixel. The class comes separately from a linear layer on $q_i$.

**Question:** What exactly does masked attention change compared with DETR/MaskFormer cross-attention?

- A. It replaces softmax with a sigmoid
- B. It adds −∞ to the attention logits at pixels where the query's previous-layer mask is ≤ 0.5, so attention is confined to the predicted foreground
- C. It masks future tokens, as in a causal decoder
- D. It drops random pixels as a regulariser

**Answer:** B. It adds −∞ to the attention logits at pixels where the query's previous-layer mask is ≤ 0.5, so attention is confined to the predicted foreground — $X_l=\operatorname{softmax}(\mathcal M_{l-1}+Q_lK_l^{\top})V_l+X_{l-1}$, with $\mathcal M_{l-1}=0$ inside the thresholded previous mask and $-\infty$ elsewhere. It is spatial and query-specific, unlike a causal mask, and it is fully determined by the model's own prediction.

**Key points**

- Mask classification: $N$ (class, mask) pairs, masks $m_i=\sigma(\operatorname{MLP}(q_i)^{\top}\mathcal F)$ on 1/4-resolution per-pixel embeddings; one architecture for semantic, instance and panoptic.
- Masked attention: $\operatorname{softmax}(\mathcal M_{l-1}+QK^{\top})V$ with $-\infty$ outside the previous layer's mask (threshold 0.5), so the masks refine themselves.
- Pixel decoder = 6 multi-scale deformable-attention layers; 9 decoder layers read 1/32, 1/16 and 1/8 features in round robin; cross-attention before self-attention.
- Hungarian matching; loss 2·CE + 5·BCE + 5·Dice, with mask terms on 12,544 sampled points (about 3× less memory).
- Swin-L: 57.8 PQ (COCO panoptic), 50.1 AP (COCO instance), 57.7 mIoU (ADE20K).

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
