Mask R-CNN

Mask R-CNN adds a small FCN branch to Faster R-CNN that predicts a 28×28 sigmoid mask per class for every RoI, and replaces RoIPool with RoIAlign, whose bilinear sampling removes the rounding that misaligned features with pixels.

8 min read760 wordscore0 viewsupdated 2026-10-15

Definition

Mask R-CNN is an instance segmentation model (Kaiming He, Georgia Gkioxari, Piotr Dollár and Ross Girshick, "Mask R-CNN", ICCV 2017, Marr Prize). It extends Faster R-CNN with a third head, parallel to classification and box regression, that outputs a binary mask for each region of interest. It also introduces RoIAlign, a feature-extraction layer without quantisation.

Instance segmentation asks for a class, a box and a pixel mask for every object, so two touching people get two separate masks. Semantic segmentation (see Image segmentation and FCN) labels pixels but cannot tell instances apart. Detectors separate instances but give only boxes.

Earlier systems such as DeepMask, MNC and FCIS proposed segments first and then classified them, or predicted position-sensitive score maps. Mask R-CNN does the reverse: detect first, then segment inside each box. Mask prediction is decoupled from classification.

Architecture

The backbone and RPN are those of Faster R-CNN, usually with a ResNet-50/101 or ResNeXt-101 backbone and a Feature Pyramid Network (FPN). Each RoI is pooled from the pyramid level that matches its size. The box head sees 7×7 features. The mask head sees 14×14 features and is a small fully convolutional network: four 3×3 convs, a 2× deconvolution and a 1×1 conv to KK channels at 28×28.

Mask R-CNN (He, Gkioxari, Dollár, Girshick 2017): Faster R-CNN + a parallel mask headimageshort side 800ResNet50 / 101backboneFPNP2–P5 (+P6)RPN+ NMSproposalsRoIAlign (no rounding)4 bilinearsamples / binlevel by RoI size7×7×256fc 1024 ×2class K+1 · box 4Kbox head, as inFaster R-CNN14×14×2564 × conv 3×3256, ReLUdeconv 2×stride 228×28×2561×1 → 28×28×Kmask of class k onlyσ per pixelresize to boxthreshold 0.5L = L_cls + L_box + L_mask, L_mask = mean over 28×28 of BCE(σ(m_k), target), k = true classpositive RoIs only (IoU ≥ 0.5); target = ground-truth mask cropped to the RoI; no competition across classes
fig.Mask R-CNN with an FPN head. RoIAlign extracts 7 × 7 × 256 features for the box head (two fc-1024 layers, then K+1 classes and 4K box offsets) and 14 × 14 × 256 features for the mask head. Four 3 × 3 convs, a stride-2 deconvolution and a 1 × 1 conv give a 28 × 28 × K stack, one mask per class (K = 80 on COCO). Only the mask of the class in question is used: the ground-truth class in training, the predicted class at test time.

The mask loss: one sigmoid mask per class

L=Lcls+Lbox+Lmask,Lmask=−1m2∑(i,j)[yijlog⁡σ(zijk)+(1−yij)log⁡(1−σ(zijk))]L=L_{\text{cls}}+L_{\text{box}}+L_{\text{mask}},\qquad L_{\text{mask}}=-\frac{1}{m^2}\sum_{(i,j)}\Big[y_{ij}\log\sigma\big(z^{k}_{ij}\big)+\big(1-y_{ij}\big)\log\big(1-\sigma(z^{k}_{ij})\big)\Big]
zkz^{k}
the m×mm\times m logits of class kk, the RoI's ground-truth class; the other K−1K-1 masks get no loss
yijy_{ij}
target: the ground-truth mask cropped to the RoI and resized to m × m (m = 28)
Lcls, LboxL_{\text{cls}},\ L_{\text{box}}
as in Fast R-CNN; LmaskL_{\text{mask}} is defined only on positive RoIs (IoU ≥ 0.5 with a ground-truth box)

Because each class has its own sigmoid mask, classes do not compete per pixel; the classification head decides the class. A per-pixel softmax over KK classes couples the two tasks, and the paper reports it costs 5.5 mask AP (24.8 vs 30.3 with ResNet-50-C4).

The mask is predicted at m×mm\times m with convolutions, not with fc layers, so it keeps the RoI's spatial layout. That only works if the RoI features line up with the image. A shift of a few pixels hardly changes a box score, but it does change a mask.

RoIAlign: sample, don't round

RoIPool rounds twice: the RoI coordinate x/16x/16 to a whole cell, and each bin boundary to a whole cell. Either can move the pooled region by several pixels in the image. RoIAlign keeps the continuous coordinates. It divides the RoI into exact bins, samples 4 regularly spaced points per bin (2×2), reads each point by bilinear interpolation of the 4 nearest feature cells, and aggregates them by max or average. The paper notes the result is not sensitive to the exact sample positions or how many there are, as long as nothing is quantised.

f(u,v)=∑a∈{⌊u′⌋,⌊u′⌋+1} ∑b∈{⌊v′⌋,⌊v′⌋+1}(1−∣u′−a∣)(1−∣v′−b∣) Fb,a,u′=u−12, v′=v−12f(u,v)=\sum_{a\in\{\lfloor u'\rfloor,\lfloor u'\rfloor+1\}}\ \sum_{b\in\{\lfloor v'\rfloor,\lfloor v'\rfloor+1\}} \big(1-|u'-a|\big)\big(1-|v'-b|\big)\,F_{b,a},\qquad u'=u-\tfrac12,\ v'=v-\tfrac12
(u,v)(u,v)
a sample point in feature-map units, e.g. x/16x/16; no rounding anywhere
Fb,aF_{b,a}
feature value of cell (a,b)(a,b), whose centre is at (a+12,b+12)(a+\tfrac12,b+\tfrac12); the half-pixel offset is torchvision's aligned=True
∂f/∂F\partial f/\partial F
the bilinear weights, so gradients reach the 4 neighbouring cells of every sample
feature patch 10×8 cells (stride 16); RoI 72×76 px at x1 = 32.10.10.12.14.18.21.22.22.22.22.10.11.14.21.31.37.40.41.41.41.11.12.17.30.48.61.66.68.68.68.11.13.20.38.63.81.88.90.90.91.11.13.20.38.65.83.90.92.93.93.11.12.18.32.52.66.71.73.73.74.10.11.15.22.34.42.45.46.46.46.10.11.12.15.20.23.25.25.25.25exact RoI + bins · bilinear samples · RoIPool box, bin (0,0)RoIPoolround, integer bins, max0.6300.9000.6500.920RoIAlign4 samples per bin, max0.3950.7690.4300.857RoIPool box edges moved 0, −6, +24, +14 pxworst 24 px = 9.3 mask pixels at 28×28RoIAlign: bins of 2.250×2.375 cells, no rounding(one mask pixel = 72/28 = 2.57 image px)bin (0, 0) as the RoI slides right by 0–32 px (two cells): RoIPool jumps 2×, RoIAlign is smooth0.00.51.00 px+8+16+24+32
RoIAlign pools
fig.RoIPool vs RoIAlign on one channel of a stride-16 map, using torchvision's rounding rules for RoIPool. At the default RoI (x1 = 32 px) RoIPool's box has moved 0, −6, +24 and +14 px from the requested box. 24 px is 9.3 mask pixels at 28 × 28 for this 72-px RoI. Its bin (0, 0) returns 0.630, and RoIAlign's returns 0.395. Slide the RoI from +13 to +14 px: RoIPool's bin (0, 0) jumps from 0.63 to 0.81 as the rounded left edge flips from −7 to +8 px, while RoIAlign moves smoothly from 0.494 to 0.508. Over the 32-px sweep RoIPool changes only twice (at +14 and +30).

Worked example: one RoIAlign bin

The RoI from the figure at its default position: (x1,y1)=(32,22)(x_1,y_1)=(32,22) px, size 72×76, pooled to 2×2. In feature units it starts at (2.0,1.375)(2.0, 1.375) and each bin is 2.25×2.3752.25\times2.375 cells. Bin (0, 0) samples at u∈{2.5625,3.6875}u\in\{2.5625, 3.6875\} and v∈{1.96875,3.15625}v\in\{1.96875, 3.15625\}.

Neighbour cell (a, b)Value FWeightContribution
(3, 2)0.300.8125 × 0.34375 = 0.27930.0838
(4, 2)0.480.1875 × 0.34375 = 0.06450.0309
(3, 3)0.380.8125 × 0.65625 = 0.53320.2026
(4, 3)0.630.1875 × 0.65625 = 0.12300.0775
Sum1.00000.3949
The sample at (u, v) = (3.6875, 3.15625): u′ = 3.1875, v′ = 2.65625, so the fractions are 0.1875 and 0.65625 (computed in Python)

The other three samples give 0.160, 0.278 and 0.200, so the bin outputs 0.395 (max) or 0.258 (average). RoIPool rounds the same RoI to cells x 2–7, y 1–6. Its bin (0, 0) covers cells x 2–4, y 1–3 and returns 0.63 from cell (4, 3), whose centre (4.5, 3.5) lies outside the true bin, which ends at u = 4.25.

Reported results (COCO)

Mask AP, test-dev
35.7 with ResNet-101-FPN and 37.1 with ResNeXt-101-FPN, single model, vs 33.6 for FCIS+++, the COCO 2016 winner
Box AP, test-dev
38.2 (ResNet-101-FPN) and 39.8 (ResNeXt-101-FPN), above Faster R-CNN with the same backbone, because of RoIAlign and multi-task training
RoIAlign ablation
relative mask-AP gains of 10–50%. With a stride-32 ResNet-50-C5 backbone, +7.3 mask AP and +10.5 AP₇₅.
Speed and training
about 5 fps (195 ms per image on a Tesla M40) for ResNet-101-FPN. Trains in 1–2 days on one 8-GPU machine, with images of short side 800.
Inference
the mask branch runs on the top 100 detections only. Each 28 × 28 float mask is resized to its box and thresholded at 0.5.

Keypoints with the same head

For human pose, each of the K=17K=17 COCO keypoints becomes a one-hot m×mm\times m mask with a single foreground pixel. The head predicts 17 maps at 56×56 and uses a softmax cross-entropy over the m2m^2 positions of each map, so it has to pick one location. A ResNet-50-FPN model reaches 62.7 keypoint AP, above the 61.8 of the COCO 2016 keypoint winner, and still runs at about 5 fps. See Pose estimation.

python
1import torch
2from torchvision.io import read_image
3from torchvision.models.detection import maskrcnn_resnet50_fpn, MaskRCNN_ResNet50_FPN_Weights
4
5weights = MaskRCNN_ResNet50_FPN_Weights.COCO_V1
6model = maskrcnn_resnet50_fpn(weights=weights).eval()
7img = weights.transforms()(read_image("people.jpg"))
8
9with torch.no_grad():
10 out = model([img])[0] # boxes [N,4], labels [N], scores [N], masks [N,1,H,W]
11
12keep = out["scores"] > 0.7
13masks = out["masks"][keep, 0] > 0.5 # already pasted into the image: soft 28x28 masks resized to each box
14names = weights.meta["categories"]
15for lab, m in zip(out["labels"][keep], masks):
16 print(names[lab], int(m.sum()), "pixels")
Instance masks from torchvision's COCO-trained Mask R-CNN (ResNet-50-FPN); weights download on first use (not run here)

Check yourself

An RoI's ground-truth class is 'dog' and COCO has K = 80 classes. Which mask outputs receive gradient from L_mask?

Check yourself

Why does RoIAlign matter more for masks than for box classification?

Remember

  • ✓Mask R-CNN = Faster R-CNN + an FCN mask head that predicts K masks of 28 × 28 per RoI. Only the mask of the RoI's class is used.
  • ✓L=Lcls+Lbox+LmaskL=L_{\text{cls}}+L_{\text{box}}+L_{\text{mask}}; LmaskL_{\text{mask}} is the average per-pixel binary cross-entropy on positive RoIs. A softmax across classes costs 5.5 AP.
  • ✓RoIAlign removes both roundings of RoIPool and reads 4 bilinear samples per bin. Its output changes smoothly with the box.
  • ✓COCO test-dev: 37.1 mask AP and 39.8 box AP with ResNeXt-101-FPN, at about 5 fps.
  • ✓The same head predicts 17 keypoints as one-hot 56 × 56 masks with a spatial softmax: 62.7 keypoint AP.
← back to Deep Learning roadmap