Definition
Mask R-CNN is an instance segmentation model (Kaiming He, Georgia Gkioxari, Piotr Dollár and Ross Girshick, "Mask R-CNN", ICCV 2017, Marr Prize). It extends Faster R-CNN with a third head, parallel to classification and box regression, that outputs a binary mask for each region of interest. It also introduces RoIAlign, a feature-extraction layer without quantisation.
Instance segmentation asks for a class, a box and a pixel mask for every object, so two touching people get two separate masks. Semantic segmentation (see Image segmentation and FCN) labels pixels but cannot tell instances apart. Detectors separate instances but give only boxes.
Earlier systems such as DeepMask, MNC and FCIS proposed segments first and then classified them, or predicted position-sensitive score maps. Mask R-CNN does the reverse: detect first, then segment inside each box. Mask prediction is decoupled from classification.
Architecture
The backbone and RPN are those of Faster R-CNN, usually with a ResNet-50/101 or ResNeXt-101 backbone and a Feature Pyramid Network (FPN). Each RoI is pooled from the pyramid level that matches its size. The box head sees 7×7 features. The mask head sees 14×14 features and is a small fully convolutional network: four 3×3 convs, a 2× deconvolution and a 1×1 conv to channels at 28×28.
The mask loss: one sigmoid mask per class
- the logits of class , the RoI's ground-truth class; the other masks get no loss
- target: the ground-truth mask cropped to the RoI and resized to m × m (m = 28)
- as in Fast R-CNN; is defined only on positive RoIs (IoU ≥ 0.5 with a ground-truth box)
Because each class has its own sigmoid mask, classes do not compete per pixel; the classification head decides the class. A per-pixel softmax over classes couples the two tasks, and the paper reports it costs 5.5 mask AP (24.8 vs 30.3 with ResNet-50-C4).
The mask is predicted at with convolutions, not with fc layers, so it keeps the RoI's spatial layout. That only works if the RoI features line up with the image. A shift of a few pixels hardly changes a box score, but it does change a mask.
RoIAlign: sample, don't round
RoIPool rounds twice: the RoI coordinate to a whole cell, and each bin boundary to a whole cell. Either can move the pooled region by several pixels in the image. RoIAlign keeps the continuous coordinates. It divides the RoI into exact bins, samples 4 regularly spaced points per bin (2×2), reads each point by bilinear interpolation of the 4 nearest feature cells, and aggregates them by max or average. The paper notes the result is not sensitive to the exact sample positions or how many there are, as long as nothing is quantised.
- a sample point in feature-map units, e.g. ; no rounding anywhere
- feature value of cell , whose centre is at ; the half-pixel offset is torchvision's
aligned=True - the bilinear weights, so gradients reach the 4 neighbouring cells of every sample
Worked example: one RoIAlign bin
The RoI from the figure at its default position: px, size 72×76, pooled to 2×2. In feature units it starts at and each bin is cells. Bin (0, 0) samples at and .
| Neighbour cell (a, b) | Value F | Weight | Contribution |
|---|---|---|---|
| (3, 2) | 0.30 | 0.8125 × 0.34375 = 0.2793 | 0.0838 |
| (4, 2) | 0.48 | 0.1875 × 0.34375 = 0.0645 | 0.0309 |
| (3, 3) | 0.38 | 0.8125 × 0.65625 = 0.5332 | 0.2026 |
| (4, 3) | 0.63 | 0.1875 × 0.65625 = 0.1230 | 0.0775 |
| Sum | 1.0000 | 0.3949 |
The other three samples give 0.160, 0.278 and 0.200, so the bin outputs 0.395 (max) or 0.258 (average). RoIPool rounds the same RoI to cells x 2–7, y 1–6. Its bin (0, 0) covers cells x 2–4, y 1–3 and returns 0.63 from cell (4, 3), whose centre (4.5, 3.5) lies outside the true bin, which ends at u = 4.25.
Reported results (COCO)
- Mask AP, test-dev
- 35.7 with ResNet-101-FPN and 37.1 with ResNeXt-101-FPN, single model, vs 33.6 for FCIS+++, the COCO 2016 winner
- Box AP, test-dev
- 38.2 (ResNet-101-FPN) and 39.8 (ResNeXt-101-FPN), above Faster R-CNN with the same backbone, because of RoIAlign and multi-task training
- RoIAlign ablation
- relative mask-AP gains of 10–50%. With a stride-32 ResNet-50-C5 backbone, +7.3 mask AP and +10.5 AP₇₅.
- Speed and training
- about 5 fps (195 ms per image on a Tesla M40) for ResNet-101-FPN. Trains in 1–2 days on one 8-GPU machine, with images of short side 800.
- Inference
- the mask branch runs on the top 100 detections only. Each 28 × 28 float mask is resized to its box and thresholded at 0.5.
Keypoints with the same head
For human pose, each of the COCO keypoints becomes a one-hot mask with a single foreground pixel. The head predicts 17 maps at 56×56 and uses a softmax cross-entropy over the positions of each map, so it has to pick one location. A ResNet-50-FPN model reaches 62.7 keypoint AP, above the 61.8 of the COCO 2016 keypoint winner, and still runs at about 5 fps. See Pose estimation.
1import torch2from torchvision.io import read_image3from torchvision.models.detection import maskrcnn_resnet50_fpn, MaskRCNN_ResNet50_FPN_Weights45weights = MaskRCNN_ResNet50_FPN_Weights.COCO_V16model = maskrcnn_resnet50_fpn(weights=weights).eval()7img = weights.transforms()(read_image("people.jpg"))89with torch.no_grad():10 out = model([img])[0] # boxes [N,4], labels [N], scores [N], masks [N,1,H,W]1112keep = out["scores"] > 0.713masks = out["masks"][keep, 0] > 0.5 # already pasted into the image: soft 28x28 masks resized to each box14names = weights.meta["categories"]15for lab, m in zip(out["labels"][keep], masks):16 print(names[lab], int(m.sum()), "pixels")Check yourself
An RoI's ground-truth class is 'dog' and COCO has K = 80 classes. Which mask outputs receive gradient from L_mask?
Check yourself
Why does RoIAlign matter more for masks than for box classification?
Remember
- ✓Mask R-CNN = Faster R-CNN + an FCN mask head that predicts K masks of 28 × 28 per RoI. Only the mask of the RoI's class is used.
- ✓; is the average per-pixel binary cross-entropy on positive RoIs. A softmax across classes costs 5.5 AP.
- ✓RoIAlign removes both roundings of RoIPool and reads 4 bilinear samples per bin. Its output changes smoothly with the box.
- ✓COCO test-dev: 37.1 mask AP and 39.8 box AP with ResNeXt-101-FPN, at about 5 fps.
- ✓The same head predicts 17 keypoints as one-hot 56 × 56 masks with a spatial softmax: 62.7 keypoint AP.