Definition
Image segmentation is dense prediction: assign every pixel of an image a label. Semantic segmentation gives each pixel a class. Instance segmentation gives a separate mask to each countable object. Panoptic segmentation (Kirillov, He, Girshick, Rother and Dollár, CVPR 2019) unifies the two: every pixel gets a class and, for object classes, an instance id.
Input: an image . Output: an label map, or a set of binary masks. Classification needs one label per image and detection one box per object; segmentation needs the exact outline. The network must see enough context to know what a region is, yet keep enough resolution to say where its border lies. Downsampling helps the first and hurts the second, and that tension shapes every architecture below.
Panoptic segmentation splits classes into things (countable objects: car, person) and stuff (amorphous regions: sky, road, grass). Stuff gets one segment per class; things get one segment per instance. Instance segmentation ignores stuff, and its masks may overlap. Panoptic segments may not.
Training: per-pixel cross-entropy
A semantic model outputs logits , usually computed at stride 8 to 32 and upsampled to full resolution. The loss is ordinary cross-entropy at every labelled pixel, averaged:
- labelled pixels; "void" pixels (often label 255, e.g. object borders in PASCAL VOC) are excluded
- ground-truth class of pixel
- logit of class at pixel
Large regions dominate this average, so small classes are often under-segmented. Common fixes are class-weighted cross-entropy, the pixel weight map of U-Net, and overlap losses such as Dice loss. Instance models instead predict one binary mask per detected object (Mask R-CNN) or per query (Mask2Former), trained with per-pixel binary cross-entropy, often plus a Dice term.
Metrics: mIoU and PQ
For semantic segmentation, accumulate a confusion matrix over the whole dataset, then compute IoU per class and average over classes:
- pixels of class predicted as
- pixels wrongly predicted as ; pixels of predicted as something else
Each class counts equally, so a small class missed entirely costs of mIoU, however few pixels it has. Pixel accuracy hides that. Instance segmentation is scored like detection, by mask AP: COCO AP averaged over mask-IoU thresholds 0.50 to 0.95. Panoptic quality matches predicted and ground-truth segments of the same class. A match needs IoU > 0.5, which makes matches unique:
- matched pairs, unmatched predicted segments, unmatched ground-truth segments
- segmentation quality: mean IoU of the matched segments
- recognition quality: an F1 score over segments
PQ is computed per class and then averaged over classes. Because SQ and RQ are also averaged per class, the dataset-level PQ is generally not the product of the dataset-level SQ and RQ.
Benchmarks
- PASCAL VOC 2012
- 20 object classes + background; 1,464 train / 1,449 val / 1,456 test images; 10,582 training images with the SBD extra annotations; metric mIoU
- Cityscapes
- street scenes at 2048 × 1024; 5,000 finely annotated images (2,975 / 500 / 1,525) plus 20,000 coarse ones; 19 classes evaluated
- ADE20K
- SceneParse150: 150 classes, 20,210 train / 2,000 val images
- COCO panoptic
- 80 thing + 53 stuff = 133 classes; 118k train / 5k val images; metric PQ (COCO instance masks: mask AP)
| Model | Paper | Idea |
|---|---|---|
| FCN | Long, Shelhamer, Darrell, CVPR 2015 | convolutionalised classifier + upsampling + skip fusion; 62.2 mIoU on VOC 2012 test |
| U-Net | Ronneberger, Fischer, Brox, MICCAI 2015 | symmetric encoder–decoder with concatenated skips; biomedical images |
| DeepLab | Chen et al., 2015–2018 | atrous convolution, ASPP; v3+ reached 89.0 mIoU on VOC 2012 test (JFT pre-training) |
| Mask R-CNN | He, Gkioxari, Dollár, Girshick, ICCV 2017 | Faster R-CNN + a per-RoI mask head; instance segmentation |
| Mask2Former | Cheng et al., CVPR 2022 | queries predict (class, mask) pairs; one model for all three tasks; 57.8 PQ on COCO (Swin-L) |
| Segment Anything Model (SAM) | Kirillov et al., 2023 | promptable masks from points/boxes; trained on SA-1B, 1.1B masks on 11M images |
Worked example: mIoU and PQ by hand
| GT \ pred | road | car | person | IoU = TP / (TP + FP + FN) |
|---|---|---|---|---|
| road | 50 | 5 | 0 | 50 / (50 + 10 + 5) = 0.769 |
| car | 10 | 30 | 0 | 30 / (30 + 7 + 10) = 0.638 |
| person | 0 | 2 | 3 | 3 / (3 + 0 + 2) = 0.600 |
The person class has 5 pixels out of 100, yet its IoU weighs a third of mIoU. For PQ, take the class car with 3 ground-truth and 3 predicted segments. Two pairs match at IoU 0.80 and 0.60. The third prediction overlaps the third car at IoU 0.3, so it is one FP and one FN. SQ = 1.40 / 2 = 0.700, RQ = 2 / (2 + 0.5 + 0.5) = 0.667, PQ = 1.40 / 3 = 0.467 = 0.700 × 0.667.
1import numpy as np23def confusion(gt, pred, K, ignore=255):4 """K×K matrix, rows = ground truth, cols = prediction; void pixels skipped."""5 m = gt != ignore6 return np.bincount(K * gt[m] + pred[m], minlength=K * K).reshape(K, K)78def miou(cm):9 tp = np.diag(cm)10 iou = tp / (cm.sum(0) + cm.sum(1) - tp) # TP / (TP + FP + FN) per class11 return iou, np.nanmean(iou), tp.sum() / cm.sum()1213def pq_one_class(gt_masks, pred_masks):14 """Panoptic quality for one class: IoU > 0.5 makes matches unique."""15 ious = [[(g & p).sum() / (g | p).sum() for p in pred_masks] for g in gt_masks]16 tp = [v for row in ious for v in row if v > 0.5]17 fp, fn = len(pred_masks) - len(tp), len(gt_masks) - len(tp)18 sq = sum(tp) / max(len(tp), 1)19 rq = len(tp) / (len(tp) + fp / 2 + fn / 2)20 return float(sq * rq), float(sq), rq2122cm = np.array([[50, 5, 0], [10, 30, 0], [0, 2, 3]]) # the worked example23iou, m, acc = miou(cm)24print(iou.round(3), round(m, 3), acc) # [0.769 0.638 0.6 ] 0.669 0.832526gt = np.zeros((3, 10, 10), bool); gt[0, 0:4, 0:5] = 1; gt[1, 5:9, 0:4] = 1; gt[2, 6:9, 6:9] = 127pr = np.zeros((3, 10, 10), bool); pr[0, 0:4, 0:4] = 1; pr[1, 5:8, 0:4] = 1; pr[2, 1:4, 6:9] = 128print([round(x, 3) for x in pq_one_class(gt, pr)]) # [0.517, 0.775, 0.667]Check yourself
A panoptic model gets, for class person, two matches with IoU 0.9 and 0.7, one extra predicted segment and no missed people. What is PQ for person?
Check yourself
On a dataset with K = 20 classes, a model never predicts one rare class but is perfect elsewhere. Roughly what happens?
Remember
- ✓Semantic: a class per pixel. Instance: a scored mask per thing, masks may overlap. Panoptic: (class, id) per pixel, stuff + things, no overlap.
- ✓Semantic training is per-pixel cross-entropy over the labelled pixels; void (255) is ignored.
- ✓mIoU: per-class from a dataset-level confusion matrix, then averaged over classes.
- ✓PQ SQ × RQ per class, with matches at IoU > 0.5.
- ✓Benchmarks: VOC 2012 (21 classes), Cityscapes (19), ADE20K (150), COCO panoptic (133).
- ✓Lineage: FCN and U-Net (2015) → DeepLab → Mask R-CNN (2017) → Mask2Former (2022) → Segment Anything Model (SAM) (2023).