Image segmentation

Segmentation labels every pixel: semantic (a class per pixel, scored by mIoU), instance (a mask per object, mask AP) or panoptic (class and id per pixel, PQ = SQ × RQ); per-pixel cross-entropy trains the dense head.

8 min read643 wordscore0 viewsupdated 2026-10-15

Definition

Image segmentation is dense prediction: assign every pixel of an image a label. Semantic segmentation gives each pixel a class. Instance segmentation gives a separate mask to each countable object. Panoptic segmentation (Kirillov, He, Girshick, Rother and Dollár, CVPR 2019) unifies the two: every pixel gets a class and, for object classes, an instance id.

Input: an image 3×H×W3\times H\times W. Output: an H×WH\times W label map, or a set of binary masks. Classification needs one label per image and detection one box per object; segmentation needs the exact outline. The network must see enough context to know what a region is, yet keep enough resolution to say where its border lies. Downsampling helps the first and hurts the second, and that tension shapes every architecture below.

Panoptic segmentation splits classes into things (countable objects: car, person) and stuff (amorphous regions: sky, road, grass). Stuff gets one segment per class; things get one segment per instance. Instance segmentation ignores stuff, and its masks may overlap. Panoptic segments may not.

One scene, three tasks: what each pixel getssemanticskyroadcarcarpersonH×W map of class idsboth cars = one 'car' regionmetric: mIoUinstance(stuff pixels: no output)car .97car .91pers .88N × (mask, class, score)things only; masks may overlapmetric: mask APpanopticsky (stuff)road (stuff)car,1car,2pers,3H×W map of (class, id)stuff + things, no overlapmetric: PQ = SQ × RQDense prediction (semantic head): one K-way classifier per pixelimage3 × H × Wencoder–decoderstride 32 → back to 1logitsK × H × Wargmax / softmaxH × W labelsloss = mean over labelled pixels of −log softmax(z_p)[y_p]; void pixels (label 255) are ignoredFCN 2015 → U-Net 2015 → DeepLab 2015–18 → Mask R-CNN 2017 → Mask2Former 2022 → SAM 2023per-pixel classifiers → per-object masks → one mask classifier for all tasks → promptable
fig.The three tasks on one scene. Semantic output paints both cars with the same class. Instance output is a list of scored masks for things only. Panoptic output labels every pixel with (class, id), so the two cars are separate segments and sky and road are one segment each. Below: the dense prediction head shared by semantic models, a K-way classifier at every pixel.

Training: per-pixel cross-entropy

A semantic model outputs logits z∈RK×H×Wz\in\mathbb R^{K\times H\times W}, usually computed at stride 8 to 32 and upsampled to full resolution. The loss is ordinary cross-entropy at every labelled pixel, averaged:

L=−1∣Ω∣∑p∈Ωlog⁡ezyp,p∑k=1Kezk,p\mathcal L=-\frac{1}{|\Omega|}\sum_{p\in\Omega}\log\frac{e^{z_{y_p,p}}}{\sum_{k=1}^{K}e^{z_{k,p}}}
Ω\Omega
labelled pixels; "void" pixels (often label 255, e.g. object borders in PASCAL VOC) are excluded
ypy_p
ground-truth class of pixel pp
zk,pz_{k,p}
logit of class kk at pixel pp

Large regions dominate this average, so small classes are often under-segmented. Common fixes are class-weighted cross-entropy, the pixel weight map of U-Net, and overlap losses such as Dice loss. Instance models instead predict one binary mask per detected object (Mask R-CNN) or per query (Mask2Former), trained with per-pixel binary cross-entropy, often plus a Dice term.

Metrics: mIoU and PQ

For semantic segmentation, accumulate a K×KK\times K confusion matrix over the whole dataset, then compute IoU per class and average over classes:

IoUk=TPkTPk+FPk+FNk,mIoU=1K∑kIoUk\text{IoU}_k=\frac{TP_k}{TP_k+FP_k+FN_k},\qquad \text{mIoU}=\frac1K\sum_{k}\text{IoU}_k
TPkTP_k
pixels of class kk predicted as kk
FPk, FNkFP_k,\ FN_k
pixels wrongly predicted as kk; pixels of kk predicted as something else

Each class counts equally, so a small class missed entirely costs 1/K1/K of mIoU, however few pixels it has. Pixel accuracy hides that. Instance segmentation is scored like detection, by mask AP: COCO AP averaged over mask-IoU thresholds 0.50 to 0.95. Panoptic quality matches predicted and ground-truth segments of the same class. A match needs IoU > 0.5, which makes matches unique:

PQ=∑(p,g)∈TPIoU(p,g)∣TP∣+12∣FP∣+12∣FN∣=∑TPIoU∣TP∣⏟SQ×∣TP∣∣TP∣+12∣FP∣+12∣FN∣⏟RQ\text{PQ}=\frac{\sum_{(p,g)\in TP}\text{IoU}(p,g)}{|TP|+\tfrac12|FP|+\tfrac12|FN|}=\underbrace{\frac{\sum_{TP}\text{IoU}}{|TP|}}_{\text{SQ}}\times\underbrace{\frac{|TP|}{|TP|+\tfrac12|FP|+\tfrac12|FN|}}_{\text{RQ}}
TP, FP, FNTP,\ FP,\ FN
matched pairs, unmatched predicted segments, unmatched ground-truth segments
SQ\text{SQ}
segmentation quality: mean IoU of the matched segments
RQ\text{RQ}
recognition quality: an F1 score over segments

PQ is computed per class and then averaged over classes. Because SQ and RQ are also averaged per class, the dataset-level PQ is generally not the product of the dataset-level SQ and RQ.

ground truth (panoptic: class + instance id)prediction (dot = wrong class)stuff: sky · road (one segment each) · things: car #1 car #2 person (one segment per instance)pixel accuracy640/640 = 100.0%mIoU (semantic)1.000PQ (panoptic)0.849 SQ (mean)0.899 RQ (mean)0.917classIoUTP FP FNseg IoUsPQsky1.0001 0 01.001.000road1.0001 0 01.001.000car1.0001 0 10.590.396person1.0001 0 01.001.000PQ_c = Σ IoU(TP) / (TP + ½FP + ½FN), then meanevery pixel has the right class (mIoU 1.000), but car instances are wrong: PQ 0.849mIoU sees only per-pixel classes; PQ also checks that each object is one segment matched at IoU > 0.5
instance errors
fig.Edit a prediction and score it three ways. Default (cars merged into one instance): all 640 pixels have the right class, so pixel accuracy is 100% and mIoU 1.000. PQ is 0.849: the merged segment matches car #1 at IoU 0.59, car #2 becomes a false negative, and car PQ is 0.396. With only "miss the person" on, accuracy is still 94.8% but mIoU and PQ fall to 0.717. Thing boundary −1 px (merge off) shrinks car #2 to IoU 0.47 and the thin person to 0.27: neither matches, each costs one FP and one FN, and PQ is 0.500 while mIoU is 0.629.

Benchmarks

PASCAL VOC 2012
20 object classes + background; 1,464 train / 1,449 val / 1,456 test images; 10,582 training images with the SBD extra annotations; metric mIoU
Cityscapes
street scenes at 2048 × 1024; 5,000 finely annotated images (2,975 / 500 / 1,525) plus 20,000 coarse ones; 19 classes evaluated
ADE20K
SceneParse150: 150 classes, 20,210 train / 2,000 val images
COCO panoptic
80 thing + 53 stuff = 133 classes; 118k train / 5k val images; metric PQ (COCO instance masks: mask AP)
ModelPaperIdea
FCNLong, Shelhamer, Darrell, CVPR 2015convolutionalised classifier + upsampling + skip fusion; 62.2 mIoU on VOC 2012 test
U-NetRonneberger, Fischer, Brox, MICCAI 2015symmetric encoder–decoder with concatenated skips; biomedical images
DeepLabChen et al., 2015–2018atrous convolution, ASPP; v3+ reached 89.0 mIoU on VOC 2012 test (JFT pre-training)
Mask R-CNNHe, Gkioxari, Dollár, Girshick, ICCV 2017Faster R-CNN + a per-RoI mask head; instance segmentation
Mask2FormerCheng et al., CVPR 2022queries predict (class, mask) pairs; one model for all three tasks; 57.8 PQ on COCO (Swin-L)
Segment Anything Model (SAM)Kirillov et al., 2023promptable masks from points/boxes; trained on SA-1B, 1.1B masks on 11M images
The family, one line each

Worked example: mIoU and PQ by hand

GT \ predroadcarpersonIoU = TP / (TP + FP + FN)
road505050 / (50 + 10 + 5) = 0.769
car1030030 / (30 + 7 + 10) = 0.638
person0233 / (3 + 0 + 2) = 0.600
A 100-pixel confusion matrix (rows = ground truth, columns = prediction). Pixel accuracy 83/100 = 0.83, mIoU = 0.669

The person class has 5 pixels out of 100, yet its IoU weighs a third of mIoU. For PQ, take the class car with 3 ground-truth and 3 predicted segments. Two pairs match at IoU 0.80 and 0.60. The third prediction overlaps the third car at IoU 0.3, so it is one FP and one FN. SQ = 1.40 / 2 = 0.700, RQ = 2 / (2 + 0.5 + 0.5) = 0.667, PQ = 1.40 / 3 = 0.467 = 0.700 × 0.667.

python
1import numpy as np
2
3def confusion(gt, pred, K, ignore=255):
4 """K×K matrix, rows = ground truth, cols = prediction; void pixels skipped."""
5 m = gt != ignore
6 return np.bincount(K * gt[m] + pred[m], minlength=K * K).reshape(K, K)
7
8def miou(cm):
9 tp = np.diag(cm)
10 iou = tp / (cm.sum(0) + cm.sum(1) - tp) # TP / (TP + FP + FN) per class
11 return iou, np.nanmean(iou), tp.sum() / cm.sum()
12
13def pq_one_class(gt_masks, pred_masks):
14 """Panoptic quality for one class: IoU > 0.5 makes matches unique."""
15 ious = [[(g & p).sum() / (g | p).sum() for p in pred_masks] for g in gt_masks]
16 tp = [v for row in ious for v in row if v > 0.5]
17 fp, fn = len(pred_masks) - len(tp), len(gt_masks) - len(tp)
18 sq = sum(tp) / max(len(tp), 1)
19 rq = len(tp) / (len(tp) + fp / 2 + fn / 2)
20 return float(sq * rq), float(sq), rq
21
22cm = np.array([[50, 5, 0], [10, 30, 0], [0, 2, 3]]) # the worked example
23iou, m, acc = miou(cm)
24print(iou.round(3), round(m, 3), acc) # [0.769 0.638 0.6 ] 0.669 0.83
25
26gt = np.zeros((3, 10, 10), bool); gt[0, 0:4, 0:5] = 1; gt[1, 5:9, 0:4] = 1; gt[2, 6:9, 6:9] = 1
27pr = np.zeros((3, 10, 10), bool); pr[0, 0:4, 0:4] = 1; pr[1, 5:8, 0:4] = 1; pr[2, 1:4, 6:9] = 1
28print([round(x, 3) for x in pq_one_class(gt, pr)]) # [0.517, 0.775, 0.667]
mIoU from a confusion matrix and PQ for one class (numpy, runs as is)

Check yourself

A panoptic model gets, for class person, two matches with IoU 0.9 and 0.7, one extra predicted segment and no missed people. What is PQ for person?

Check yourself

On a dataset with K = 20 classes, a model never predicts one rare class but is perfect elsewhere. Roughly what happens?

Remember

  • ✓Semantic: a class per pixel. Instance: a scored mask per thing, masks may overlap. Panoptic: (class, id) per pixel, stuff + things, no overlap.
  • ✓Semantic training is per-pixel cross-entropy over the labelled pixels; void (255) is ignored.
  • ✓mIoU: per-class TP/(TP+FP+FN)TP/(TP+FP+FN) from a dataset-level confusion matrix, then averaged over classes.
  • ✓PQ =∑IoU/(∣TP∣+12∣FP∣+12∣FN∣)==\sum\text{IoU}/(|TP|+\tfrac12|FP|+\tfrac12|FN|)= SQ × RQ per class, with matches at IoU > 0.5.
  • ✓Benchmarks: VOC 2012 (21 classes), Cityscapes (19), ADE20K (150), COCO panoptic (133).
  • ✓Lineage: FCN and U-Net (2015) → DeepLab → Mask R-CNN (2017) → Mask2Former (2022) → Segment Anything Model (SAM) (2023).
← back to Deep Learning roadmap