Segment Anything Model (SAM)

SAM (Kirillov et al., 2023) segments whatever a point, box or mask prompt points at: a ViT-H encoder runs once per image, a light decoder returns 3 ranked masks per prompt; trained on SA-1B, 1.1B masks on 11M images.

8 min read755 wordscore0 viewsupdated 2026-10-15

Definition

Segment Anything Model (SAM) is a promptable segmentation model from Meta AI: given an image and a prompt (foreground and background points, a box, or a rough mask), it returns a mask of the object the prompt refers to. It was introduced in "Segment Anything" by Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár and Ross Girshick (ICCV 2023), together with the SA-1B dataset.

Earlier segmentation models such as Mask R-CNN and Mask2Former are trained on a fixed label set (80 COCO classes, for example) and segment only those. Interactive tools segment arbitrary objects from clicks but were trained on small datasets. SAM's goal was a foundation model for segmentation: one network that transfers zero-shot to new image domains and tasks through prompting.

The task is defined as promptable segmentation: for any prompt, output a valid mask. Valid means that when a prompt is ambiguous, the mask is reasonable for at least one of the objects it could mean. A click on a shirt logo could mean the logo, the shirt or the person.

Architecture: heavy once, light per prompt

The image encoder is a Vision Transformers (ViT) ViT-H/16 pretrained as a masked autoencoder (MAE). The image is resized so its longer side is 1,024 and padded to 1024×10241024\times1024, giving 64×6464\times64 patch tokens. Most blocks use 14×1414\times14 windowed attention, with four equally spaced global-attention blocks. A neck of 1×11\times1 and 3×33\times3 convolutions reduces the output to a 64×64×25664\times64\times256 embedding.

The prompt encoder turns each point into a positional encoding of its coordinates plus a learned embedding saying whether it is a foreground point, a background point or a box corner. A box becomes two corner tokens. A mask prompt is downscaled by convolutions and added element-wise to the image embedding. The paper also explored free-form text through a CLIP text encoder; that variant was not released.

PE(p)=[cos⁡(2πBp), sin⁡(2πBp)],token=PE(p)+etype\mathrm{PE}(\mathbf p)=\big[\cos(2\pi B\mathbf p),\ \sin(2\pi B\mathbf p)\big],\qquad \text{token}=\mathrm{PE}(\mathbf p)+e_{\text{type}}
p\mathbf p
point coordinates normalised to [0,1]2[0,1]^2 (in the code, mapped to [−1,1][-1,1] first)
BB
a fixed random Gaussian matrix of spatial frequencies (2×1282\times128, so the token has 256 dimensions)
etypee_{\text{type}}
learned embedding: positive point, negative point, top-left corner or bottom-right corner

The mask decoder is a two-layer transformer over a small set of tokens: the prompt tokens plus learned output tokens (mask tokens and one IoU token). Each layer applies self-attention over the tokens, Cross-attention from tokens to the image embedding, an MLP, and cross-attention back from the image to the tokens. The embedding is then upsampled 4×4\times by transposed convolutions, and each mask token, passed through an MLP, is dot-multiplied with it at every location to give a 256×256256\times256 mask logit map.

SAM: heavy image encoder once per image, light prompt encoder + mask decoder per prompt1024×1024×3image encoder: ViT-H/16MAE-pretrained, 632M params64×64 patch tokens, dim 128014×14 windowed attention +4 global-attention blocksneck: 1×1 + 3×3 conv → 256image embedding64×64×256computed onceprompt encoder (per prompt)point (x, y, ±)Fourier PE + type emb.box2 corner tokens (PE)text (explored)CLIP text embeddingmask (dense prompt)convs ↓4 → 64×64×256added to the image embeddingmask decoder (2 layers)tokens: 4 mask + 1 IoU + promptsself-attn over tokenscross-attn token → imageMLP on tokenscross-attn image → token↑4 transposed convs: 256²×32mask = MLP(token) · embeddingperson · ŝ1shirt · ŝ2logo · ŝ33 masksfor oneclickŝ = predicted IoU, usedto rank the 3 masks256×256 → image sizetraining: 20·focal + dice on the lowest-loss mask only · IoU head: MSE · 11 prompt roundsdata engine: assisted-manual → semi-automatic → fully automatic (32×32 points)→ SA-1B: 11M images, 1.1B masks · prompt encoder + decoder ≈ 50 ms in a browser
fig.SAM's split. The ViT-H image encoder (632M parameters) runs once per image and yields a 64×64×256 embedding. For each prompt, the prompt encoder makes tokens from points and boxes (a mask prompt is added to the embedding), and the two-layer two-way mask decoder returns three masks with predicted-IoU scores ŝ. A click on the logo is ambiguous, so the three outputs cover the whole person, the shirt and the logo.

Ambiguity: three masks and an IoU head

A model with a single output, trained on ambiguous prompts, learns to average the plausible masks. SAM instead predicts three masks per prompt, which in practice specialise into whole, part and subpart. During training only the mask with the lowest loss is backpropagated. A small head predicts each mask's IoU with the target, so the masks can be ranked at inference.

L=min⁡k∈{1,2,3}[ 20⋅FL(mk,y)+Dice(mk,y)]+∑k(s^k−IoU(mk,y))2\mathcal L=\min_{k\in\{1,2,3\}}\Big[\,20\cdot\mathrm{FL}(m_k,y)+\mathrm{Dice}(m_k,y)\Big]+\sum_{k}\big(\hat s_k-\mathrm{IoU}(m_k,y)\big)^2
mk, ym_k,\ y
predicted mask kk and the ground-truth mask for this prompt
FL, Dice\mathrm{FL},\ \mathrm{Dice}
per-pixel Focal loss and Dice loss, weighted 20 : 1
s^k\hat s_k
predicted IoU of mask kk; regressed with a mean-squared error to the actual IoU

With more than one prompt (say, two clicks or a box), ambiguity is rare and three similar masks would be redundant. SAM then returns a single mask from a fourth output token. Training simulates interactive use: prompts are sampled over 11 rounds per mask, with later clicks placed in the error region of the previous prediction.

Data engine and SA-1B

Stage 1, assisted-manual
annotators click with an early SAM in a browser tool; 4.3M masks on 120k images
Stage 2, semi-automatic
SAM pre-fills confident masks, annotators add the missing objects; 5.9M more masks on 180k images
Stage 3, fully automatic
a 32×32 grid of points prompts SAM on each image; confident, stable masks are kept after NMS
SA-1B
11M licensed, privacy-protected images and 1.1B masks (about 100 per image), 99.1% generated fully automatically
Zero-shot
single-point segmentation on 23 datasets: SAM beat the RITM interactive baseline on 16 of 23
Speed
prompt encoder and mask decoder run in about 50 ms in a web browser on CPU, once the image embedding exists
one click on the logo · the annotator meant the shirtloss = 20·focal(γ=2) + dice · only the lowest-loss mask is trained · orange = intendedlogo on shirt + head + legs= person · separate boxmask 1: logoIoU 0.11120·focal 2.951dice 0.666loss 3.617no gradientmask 2: shirtIoU 0.86820·focal 0.156dice 0.281loss 0.437▲ gets the gradientmask 3: personIoU 0.42020·focal 1.832dice 0.505loss 2.337no gradientIoU head: regresses each mask's IoU, ranks themsingle-output modellearns the average of the3 targets it is shown: a blurIoU vs shirt 0.694 · loss 0.775expected loss over the intendedobjects (logo, shirt, person):SAM min-of-3 0.543 · single 1.4573 plausible objects under one click: three outputs let each specialise (subpart, part, whole)
click
annotator
fig.SAM's min-over-three loss on a toy 20×20 image. The three output masks are the nested objects under the click with seeded noise; the loss is 20·focal + dice. At the default (click on the logo, annotator meant the shirt), mask 2 wins with IoU 0.868 and loss 0.437 against 3.617 and 2.337, so only mask 2 is trained. A single-output model averages the three and scores IoU 0.694. Averaged over the three plausible targets, the expected loss is 0.543 with three outputs and 1.457 with one. Click the box (unambiguous) and the gap vanishes: 0.425 vs 0.409.

Worked example: why windowed attention

At 1024×10241024\times1024 with 16×1616\times16 patches there are 642=4,09664^2=4{,}096 tokens. A global attention layer compares every pair. The windowed layers pad the grid to 70×7070\times70 and attend inside 14×1414\times14 windows.

Layer typeComputationPairs per layer
global4,096²16,777,216
windowed 14×14(70 / 14)² = 25 windows × 196²960,400 (17.5× fewer)
SAM encoder, all 32 blocks4 × 16,777,216 + 28 × 960,40094,000,064
all-global encoder32 × 16,777,216536,870,912 (5.7× more)
Attention pairs per head in the ViT-H encoder (32 blocks: 4 global, 28 windowed)

Even so, the four global blocks account for 71% of the attention pairs, and the encoder dominates the cost. That is why SAM computes the embedding once per image: every further click costs only the small decoder.

python
1import numpy as np, cv2
2from segment_anything import sam_model_registry, SamPredictor, SamAutomaticMaskGenerator
3
4sam = sam_model_registry["vit_h"](checkpoint="sam_vit_h_4b8939.pth").to("cuda")
5predictor = SamPredictor(sam)
6image = cv2.cvtColor(cv2.imread("street.jpg"), cv2.COLOR_BGR2RGB)
7predictor.set_image(image) # runs the ViT-H image encoder once
8
9# one ambiguous click -> three masks with predicted IoU scores
10masks, scores, logits = predictor.predict(point_coords=np.array([[500, 375]]),
11 point_labels=np.array([1]), multimask_output=True)
12print(masks.shape, scores) # (3, H, W) booleans, 3 predicted IoUs
13best = int(np.argmax(scores))
14
15# refine: add a background click and feed back the previous low-res logits; one mask out
16mask, score, _ = predictor.predict(point_coords=np.array([[500, 375], [560, 300]]),
17 point_labels=np.array([1, 0]),
18 mask_input=logits[best][None], multimask_output=False)
19
20# a box prompt, and the "segment everything" mode used in the data engine
21box_mask, _, _ = predictor.predict(box=np.array([425, 600, 700, 875]), multimask_output=False)
22all_masks = SamAutomaticMaskGenerator(sam, points_per_side=32).generate(image)
Point and box prompting with Meta's segment-anything package (not run here: needs PyTorch and the ViT-H checkpoint, about 2.5 GB)

Check yourself

Why does SAM train only the lowest-loss of its three output masks rather than all three?

Check yourself

A user makes 20 clicks on the same image. How many times does SAM run its ViT-H image encoder?

Remember

  • ✓Promptable segmentation: points, boxes or masks in, a valid mask out, with no class labels.
  • ✓MAE-pretrained ViT-H/16 encoder at 1024×1024 → 64×64×256 embedding, computed once per image.
  • ✓Two-layer two-way transformer decoder; 3 masks per ambiguous prompt, min-loss training, IoU head to rank them.
  • ✓Loss: 20·focal + dice on the best mask, plus MSE on the predicted IoU.
  • ✓Data engine (assisted-manual → semi-automatic → fully automatic) produced SA-1B: 11M images, 1.1B masks.
← back to Deep Learning roadmap