Definition
Segment Anything Model (SAM) is a promptable segmentation model from Meta AI: given an image and a prompt (foreground and background points, a box, or a rough mask), it returns a mask of the object the prompt refers to. It was introduced in "Segment Anything" by Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár and Ross Girshick (ICCV 2023), together with the SA-1B dataset.
Earlier segmentation models such as Mask R-CNN and Mask2Former are trained on a fixed label set (80 COCO classes, for example) and segment only those. Interactive tools segment arbitrary objects from clicks but were trained on small datasets. SAM's goal was a foundation model for segmentation: one network that transfers zero-shot to new image domains and tasks through prompting.
The task is defined as promptable segmentation: for any prompt, output a valid mask. Valid means that when a prompt is ambiguous, the mask is reasonable for at least one of the objects it could mean. A click on a shirt logo could mean the logo, the shirt or the person.
Architecture: heavy once, light per prompt
The image encoder is a Vision Transformers (ViT) ViT-H/16 pretrained as a masked autoencoder (MAE). The image is resized so its longer side is 1,024 and padded to , giving patch tokens. Most blocks use windowed attention, with four equally spaced global-attention blocks. A neck of and convolutions reduces the output to a embedding.
The prompt encoder turns each point into a positional encoding of its coordinates plus a learned embedding saying whether it is a foreground point, a background point or a box corner. A box becomes two corner tokens. A mask prompt is downscaled by convolutions and added element-wise to the image embedding. The paper also explored free-form text through a CLIP text encoder; that variant was not released.
- point coordinates normalised to (in the code, mapped to first)
- a fixed random Gaussian matrix of spatial frequencies (, so the token has 256 dimensions)
- learned embedding: positive point, negative point, top-left corner or bottom-right corner
The mask decoder is a two-layer transformer over a small set of tokens: the prompt tokens plus learned output tokens (mask tokens and one IoU token). Each layer applies self-attention over the tokens, Cross-attention from tokens to the image embedding, an MLP, and cross-attention back from the image to the tokens. The embedding is then upsampled by transposed convolutions, and each mask token, passed through an MLP, is dot-multiplied with it at every location to give a mask logit map.
Ambiguity: three masks and an IoU head
A model with a single output, trained on ambiguous prompts, learns to average the plausible masks. SAM instead predicts three masks per prompt, which in practice specialise into whole, part and subpart. During training only the mask with the lowest loss is backpropagated. A small head predicts each mask's IoU with the target, so the masks can be ranked at inference.
- predicted mask and the ground-truth mask for this prompt
- per-pixel Focal loss and Dice loss, weighted 20 : 1
- predicted IoU of mask ; regressed with a mean-squared error to the actual IoU
With more than one prompt (say, two clicks or a box), ambiguity is rare and three similar masks would be redundant. SAM then returns a single mask from a fourth output token. Training simulates interactive use: prompts are sampled over 11 rounds per mask, with later clicks placed in the error region of the previous prediction.
Data engine and SA-1B
- Stage 1, assisted-manual
- annotators click with an early SAM in a browser tool; 4.3M masks on 120k images
- Stage 2, semi-automatic
- SAM pre-fills confident masks, annotators add the missing objects; 5.9M more masks on 180k images
- Stage 3, fully automatic
- a 32×32 grid of points prompts SAM on each image; confident, stable masks are kept after NMS
- SA-1B
- 11M licensed, privacy-protected images and 1.1B masks (about 100 per image), 99.1% generated fully automatically
- Zero-shot
- single-point segmentation on 23 datasets: SAM beat the RITM interactive baseline on 16 of 23
- Speed
- prompt encoder and mask decoder run in about 50 ms in a web browser on CPU, once the image embedding exists
Worked example: why windowed attention
At with patches there are tokens. A global attention layer compares every pair. The windowed layers pad the grid to and attend inside windows.
| Layer type | Computation | Pairs per layer |
|---|---|---|
| global | 4,096² | 16,777,216 |
| windowed 14×14 | (70 / 14)² = 25 windows × 196² | 960,400 (17.5× fewer) |
| SAM encoder, all 32 blocks | 4 × 16,777,216 + 28 × 960,400 | 94,000,064 |
| all-global encoder | 32 × 16,777,216 | 536,870,912 (5.7× more) |
Even so, the four global blocks account for 71% of the attention pairs, and the encoder dominates the cost. That is why SAM computes the embedding once per image: every further click costs only the small decoder.
1import numpy as np, cv22from segment_anything import sam_model_registry, SamPredictor, SamAutomaticMaskGenerator34sam = sam_model_registry["vit_h"](checkpoint="sam_vit_h_4b8939.pth").to("cuda")5predictor = SamPredictor(sam)6image = cv2.cvtColor(cv2.imread("street.jpg"), cv2.COLOR_BGR2RGB)7predictor.set_image(image) # runs the ViT-H image encoder once89# one ambiguous click -> three masks with predicted IoU scores10masks, scores, logits = predictor.predict(point_coords=np.array([[500, 375]]),11 point_labels=np.array([1]), multimask_output=True)12print(masks.shape, scores) # (3, H, W) booleans, 3 predicted IoUs13best = int(np.argmax(scores))1415# refine: add a background click and feed back the previous low-res logits; one mask out16mask, score, _ = predictor.predict(point_coords=np.array([[500, 375], [560, 300]]),17 point_labels=np.array([1, 0]),18 mask_input=logits[best][None], multimask_output=False)1920# a box prompt, and the "segment everything" mode used in the data engine21box_mask, _, _ = predictor.predict(box=np.array([425, 600, 700, 875]), multimask_output=False)22all_masks = SamAutomaticMaskGenerator(sam, points_per_side=32).generate(image)Check yourself
Why does SAM train only the lowest-loss of its three output masks rather than all three?
Check yourself
A user makes 20 clicks on the same image. How many times does SAM run its ViT-H image encoder?
Remember
- ✓Promptable segmentation: points, boxes or masks in, a valid mask out, with no class labels.
- ✓MAE-pretrained ViT-H/16 encoder at 1024×1024 → 64×64×256 embedding, computed once per image.
- ✓Two-layer two-way transformer decoder; 3 masks per ambiguous prompt, min-loss training, IoU head to rank them.
- ✓Loss: 20·focal + dice on the best mask, plus MSE on the predicted IoU.
- ✓Data engine (assisted-manual → semi-automatic → fully automatic) produced SA-1B: 11M images, 1.1B masks.