# Segment Anything Model (SAM)

> SAM (Kirillov et al., 2023) segments whatever a point, box or mask prompt points at: a ViT-H encoder runs once per image, a light decoder returns 3 ranked masks per prompt; trained on SA-1B, 1.1B masks on 11M images.

Source: https://www.mlfoundry.in/learn/segment-anything-model-sam  
Section: Deep Learning › Deep Learning for Computer Vision  
Level: core · 8 min read · updated 2026-10-15

**Segment Anything Model:** (SAM) is a promptable segmentation model from Meta AI: given an image and a prompt (foreground and background points, a box, or a rough mask), it returns a mask of the object the prompt refers to. It was introduced in "Segment Anything" by Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár and Ross Girshick (ICCV 2023), together with the SA-1B dataset.

Earlier segmentation models such as **Mask R-CNN** and **Mask2Former** are trained on a fixed label set (80 COCO classes, for example) and segment only those. Interactive tools segment arbitrary objects from clicks but were trained on small datasets. SAM's goal was a **foundation model** for segmentation: one network that transfers zero-shot to new image domains and tasks through prompting.

The task is defined as **promptable segmentation**: for any prompt, output a *valid* mask. Valid means that when a prompt is ambiguous, the mask is reasonable for at least one of the objects it could mean. A click on a shirt logo could mean the logo, the shirt or the person.

## Architecture: heavy once, light per prompt

The **image encoder** is a **Vision Transformers (ViT)** ViT-H/16 pretrained as a masked autoencoder (MAE). The image is resized so its longer side is 1,024 and padded to $1024\times1024$, giving $64\times64$ patch tokens. Most blocks use $14\times14$ windowed attention, with four equally spaced global-attention blocks. A neck of $1\times1$ and $3\times3$ convolutions reduces the output to a $64\times64\times256$ embedding.

The **prompt encoder** turns each point into a positional encoding of its coordinates plus a learned embedding saying whether it is a foreground point, a background point or a box corner. A box becomes two corner tokens. A mask prompt is downscaled by convolutions and added element-wise to the image embedding. The paper also explored free-form text through a CLIP text encoder; that variant was not released.

$$
\mathrm{PE}(\mathbf p)=\big[\cos(2\pi B\mathbf p),\ \sin(2\pi B\mathbf p)\big],\qquad \text{token}=\mathrm{PE}(\mathbf p)+e_{\text{type}}
$$

- $\mathbf p$: point coordinates normalised to $[0,1]^2$ (in the code, mapped to $[-1,1]$ first)
- $B$: a fixed random Gaussian matrix of spatial frequencies ($2\times128$, so the token has 256 dimensions)
- $e_{\text{type}}$: learned embedding: positive point, negative point, top-left corner or bottom-right corner

The **mask decoder** is a two-layer transformer over a small set of tokens: the prompt tokens plus learned output tokens (mask tokens and one IoU token). Each layer applies self-attention over the tokens, **Cross-attention** from tokens to the image embedding, an MLP, and cross-attention back from the image to the tokens. The embedding is then upsampled $4\times$ by transposed convolutions, and each mask token, passed through an MLP, is dot-multiplied with it at every location to give a $256\times256$ mask logit map.

> Figure: SAM's split. The ViT-H image encoder (632M parameters) runs once per image and yields a 64×64×256 embedding. For each prompt, the prompt encoder makes tokens from points and boxes (a mask prompt is added to the embedding), and the two-layer two-way mask decoder returns three masks with predicted-IoU scores ŝ. A click on the logo is ambiguous, so the three outputs cover the whole person, the shirt and the logo.

## Ambiguity: three masks and an IoU head

A model with a single output, trained on ambiguous prompts, learns to average the plausible masks. SAM instead predicts **three** masks per prompt, which in practice specialise into whole, part and subpart. During training only the mask with the lowest loss is backpropagated. A small head predicts each mask's IoU with the target, so the masks can be ranked at inference.

$$
\mathcal L=\min_{k\in\{1,2,3\}}\Big[\,20\cdot\mathrm{FL}(m_k,y)+\mathrm{Dice}(m_k,y)\Big]+\sum_{k}\big(\hat s_k-\mathrm{IoU}(m_k,y)\big)^2
$$

- $m_k,\ y$: predicted mask $k$ and the ground-truth mask for this prompt
- $\mathrm{FL},\ \mathrm{Dice}$: per-pixel **Focal loss** and **Dice loss**, weighted 20 : 1
- $\hat s_k$: predicted IoU of mask $k$; regressed with a mean-squared error to the actual IoU

With more than one prompt (say, two clicks or a box), ambiguity is rare and three similar masks would be redundant. SAM then returns a single mask from a fourth output token. Training simulates interactive use: prompts are sampled over 11 rounds per mask, with later clicks placed in the error region of the previous prediction.

**Data engine and SA-1B**

- **Stage 1, assisted-manual:** annotators click with an early SAM in a browser tool; 4.3M masks on 120k images
- **Stage 2, semi-automatic:** SAM pre-fills confident masks, annotators add the missing objects; 5.9M more masks on 180k images
- **Stage 3, fully automatic:** a 32×32 grid of points prompts SAM on each image; confident, stable masks are kept after NMS
- **SA-1B:** 11M licensed, privacy-protected images and 1.1B masks (about 100 per image), 99.1% generated fully automatically
- **Zero-shot:** single-point segmentation on 23 datasets: SAM beat the RITM interactive baseline on 16 of 23
- **Speed:** prompt encoder and mask decoder run in about 50 ms in a web browser on CPU, once the image embedding exists

> Figure: SAM's min-over-three loss on a toy 20×20 image. The three output masks are the nested objects under the click with seeded noise; the loss is 20·focal + dice. At the default (click on the logo, annotator meant the shirt), mask 2 wins with IoU 0.868 and loss 0.437 against 3.617 and 2.337, so only mask 2 is trained. A single-output model averages the three and scores IoU 0.694. Averaged over the three plausible targets, the expected loss is 0.543 with three outputs and 1.457 with one. Click the box (unambiguous) and the gap vanishes: 0.425 vs 0.409.

## Worked example: why windowed attention

At $1024\times1024$ with $16\times16$ patches there are $64^2=4{,}096$ tokens. A global attention layer compares every pair. The windowed layers pad the grid to $70\times70$ and attend inside $14\times14$ windows.

Attention pairs per head in the ViT-H encoder (32 blocks: 4 global, 28 windowed)

| Layer type | Computation | Pairs per layer |
| --- | --- | --- |
| global | 4,096² | 16,777,216 |
| windowed 14×14 | (70 / 14)² = 25 windows × 196² | 960,400 (17.5× fewer) |
| SAM encoder, all 32 blocks | 4 × 16,777,216 + 28 × 960,400 | 94,000,064 |
| all-global encoder | 32 × 16,777,216 | 536,870,912 (5.7× more) |

Even so, the four global blocks account for 71% of the attention pairs, and the encoder dominates the cost. That is why SAM computes the embedding once per image: every further click costs only the small decoder.

```python
import numpy as np, cv2
from segment_anything import sam_model_registry, SamPredictor, SamAutomaticMaskGenerator

sam = sam_model_registry["vit_h"](checkpoint="sam_vit_h_4b8939.pth").to("cuda")
predictor = SamPredictor(sam)
image = cv2.cvtColor(cv2.imread("street.jpg"), cv2.COLOR_BGR2RGB)
predictor.set_image(image)                       # runs the ViT-H image encoder once

# one ambiguous click -> three masks with predicted IoU scores
masks, scores, logits = predictor.predict(point_coords=np.array([[500, 375]]),
                                          point_labels=np.array([1]), multimask_output=True)
print(masks.shape, scores)                       # (3, H, W) booleans, 3 predicted IoUs
best = int(np.argmax(scores))

# refine: add a background click and feed back the previous low-res logits; one mask out
mask, score, _ = predictor.predict(point_coords=np.array([[500, 375], [560, 300]]),
                                   point_labels=np.array([1, 0]),
                                   mask_input=logits[best][None], multimask_output=False)

# a box prompt, and the "segment everything" mode used in the data engine
box_mask, _, _ = predictor.predict(box=np.array([425, 600, 700, 875]), multimask_output=False)
all_masks = SamAutomaticMaskGenerator(sam, points_per_side=32).generate(image)
```

Point and box prompting with Meta's segment-anything package (not run here: needs PyTorch and the ViT-H checkpoint, about 2.5 GB)

> **SAM does not know what things are:** SAM outputs masks, not class labels. To get "all cars", pair it with a detector or an open-vocabulary model that supplies boxes (for example **CLIP**-based detectors), and let SAM turn each box into a mask. Its masks are also coarser on thin structures than a specialist model trained for the domain, such as medical images.

> **SAM 2:** "SAM 2: Segment Anything in Images and Videos" (Ravi et al., 2024) extends the same prompt-and-decode design to video. A streaming memory of past frames and prompts lets a click on one frame propagate a mask through the clip. It was trained with a video data engine on the new SA-V dataset.

**Question:** Why does SAM train only the lowest-loss of its three output masks rather than all three?

- A. To save compute during backpropagation
- B. So that each output can specialise to a different valid interpretation of an ambiguous prompt instead of all averaging them
- C. Because the IoU head needs one fixed target
- D. To make the masks binary

**Answer:** B. So that each output can specialise to a different valid interpretation of an ambiguous prompt instead of all averaging them — If all three outputs were trained on every target, they would each converge to the same average of the plausible masks, as the single-output model in the figure does. With the min, an output is only pulled toward targets it is already closest to, so whole, part and subpart masks emerge, and the IoU head ranks them.

**Question:** A user makes 20 clicks on the same image. How many times does SAM run its ViT-H image encoder?

- A. 20
- B. Once
- C. 60, three per click
- D. Never; the encoder is only used in training

**Answer:** B. Once — The $64\times64\times256$ embedding depends only on the image, so it is computed once (`set_image`). Each click reruns only the prompt encoder and the light mask decoder, which is what makes interactive use possible.

**Key points**

- Promptable segmentation: points, boxes or masks in, a valid mask out, with no class labels.
- MAE-pretrained ViT-H/16 encoder at 1024×1024 → 64×64×256 embedding, computed once per image.
- Two-layer two-way transformer decoder; 3 masks per ambiguous prompt, min-loss training, IoU head to rank them.
- Loss: 20·focal + dice on the best mask, plus MSE on the predicted IoU.
- Data engine (assisted-manual → semi-automatic → fully automatic) produced SA-1B: 11M images, 1.1B masks.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
