Segment Anything (SAM) and Dense Prediction

Segmentation used to be the most data-hungry task in vision. Every new domain - aerial imagery, medical scans, retail catalogues - meant collecting hundreds of thousands of pixel-perfect masks and training a fresh U-Net. Kirillov et al's 2023 Segment Anything paper changed the framing: train one model on a billion masks, make segmentation promptable (give me the mask for this point, this box, this region), and ship a foundation model that generalises to new domains zero-shot. SAM is to segmentation what CLIP is to classification.

Promptable segmentation as a foundation task

The task definition is deceptively simple. Given an image and a prompt (point, box, mask, or text), return a valid mask. "Valid" means the model picks the most sensible segment containing the prompt even when ambiguous - if you click on a person's shirt, the model should be able to return the shirt, the upper body, or the whole person. SAM emits three candidate masks per prompt with confidence scores to handle this ambiguity explicitly.

This framing has three consequences:

• It is interactive by construction. The model is built to be steered, which is exactly the workflow product teams want.
• It is task-general. The same model handles instance segmentation, object boundary refinement, and selection tooling.
• It is data-bootstrappable. The promptable interface lets you build a data engine where the model proposes masks, humans correct them, and corrections feed back into training. SAM's SA-1B dataset (1.1B masks on 11M images) was built this way.

The encoder + lightweight decoder split

SAM's architecture is designed around a specific deployment pattern: the image is encoded once, then prompted many times.

The image encoder produces a 64x64x256 feature grid that gets cached. Every subsequent click, box, or refinement uses the cached features and runs only the cheap decoder. Interactive selection becomes responsive - sub-100ms per click - because the heavy lifting happens once upfront.

The mask decoder is a small two-way transformer: image features cross-attend to prompt tokens, prompt tokens cross-attend to image features, and output tokens predict the mask via a small MLP. The whole decoder is intentionally tiny so that prompt latency stays interactive.

SAM 2 and video

SAM 2 (Ravi et al, August 2024) extends the promptable design to video. The core changes:

• Streaming memory. A small memory bank holds features from recent frames; the current frame attends to it. Mask propagation across frames is automatic.
• Single unified model for image and video. A single image is just a one-frame video.
• Trained on SA-V, a 51k-video, 643k-masklet dataset built with a new video data engine.