¶ SAM 3: Segment Anything with Concepts
Segment Anything Model 3 (SAM 3) is a unified foundation model for promptable segmentation in images and videos. It was introduced in the paper SAM 3: Segment Anything with Concepts by Carion, Gustafson, Hu, Debnath, Hu, Suris, Ryali, Alwala, Khedr, Huang, Lei, Ma, Guo, Kalla, Marks, Greer, Wang, Sun, Rädle, Afouras, Mavroudi, Xu, Wu, Zhou, Momeni, Hazra, Ding, Vaze, Porcher, Li, Li, Kamath, Cheng, Dollár, Ravi, Saenko, Zhang, and Feichtenhofer (2025).
Code, checkpoints, and example notebooks are publicly available in the official GitHub repository.
¶ Model description
SAM 3 is a unified model for promptable segmentation, detection, and tracking in both images and videos. It extends the Segment Anything family beyond interactive segmentation by allowing users to specify objects not only with points, boxes, and masks, but also with short text prompts or visual exemplars.
Compared to SAM 2, SAM 3 introduces the ability to exhaustively segment all instances of an open-vocabulary concept. This means it can identify and segment all objects matching a phrase such as “red car,” “player in white,” or “traffic cone,” even when the concept is not part of a fixed predefined label set.
Key traits of SAM 3:
- Unified image-and-video model: Supports object detection, segmentation, and tracking in one framework.
- Flexible prompting: Accepts text prompts, exemplar prompts, and standard visual prompts such as points, boxes, and masks.
- Open-vocabulary segmentation: Can handle a very large concept space defined by natural-language phrases.
- Decoupled detector-tracker design: Uses separate detector and tracker components with a shared vision encoder, reducing task interference.
- Presence token: Improves discrimination between closely related prompts, such as “a player in white” versus “a player in red.”
¶ SAM 3
Figure 1 (from the paper) illustrates the architecture and workflow of SAM 3:
- A shared vision encoder processes the image or video input.
- A DETR-based detector is conditioned on text, geometry, and image exemplars to identify and segment all matching instances in images.
- A tracker, inherited from the SAM 2 transformer encoder-decoder design, enables segmentation and identity propagation across video frames.
- A presence token helps the model determine whether the prompted concept is actually present, improving robustness for fine-grained concepts.
- The system is trained with a large-scale data engine that automatically annotated over 4 million unique concepts, producing the largest high-quality open-vocabulary segmentation dataset released to date.
¶ Intended uses & limitations
SAM 3 is intended for:
- Promptable object segmentation in images using text, boxes, points, masks, or exemplars.
- Video object segmentation and tracking, including interactive correction and refinement.
- Open-vocabulary visual understanding tasks where fixed class taxonomies are too restrictive.
- Dataset annotation and human-in-the-loop labeling for large-scale vision pipelines.
Limitations:
- SAM 3 is a large foundation model with 848M parameters, so it requires substantial compute resources for practical use.
- Its reference setup expects Python 3.12+, PyTorch 2.7+, and CUDA 12.6+.
- Access to checkpoints requires approval through the Hugging Face repository.
- As with other prompt-based systems, output quality depends on the clarity and specificity of the provided prompts, and follow-up interaction may still be needed.
¶ BibTeX entry and citation info
@misc{carion2025sam3segmentconcepts,
title={SAM 3: Segment Anything with Concepts},
author={Nicolas Carion and Laura Gustafson and Yuan-Ting Hu and Shoubhik Debnath and Ronghang Hu and Didac Suris and Chaitanya Ryali and Kalyan Vasudev Alwala and Haitham Khedr and Andrew Huang and Jie Lei and Tengyu Ma and Baishan Guo and Arpit Kalla and Markus Marks and Joseph Greer and Meng Wang and Peize Sun and Roman R{\"a}dle and Triantafyllos Afouras and Effrosyni Mavroudi and Katherine Xu and Tsung-Han Wu and Yu Zhou and Liliane Momeni and Rishi Hazra and Shuangrui Ding and Sagar Vaze and Francois Porcher and Feng Li and Siyuan Li and Aishwarya Kamath and Ho Kei Cheng and Piotr Doll{\'a}r and Nikhila Ravi and Kate Saenko and Pengchuan Zhang and Christoph Feichtenhofer},
year={2025},
eprint={2511.16719},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2511.16719},
}