ByteDance's Sa2VA Turns Any Image Mask Into Just Two Tokens
ByteDance's Pixel LLM codebase unifies SAM-2 and LLaVA into a single model that can segment, track, and converse about objects in images and videos — with a CVPR 2026 highlight paper on top.
Takeaways
- Sa2VA is ByteDance's open-source Pixel LLM codebase that unifies SAM-2 video segmentation with LLaVA-style vision-language models into a single architecture.
- SAMTok, a key sub-project, was accepted as a CVPR 2026 Highlight — it compresses any segmentation mask into just two discrete tokens, making pixel-wise tasks trainable like regular language tasks.
- Sa2VA-8B scores 81.9 cIoU on RefCOCO and outperforms the previous video segmentation SOTA (VISA-13B) by up to 6.7 J&F on ReVOS.
- A known training/inference mismatch limits video performance; a community fix (Sa2VA-i) recovers up to +11.6 J&F on MeViS using the same weights.
- The repo now hosts five projects (Sa2VA, SAMTok, VRT, SaSaSa2VA, Pixel-SAIL) under Apache 2.0, with models on HuggingFace from 1B to 26B parameters.
- Models support Qwen3-VL and InternVL3 backbones, run in bfloat16 or 8-bit quantized, and are usable via a ComfyUI node for image workflows.
Sa2VA is ByteDance's open-source framework for what they call "Pixel LLMs" , multimodal language models that don't just describe what they see, but can draw precise pixel-level masks around it. The codebase has quietly grown into a full research platform, now hosting four distinct projects that push the boundary of what vision-language models can do at the pixel level.
The core idea: one model, two superpowers
Most vision-language models (VLMs) are great at talking about images but blind to spatial detail. Segmentation models like SAM-2 are great at drawing masks but can't understand language. Sa2VA is the first unified model that achieves dense grounded understanding across both images and videos, marrying SAM-2's foundation video segmentation capabilities with LLaVA's vision-language understanding by unifying text, image, and video into a shared LLM token space.
The mechanism is elegant: the MLLM accepts inputs of images, videos, and text instructions, and outputs text responses. When the user instruction requires segmentation results, the text response includes a special [SEG] token, whose hidden states serve as implicit prompts and are converted through SAM-2 into image and video-level object segmentation masks.
The masks obtained from key frames are propagated as prompts through SAM-2 across the entire video, enabling segmentation of all frames corresponding to the textual description. This two-stage process , first segmenting key frames, then propagating the masks , ensures the model captures both object appearance and temporal dynamics.

What's inside the codebase
The repo is structured as a growing family of projects, all sharing the same environment and base architecture:
- Sa2VA , the core model. SAM-2 + MLLM for referring segmentation, grounded conversation, visual prompting, and image/video chat. Supports InternVL2.5/3 and Qwen2.5-VL/Qwen3-VL backbones.