Skip to main content
Back

ByteDance's Sa2VA Turns Any Image Mask Into Just Two Tokens

ByteDance's Pixel LLM codebase unifies SAM-2 and LLaVA into a single model that can segment, track, and converse about objects in images and videos — with a CVPR 2026 highlight paper on top.

  • Sa2VA is ByteDance's open-source Pixel LLM codebase that unifies SAM-2 video segmentation with LLaVA-style vision-language models into a single architecture.
  • SAMTok, a key sub-project, was accepted as a CVPR 2026 Highlight — it compresses any segmentation mask into just two discrete tokens, making pixel-wise tasks trainable like regular language tasks.
  • Sa2VA-8B scores 81.9 cIoU on RefCOCO and outperforms the previous video segmentation SOTA (VISA-13B) by up to 6.7 J&F on ReVOS.
  • A known training/inference mismatch limits video performance; a community fix (Sa2VA-i) recovers up to +11.6 J&F on MeViS using the same weights.
  • The repo now hosts five projects (Sa2VA, SAMTok, VRT, SaSaSa2VA, Pixel-SAIL) under Apache 2.0, with models on HuggingFace from 1B to 26B parameters.
  • Models support Qwen3-VL and InternVL3 backbones, run in bfloat16 or 8-bit quantized, and are usable via a ComfyUI node for image workflows.

Sa2VA is ByteDance's open-source framework for what they call "Pixel LLMs" , multimodal language models that don't just describe what they see, but can draw precise pixel-level masks around it. The codebase has quietly grown into a full research platform, now hosting four distinct projects that push the boundary of what vision-language models can do at the pixel level.

The core idea: one model, two superpowers

Most vision-language models (VLMs) are great at talking about images but blind to spatial detail. Segmentation models like SAM-2 are great at drawing masks but can't understand language. Sa2VA is the first unified model that achieves dense grounded understanding across both images and videos, marrying SAM-2's foundation video segmentation capabilities with LLaVA's vision-language understanding by unifying text, image, and video into a shared LLM token space.

The mechanism is elegant: the MLLM accepts inputs of images, videos, and text instructions, and outputs text responses. When the user instruction requires segmentation results, the text response includes a special [SEG] token, whose hidden states serve as implicit prompts and are converted through SAM-2 into image and video-level object segmentation masks.

The masks obtained from key frames are propagated as prompts through SAM-2 across the entire video, enabling segmentation of all frames corresponding to the textual description. This two-stage process , first segmenting key frames, then propagating the masks , ensures the model captures both object appearance and temporal dynamics.

Sa2VA benchmark comparison radar chart showing performance across MeVis, RefCOCO, and video understanding tasks

What's inside the codebase

The repo is structured as a growing family of projects, all sharing the same environment and base architecture:

  • Sa2VA , the core model. SAM-2 + MLLM for referring segmentation, grounded conversation, visual prompting, and image/video chat. Supports InternVL2.5/3 and Qwen2.5-VL/Qwen3-VL backbones.