Skip to main content
Back

Meta's SAM Shrinks 30x With No Accuracy Loss, Survey Finds

A comprehensive survey benchmarks dozens of lightweight SAM variants across CPUs, GPUs, and edge devices, revealing which tricks actually pay off.

  • A survey benchmarks 30+ efficient SAM variants under identical hardware and prompt conditions on arXiv.
  • SAM's 632M-parameter ViT-H encoder is the main bottleneck, accounting for over 98% of parameters.
  • EfficientViT-SAM-L0 delivers ~29x GPU speedup while matching SAM-H accuracy on COCO SegAny.
  • NanoSAM leads on Jetson Nano at 345 ms; EdgeSAM is the mobile sweet spot at 9.6M params.
  • All lightweight variants collapse on rare categories like Rail and Brain-Tumor in SGinW.
  • Full paper index and code links available on GitHub.

Meta's Segment Anything Model changed how people think about image segmentation, but its biggest version ships with a 632M parameter image encoder that makes real-time or on-device use painful. A new survey rounds up roughly three dozen efficient SAM variants, groups them by acceleration strategy, and runs them all through the same benchmarks to show which tricks actually pay off. The full paper and code index are available on arXiv and GitHub.

SAM consists of three components: an image encoder, a prompt decoder, and a mask decoder. In SAM-H, the ViT-H image encoder holds roughly 632M of the model's 641M parameters. Even in the smallest variant, SAM-B, the encoder still accounts for over 90% of the total. Almost every acceleration effort ends up attacking that encoder by swapping it, distilling it, quantizing it, pruning it, or rewriting it in faster code.

SAM architecture diagram showing image encoder, prompt encoder, mask decoder pipeline

Two tasks, two very different bottlenecks

The survey draws a useful distinction between the two workloads SAM handles. The promptable task, Segment Anything (SegAny), is bottlenecked by the encoder pass. The all-masks task, Segment Everything (SegEvery), has a different problem: it relies on a dense point-grid sampling strategy that predicts many candidate masks and then filters them, which is expensive regardless of how fast the backbone runs. SegEvery needs smarter prompt sampling, not just a leaner encoder.

The taxonomy of speedups

The authors organize the zoo into a handful of families, and it's worth knowing what each one is really doing:

  • Lightweight backbones with distillation. The most common recipe. MobileSAM's first attempt replaced SAM's heavy ViT encoder with the more efficient TinyViT, trained with a simple MSE loss between the two encoders' embeddings. Later work like TinySAM added full-stage distillation and hard-mask weighting to fix the accuracy drop.