Skip to main content
Back

Zyphra's Zamba2-1.2B Beats Gemma2-2B While Using 6x Less Memory

Zyphra's 1.2B hybrid SSM-Transformer model beats Gemma2-2B on MT-Bench while using 6x less KV cache memory

  • Size vs. quality: Zamba2-1.2B-Instruct outperforms Gemma2-2B-Instruct (2x larger) on MT-Bench with a score of 59.53 vs 51.69.
  • Hybrid architecture: Combines Mamba2 state-space layers with shared transformer attention blocks at a 6:1 ratio, using LoRA for per-layer specialization.
  • Memory efficiency: Achieves a 6x reduction in KV cache memory and 30-50% faster time-to-first-token vs comparable pure-transformer models.
  • Training: Pretrained on 3T tokens (Zyda-2 dataset), then fine-tuned with SFT + DPO on open-source instruction and preference datasets.
  • Free and open: Available on Hugging Face under Apache 2.0; requires installing mamba-ssm kernels for optimal performance.
  • Edge-first design: Ideal for on-device, mobile, and latency-sensitive deployments; 408K+ downloads signal strong real-world adoption.

Zamba2-1.2B-Instruct is a small language model from Zyphra that punches well above its weight class. At just 1.2 billion parameters, it outperforms Gemma2-2B-Instruct on MT-Bench despite being less than half the size. With over 408,000 downloads on Hugging Face, it has clearly found a real audience. The secret is a hybrid architecture that blends Mamba2 state-space layers with shared transformer attention blocks, delivering both strong quality and dramatically lower inference costs.

The problem with small transformers

Standard transformer models have a well-known scaling problem: their KV cache (the memory used to store past context during generation) grows linearly with sequence length. At small model sizes, this becomes especially painful because memory is already tight, and you want to run these models on constrained hardware like edge devices or mobile phones. Pure state-space models (SSMs) solve the memory problem but historically struggled with in-context retrieval tasks. Zamba2 is designed to get the best of both worlds.

A hybrid architecture with a clever twist

Zamba2-1.2B uses a backbone of Mamba2 layers interleaved with shared attention layers, where the attention has shared weights to minimize the parameter cost of the model. Mamba2 is a state-space model (SSM) that processes sequences recurrently in constant memory rather than storing the full context like attention does. The key insight is that you don't need full attention everywhere: a few shared attention layers are enough to preserve the in-context retrieval capability that pure SSMs give up.

Zamba2 architecture diagram showing Mamba2 blocks feeding into shared attention layers with LoRA modules

There are three specific innovations in Zamba2 over its predecessor:

  • Mamba2 backbone: Mamba2 blocks have roughly 4x the throughput of standard transformer blocks at equivalent size, so using them as the primary compute layer is a major efficiency win.
  • Shared attention with LoRA: