Skip to main content
Back

Unsloth's Gemma 4 26B Runs on 15GB, Closing the Local AI Gap

Unsloth's quantization-aware training GGUFs squeeze Google's 26B Mixture-of-Experts model into 15GB while keeping near-bfloat16 accuracy intact.

Model, Full Access
  • Unsloth released QAT GGUFs for Google's Gemma 4 26B-A4B Mixture-of-Experts model
  • Quantization-Aware Training cuts memory roughly 72% while keeping near-bfloat16 accuracy
  • The 26B-A4B QAT build runs on about 15GB of total memory, laptop-friendly
  • Unsloth's dynamic Q4_K_XL format reports up to 29x better KLD than naive Q4_0
  • Activates only 4B of 26B parameters per token, with 256K context and image input
  • Drop-in compatible with llama.cpp, Ollama, LM Studio, and Unsloth Studio under Apache 2.0

Running a 26-billion-parameter Mixture-of-Experts model on a laptop used to mean accepting a noticeable quality hit from aggressive 4-bit quantization. Unsloth's new Gemma 4 26B-A4B QAT GGUF changes that math. By repackaging Google's Quantization-Aware Training checkpoints into a custom dynamic format, the team is shipping a build that fits in roughly 15GB of memory while staying close to the original bfloat16 quality.

This model card is for the new versions of the Gemma 4 family optimized with Quantization-Aware Training (QAT), which allows preserving similar quality to bfloat16 while dramatically reducing the memory requirements to load the model. In Unsloth's hands, that translates to a single GGUF file that drops into llama.cpp, Ollama, LM Studio, or their own Unsloth Studio with no extra setup.

What the A4B actually means

The 26B-A4B is the Mixture-of-Experts middle child of Google's Gemma 4 lineup. The "A" in 26B A4B stands for "active parameters" in contrast to the total number of parameters the model contains. By only activating a 4B subset of parameters during inference, the Mixture-of-Experts model runs much faster than its 26B total might suggest. This makes it an excellent choice for fast inference compared to the dense 31B model since it runs almost as fast as a 4B-parameter model.

Architecturally, it shares the rest of the family's tricks: a hybrid attention mechanism that interleaves local sliding window attention with full global attention, ensuring the final layer is always global. This hybrid design delivers the processing speed and low memory footprint of a lightweight model without sacrificing the deep awareness required for complex, long-context tasks. To optimize memory for long contexts, global layers feature unified Keys and Values, and apply Proportional RoPE (p-RoPE). The model handles a 256K token context window, native function calling, configurable thinking/reasoning mode, and structured output support, plus multimodal text and image input.