Google DeepMind's Gemma 4 QAT Slashes Memory 72% for Local Deployment
Google releases QAT checkpoints for all Gemma 4 sizes, cutting VRAM by ~72% while preserving near-BF16 quality for on-device and local deployment.
Takeaways
- Gemma 4 QAT checkpoints are live on Hugging Face for all five model sizes (E2B, E4B, 12B, 26B-A4B, 31B), free under Apache 2.0.
- ~72% VRAM reduction with near-BF16 quality: the 26B-A4B now fits on a 16GB laptop; E2B fits in 1GB on mobile.
- Custom mobile quantization schema (wNa8o8) uses static activations, 2-bit token-generation layers, and KV cache compression for native edge hardware performance.
- Naive Q4_0 conversion loses ~15 accuracy points; use Unsloth's UD-Q4_K_XL GGUFs for llama.cpp/Ollama to recover accuracy.
- Four formats ship: unquantized QAT, GGUF (Q4_0), compressed tensors for vLLM, and mobile-optimized wNa8o8 for E2B/E4B.
- Full ecosystem support from day one: Ollama, llama.cpp, vLLM, SGLang, MLX, LiteRT-LM, Transformers.js, and Unsloth fine-tuning all supported.
Running a capable open-weight model locally has always been a hardware negotiation. You either pay for cloud inference, settle for a weaker model, or buy more GPU memory. Google DeepMind just shifted that equation significantly: Gemma 4 QAT checkpoints are now live on Hugging Face for all five model sizes, cutting memory requirements by roughly 72% while keeping quality close to the full-precision originals.
The problem with squishing models after the fact
Standard quantization -- called Post-Training Quantization (PTQ) -- works by taking a fully trained model and rounding its weights down to lower-precision values. It is fast and cheap, but every rounding step introduces error, and those errors compound across dozens of transformer layers. The result is a smaller model that is also noticeably less capable.
Quantization-Aware Training (QAT) takes a different approach. By simulating quantization during training, QAT minimizes quality loss when the model is compressed. The model learns to work within the constraints of low precision from the start, rather than having compression imposed on it afterward. QAT consistently beats standard PTQ at the same compression level, landing within a few points of the BF16 originals.
What actually ships
The Gemma 4 models optimized with QAT are available in five sizes: Gemma 4 E2B, Gemma 4 E4B, Gemma 4 12B, Gemma 4 26B A4B, and Gemma 4 31B. Four checkpoint formats are provided to match different deployment targets:
- Unquantized QAT (Q4_0): Half-precision weights extracted from the QAT pipeline, ideal for custom downstream compilation and research.
- GGUF (Q4_0): Ready-to-deploy formats for broad ecosystem compatibility. Works directly with llama.cpp, Ollama, and LM Studio.
- Compressed tensors (w4a16-ct): Optimized for vLLM and SGLang for production multi-user serving.
- Mobile-optimized (wNa8o8): A custom schema engineered explicitly for mobile hardware efficiency, featuring targeted 2-bit decoding layers, optimized KV caches, and static activations to maximize VRAM savings. Available for E2B and E4B only.
The headline memory numbers are striking. This makes it possible to run larger models, such as Gemma 4 26B-A4B, locally on consumer GPUs with as little as 16GB of RAM -- with ~72% lower memory usage and near original performance. At the other end of the scale, the E2B variant reaches roughly 1 GB in the new mobile format.
For reference, the approximate RAM/VRAM requirements across model sizes with QAT Q4_0:
| Model | QAT Q4_0 Memory | Context Window |
|---|---|---|
| Gemma 4 E2B | ~3 GB | 128K |
| Gemma 4 E4B | ~5 GB | 128K |
| Gemma 4 12B | ~7 GB | 256K |
| Gemma 4 26B-A4B | ~15 GB | 256K |
| Gemma 4 31B | ~18 GB | 256K |

The mobile schema is the real engineering story
The Q4_0 checkpoints are a meaningful improvement, but the mobile-specific quantization schema is where Google did something genuinely novel. Standard 4-bit formats are designed for server GPUs and do not map cleanly onto the instruction sets of mobile accelerators. Google rethought the compression layout from scratch for edge hardware:
- Static activations: Normally, models compute scaling factors on the fly during inference. Pre-calculating these during training reduces workload on mobile chips and speeds up responses.
- Channel-wise quantization: Data is structured to fit the design of mobile accelerators, allowing native calculations without slow software workarounds.
- Targeted 2-bit quantization: The token-generation layers are compressed to 2-bit, while the core reasoning layers stay at higher precision -- saving storage without hurting reasoning quality.
- Embedding and KV cache optimization: Compression is focused on the vocabulary table and the model's short-term memory (the KV cache), dramatically reducing active memory during long conversations.
Because audio and vision encoders are not needed in many use cases, you can optimize your memory footprint even further by deploying only the modalities you need. The text-only E2B without Per-Layer Embeddings drops below 1 GB -- a number that was unthinkable for a model of this capability class even a year ago.
A deployment pitfall you need to know about
There is a catch that has already tripped up early adopters. If you naively convert the QAT BF16 checkpoint to llama.cpp's Q4_0 format, you lose a chunk of the accuracy QAT was supposed to preserve. The reason: llama.cpp's Q4_0 uses F16 scales, while the QAT weights were trained against BF16 scales, and llama.cpp does not pick the scales optimally.
Converting to Q4_0 naively gets only 70.2% top-1 accuracy for 26B-A4B. Unsloth's dynamic method pushes it up to 85.6% (+15.6%) while also being 200MB smaller. The practical fix: for llama.cpp and Ollama, use Unsloth's UD-Q4_K_XL GGUFs rather than rolling your own Q4_0 conversion.
Getting started
The weights are free and available now under the Apache 2.0 license, which permits commercial use. Here is the fastest path to each deployment scenario:
- Desktop / laptop (Ollama):
ollama run gemma4-- Ollama pulls the QAT-optimized GGUF automatically. - llama.cpp: Download Unsloth's UD-Q4_K_XL GGUF from Hugging Face to avoid the naive conversion accuracy drop.
- Production serving (vLLM): Use the
w4a16-ctcompressed-tensors checkpoints directly. - Mobile / edge: Use Google's LiteRT-LM runtime with the wNa8o8 mobile checkpoints for E2B or E4B.
- Apple Silicon: MLX-optimized versions are available in the Hugging Face collection.
- Fine-tuning: Use the MTP QAT checkpoints to preserve the speedup of Multi-Token Prediction while quantizing the models, and fine-tune weights directly using Hugging Face Transformers and Unsloth.
For production serving with vLLM, the command looks like:
vllm serve google/gemma-4-31B-it-qat-w4a16-ct \
--max-model-len 32768 \
--port 8000
Why this matters beyond the headline
A 26B-A4B model scoring in the 80s on MMLU Pro and LiveCodeBench now runs in roughly 15GB. Before QAT, that quality meant a much larger memory footprint or a cloud API bill. The practical implications are real: these are multimodal, hybrid-thinking models that support 140+ languages and up to 256K context -- and they now run on hardware most developers already own.
The broader shift here is that QAT is moving from a research technique to a standard part of the model release pipeline. Google shipping QAT checkpoints alongside the base weights -- not as an afterthought but as a first-class artifact -- signals that on-device and local deployment are no longer second-class citizens in the open-model ecosystem. Gemma 4 is released under the Apache 2.0 license, which allows free commercial use, fine-tuning, redistribution, and modification without MAU caps or use-case restrictions. That combination of quality, accessibility, and permissive licensing is what makes this release worth paying attention to.