Google DeepMind's Gemma 4 Now Runs a 26B Model on a 16GB Laptop
Google's Gemma 4 QAT checkpoints slash VRAM by ~72%, putting a 26B model on a 16GB laptop and a 2B model under 1GB on mobile
Takeaways
- Google DeepMind released QAT checkpoints for all five Gemma 4 sizes (E2B, E4B, 12B, 26B-A4B, 31B), cutting VRAM ~72%.
- The Gemma 4 E2B text-only model now fits under 1GB on mobile using a custom mixed-precision schema with targeted 2-bit decoding layers.
- The 26B-A4B Mixture-of-Experts model now runs on a 16GB laptop or Apple Silicon Mac at 4-bit precision.
- QAT bakes quantization into training rather than applying it after, preserving quality within a few percentage points of the FP16 baseline vs. 10-15% loss with standard PTQ.
- Weights are available now on Hugging Face in GGUF format, compatible with Ollama, llama.cpp, LM Studio, vLLM, and SGLang.
- Do not quantize the base model yourself: naive Q4_0 conversion drops accuracy by ~15 points; use the official QAT checkpoints instead.
Google DeepMind just released Quantization-Aware Training (QAT) checkpoints for the entire Gemma 4 family, and the numbers are hard to ignore: the flagship 26B model now fits on a 16GB laptop, and the smallest E2B variant runs in under 1GB on a phone. This is not a minor tweak. It is a rethinking of how compression should be done for large language models targeting edge hardware.
The problem with shrinking models the old way
Every time you run a large model locally, you are fighting memory. The standard fix has been Post-Training Quantization (PTQ): take a fully trained model and round its weights down from 16-bit floats to 4-bit integers after the fact. PTQ takes a fully trained model and compresses its weights into a lower-precision format after training, but accuracy often degrades because the model never learned to compensate for the quantization noise. The quality loss is real and measurable. Traditional PTQ loses 10-15% quality compared to the full-precision baseline.
QAT (Quantization-Aware Training) takes a fundamentally different approach. Instead of simply quantizing the model after training, QAT integrates the quantization process directly into training. Concretely, during training, the model simulates the quantization step in its forward pass, so it learns weights that are robust to the rounding error introduced by lower precision. The model is not surprised by compression at deployment time because it was trained to expect it.
What's actually in the release
The Gemma 4 models optimized with QAT are available in five sizes: Gemma 4 E2B, Gemma 4 E4B, Gemma 4 12B, Gemma 4 26B A4B, and Gemma 4 31B. Two distinct formats are being shipped:
- Q4_0 GGUF checkpoints for all five sizes, ready for use with llama.cpp, Ollama, and LM Studio. These are the go-to format for consumer GPU and laptop deployments.
- A custom mobile quantization schema engineered specifically for the E2B and E4B edge models, targeting phones and other memory-constrained hardware.
The 4-bit format achieves roughly 72% lower memory usage with near-original performance. In practice, approximate total memory using recommended GGUF builds is: E2B ~3GB, E4B ~5GB, 12B ~7GB, 26B-A4B ~15GB, and 31B ~18GB.

The mobile schema is the real engineering story
The Q4_0 GGUF format is well-understood territory. The novel piece is the custom mobile quantization schema Google built for the E2B and E4B edge models. Rather than squeezing the whole model evenly, Google pushed the token-generating layers down to 2-bit while keeping the core reasoning layers at higher precision, and pre-computed activation scaling during training so mobile chips skip that work at runtime. This is a meaningful architectural decision: reasoning quality is preserved where it matters most, and storage is saved where the model can afford it.
The four pillars of the mobile schema are:
- Static activations: Scaling factors are pre-computed during training rather than calculated on-the-fly, reducing the workload on mobile chips at inference time.
- Channel-wise quantization: Data is structured to match the native design of mobile accelerators, allowing calculations to run without slow software workarounds.
- Targeted 2-bit decoding layers: The token-generation layers are heavily compressed to 2-bit, while core reasoning layers stay at higher precision.
- Embedding and KV cache optimization: Compression is focused on the vocabulary table and the model's short-term memory (the KV cache), slashing the active memory footprint during long conversations.
Using this mobile format, Google has reduced the memory footprint of Gemma 4 E2B to 1GB. And if you only need text (no vision or audio), stripping the multimodal encoders brings it even lower: the text-only E2B without Per-Layer Embeddings runs in under 1GB.
How it performs
The quality story is strong. Quality remains within a few percentage points of the FP16 reference on standard benchmarks like MMLU and GSM8K, and accuracy on INT4 QAT is significantly higher than INT4 PTQ, despite similar resource usage. In independent testing, the QAT version holds a small but consistent edge on HumanEval-style coding prompts, roughly 2-3 percentage points, with the bigger win being stability: long-context summarization at 128k tokens does not drift the way some PTQ models do.
Speed gains are also notable. The team reports up to 2x faster inference on mobile-class NPUs (Neural Processing Units) compared to FP16 versions. One important caveat: Gemma 4's architecture has a few traits that make PTQ painful specifically, including interleaved local/global attention and the 256k context window, which can amplify rounding errors. QAT was built to handle exactly this.
There is one trap to avoid. Naive Q4_0 conversion of the base model drops accuracy by 15 points , which is why you should use the official QAT checkpoints rather than quantizing the base weights yourself.
Getting it running
The weights are available now on Hugging Face in both GGUF and compressed-tensor formats. The ecosystem support at launch is broad:
- Desktop inference: Ollama, llama.cpp, and LM Studio all support the GGUF checkpoints out of the box.
- Server-side: vLLM and SGLang support the compressed-tensor format for efficient batch serving.
- Mobile/edge: Google's LiteRT-LM runtime handles on-device deployment; Transformers.js covers browser-based inference via ONNX.
- Apple Silicon: MLX community builds are available for M-series Macs.
- Fine-tuning: Unsloth and Hugging Face Transformers both support fine-tuning directly on the QAT weights.
For Ollama users, pulling a model is a single command:
ollama pull gemma4:12b-q4_0
ollama run gemma4:12b-q4_0 "Explain quantization-aware training in two sentences."Why this matters beyond the headline
Gemma 4 QAT may not be the loudest release of the year, but it may be one of the most consequential: it pushes the on-device AI boundary forward in a way that is accessible to independent developers, not just well-funded labs. The 26B-A4B model is a Mixture-of-Experts architecture that activates only 3.8B parameters per token, meaning the QAT 26B-A4B checkpoint loads in roughly 15GB at 4-bit, so a 16GB Apple Silicon Mac or a 16GB GPU can run it.
The broader implication is a shift in what "local AI" means. Running a capable multimodal model privately, on your own hardware, without a cloud bill, is no longer a research exercise. The practical use cases this unlocks are significant:
- Privacy-sensitive applications (medical notes, legal documents, personal data) that cannot leave the device
- Offline-capable mobile apps with real reasoning quality
- Low-latency agentic workflows where round-trip API calls are too slow
- Cost elimination for high-volume inference that would otherwise require cloud spend
The release of Gemma 4 QAT signals a major shift toward the democratization of high-performance AI on edge devices. The weights are free, the tooling is mature, and the hardware bar just dropped considerably. The main constraint now is not memory or money, but knowing which format to grab and which trap to avoid when you do.