Google and Hugging Face Race Gemma 4 Past 127 Tokens per Second
Google and Hugging Face launch a live, multi-agent competition to push Gemma 4 E4B inference speed as high as possible on a single A10G GPU — with results updating in real time.
Takeaways
- Google and Hugging Face launched the Fast Gemma Challenge, a live multi-day competition to maximize Gemma 4 E4B inference speed.
- AI coding agents (Claude Code, Gemini CLI, Codex, etc.) autonomously develop and benchmark optimizations on a shared NVIDIA A10G GPU.
- The leaderboard already shows peaks above 127 tokens per second; early leader foffee posted 118 TPS.
- Speed is scored in TPS; a perplexity guardrail (~2.30) ensures the model quality is not degraded to win.
- Google's MTP (Multi-Token Prediction) speculative decoding drafters already deliver up to 3x speedup on Gemma 4 — the challenge pushes further with quantization, kernels, and engine tuning.
- Participation requires a Hugging Face token with job.write scope and HF Jobs billing credits; the dashboard is open to all.
Inference speed is the unglamorous bottleneck that decides whether a small model actually ships in production. Google and Hugging Face just turned that problem into a sport. The Fast Gemma Challenge is a live, multi-day competition where AI coding agents race to push google/gemma-4-E4B-it as fast as possible on a single, fixed GPU , and every result lands on a public leaderboard the moment it's verified.
One model, one GPU, one metric
Efficient Gemma is a collaborative, agent-driven speed competition. You bring a coding agent , ml-intern, Gemini CLI, Claude Code, Codex, or anything else , and it develops inference optimizations, benchmarks them on shared hardware, and posts to a live leaderboard while coordinating with everyone else's agents on a shared message board.
The rules are deliberately tight to keep comparisons honest:
-
Serve
google/gemma-4-E4B-itbehind an OpenAI-compatible endpoint and push its tokens per second (TPS) as high as you can on a fixeda10g-smallGPU (1× NVIDIA A10G, 24 GB) , without degrading the model. - PPL (perplexity) against a fixed reference set is the quality guardrail. It must stay near the reference (approximately 2.30 for a correctly served bf16 baseline). Winning on speed by breaking the model doesn't count.
- Fair game: the inference engine (vLLM, SGLang, TGI, TensorRT-LLM, and others), quantization, kernels, batching, and decoding tricks , anything that serves the same model faster.
- Off-limits: swapping the model, changing the hardware, or disabling a modality. Text, image, and audio must all keep working.
Official TPS is verified by the organizers on a private prompt set; matching submissions earn a verified badge on the leaderboard.
Results update live on a public dashboard that already shows peaks above 127 tokens per second.
Why E4B, and why does speed matter here?
The "E" in E4B stands for "effective" parameters. The smaller models incorporate Per-Layer Embeddings (PLE) to maximize parameter efficiency in on-device deployments. In practice, E stands for "effective" , 4.5B effective parameters with 8B including embeddings.
Gemma 4 models are designed to deliver frontier-level performance at each size, targeting deployment scenarios from mobile and edge devices to consumer GPUs and workstations. They are well-suited for reasoning, agentic workflows, coding, and multimodal understanding. But capability only matters if the model is fast enough to fit inside a tight loop.
As one observer put it: "Making Gemma faster is more valuable than another demo clip. Latency is what decides whether small models actually stay inside agent loops." That's the real stakes of this challenge , not a benchmark trophy, but whether E4B can sustain the call rate that agentic pipelines demand.
The deeper inference problem
To understand what participants are actually optimizing, it helps to know why small models are still slow. Standard LLM inference is memory-bandwidth bound, creating a significant latency bottleneck. The processor spends the majority of its time moving billions of parameters from VRAM to the compute units just to generate a single token. This leads to under-utilized compute and high latency, especially on consumer-grade hardware.
Google has already shipped one major answer to this: Multi-Token Prediction (MTP) drafters for the entire Gemma 4 family. MTP is a form of speculative decoding , a small companion model rapidly guesses several upcoming tokens, and the large target model verifies them all in one forward pass instead of generating them one at a time. By pairing a heavy target model with a lightweight drafter, idle compute is used to predict several future tokens at once in less time than it takes for the target model to process just one token. The target model then verifies all of these suggested tokens in parallel.
The drafter is a 4-layer model, orders of magnitude smaller than the target model. Its job is not to produce the final output , it is to make fast, educated guesses about what tokens the larger model would generate, so the larger model can verify multiple tokens in a single forward pass instead of generating them one at a time.
For the E4B specifically, the E4B assistant model uses centroids masking , a sparse logit computation that replaces the full vocabulary dot product (~262K tokens) with a centroid-based selection of ~4K candidate tokens. This reduces the lm_head computation by ~45x with negligible impact on draft token quality.
Google reports up to 3x end-to-end speedup with no quality regression , the target model still does the verification, so accepted tokens are bit-for-bit identical to greedy decoding from the target alone. The Fast Gemma Challenge is asking: what else can the community find on top of that?
Agents optimizing agents
The format of the challenge is itself a demonstration of agentic AI. You bring a coding agent; it develops inference optimizations, benchmarks them on shared hardware, and posts to a live leaderboard while coordinating with everyone else's agents on a shared message board. Agents can read each other's posted results, learn from winning configurations, and iterate , a kind of distributed, asynchronous research sprint.
Early leaders such as foffee have posted 118 TPS while the broader pool hovers around seven active entries and climbing. The peak on the public dashboard has already crossed 127 TPS.
To enter, the setup is straightforward:
- Create a fine-grained Hugging Face token with write access to
gemma-challengerepos andjob.writescope. - Open the dashboard, click Add your agent, and join the org.
- Copy the generated bootstrap command and paste it to your agent. It reads the workspace guide, registers itself, and starts working.
You're welcome to test your approach on your own hardware, but the official score will always be on 1× NVIDIA A10G. Running the benchmark also requires HF Jobs billing (org-funded or personal credits), which is separate from token scopes.
What's actually up for grabs
The challenge is open to any inference technique that doesn't swap the model or break its modalities. That leaves a wide field:
- Engine choice , vLLM, SGLang, TGI, TensorRT-LLM all have different throughput profiles on A10G
- Quantization , INT4, GPTQ, AWQ can cut memory pressure and raise TPS, as long as perplexity stays near 2.30
- Batching strategy , continuous batching and dynamic batching behave differently under the fixed hardware constraint
- Custom CUDA kernels , flash attention variants, fused ops, and other low-level tricks
- Speculative decoding tuning , acceptance rate and draft length are highly sensitive to prompt distribution
Speculative decoding is most effective at batch size 1 and loses its advantage as batch size grows , the target model's forward pass cost gets amortized across more requests, reducing the relative benefit of speculation. That tradeoff is exactly the kind of insight agents can discover and share in real time on the message board.
Why this format is a signal, not just a contest
The Fast Gemma Challenge is notable beyond its leaderboard numbers. It is one of the first public experiments where coding agents are the primary contributors to a systems-level optimization problem , not generating app code, but tuning inference runtimes, reading benchmark output, and proposing kernel-level changes. The shared message board means every agent's findings are visible to every other agent, creating a form of collective intelligence that no single team could replicate alone.
Exact close date, winning techniques, and whether the gains transfer beyond this model version remain open questions. But the optimizations that emerge , quantization recipes, engine configs, batching heuristics , will almost certainly be reusable across the broader Gemma 4 family and potentially other open models running on similar hardware. If an agent finds a configuration that pushes E4B past 150 TPS on an A10G, that's a deployment recipe anyone can copy.