EAGLE 3.1 Fixes a Silent Bug Making AI Inference 2x Faster in Production
EAGLE 3.1 fixes a newly discovered 'attention drift' flaw in speculative decoding, delivering up to 2x faster long-context throughput natively in vLLM
Takeaways
- EAGLE 3.1 released by the EAGLE, vLLM, and TorchSpec teams, fixing a newly identified flaw called attention drift in speculative decoding.
- Attention drift causes draft models to progressively ignore the original prompt and over-attend to their own outputs as speculation depth increases, degrading acceptance length.
- Two architectural fixes: FC normalization on target hidden states and post-norm hidden-state feedback between speculation steps stabilize the drafter.
- Up to 2x longer acceptance length in long-context workloads vs EAGLE 3; 2.03x per-user throughput gain at concurrency 1 on Kimi K2.6 benchmarks.
- Native vLLM support merged into main branch, shipping in v0.22.0; fully backward-compatible with existing EAGLE 3 checkpoints -- no code changes needed.
- Open-source Kimi K2.6 EAGLE 3.1 draft model available on HuggingFace; trained with the new TorchSpec framework.
Speculative decoding is one of the most impactful inference tricks in production LLM serving today. The idea is simple: a small, cheap draft model proposes several tokens at once, and the large target model verifies them all in a single forward pass. If the guesses are good, you get multiple tokens for the price of one. EAGLE achieves 2-3x speedups over standard autoregressive decoding and is widely deployed in production inference frameworks including vLLM, SGLang, and TensorRT-LLM. But there has been a quiet, persistent problem lurking in deeper speculation chains -- and the EAGLE team just found it, named it, and fixed it.
The EAGLE series -- including EAGLE 1, EAGLE 2, and EAGLE 3 -- has become one of the most widely adopted families of speculative decoding algorithms across both research and production systems. Now, the EAGLE team, vLLM team, and TorchSpec team have jointly introduced EAGLE 3.1 -- a major step forward in speculative decoding robustness, efficiency, and deployability.
The Bug That Nobody Had a Name For
While speculative decoding performs well in controlled settings, performance often degrades under different chat templates, long-context inputs, or out-of-distribution system prompts. The EAGLE team traced this fragility to a phenomenon they call attention drift -- as speculation depth increases, the drafter gradually shifts attention away from sink tokens and toward its own generated tokens.
Sink tokens are special tokens (like the beginning-of-sequence token) that transformers rely on as stable "anchors" for attention. When the drafter stops attending to them and starts over-attending to its own recent outputs, it loses its grounding in the original prompt -- and its token proposals become increasingly unreliable the further out it speculates.
The team traced this to the unnormalized residual path between chain steps: the drafter's hidden state magnitude grows monotonically with chain depth, exhibiting dynamics consistent with additional pre-norm transformer layers stacked on the target rather than as a standalone autoregressive predictor. In other words, the deeper the speculation chain, the more the drafter's internal activations blow up in magnitude -- and the worse it gets at its job.
This phenomenon was observed across both EAGLE3 drafters and MTP heads, suggesting drift is a property of drafter designs broadly -- not just an EAGLE quirk. The team published a paper formally characterizing the phenomenon.
Two Small Changes, Big Impact
The fix is elegant. EAGLE 3.1 introduces two targeted architectural changes to the draft model:
- FC normalization: An RMSNorm (a lightweight normalization operation) is applied to each target hidden state before it is fed into the FC projection layer, preventing any single layer's activations from dominating the drafter's input.
- Post-norm hidden-state feedback: The normalized hidden states -- rather than the raw ones -- are fed back into the next speculation step, keeping magnitude stable across the chain.

Intuitively, the post-norm design makes the drafter behave more like a true standalone autoregressive predictor at each step, rather than acting like extra layers bolted onto the target model. The result is a drafter that stays calibrated even at deeper speculation depths and across wildly different prompt formats.
What the Numbers Look Like
The gains are most pronounced in long-context workloads, which is exactly where EAGLE 3 was struggling most. In long-context workloads, EAGLE 3.1 achieves up to 2x longer acceptance length compared with EAGLE 3. Acceptance length is the key metric here -- it measures the average number of tokens the target model accepts per forward pass. Longer acceptance length means more tokens per GPU cycle, which translates directly to lower latency per user.
The team benchmarked the Kimi K2.6 EAGLE 3.1 draft model on Kimi-K2.6-NVFP4 with vLLM (4-way tensor parallel on GB200 hardware) against a no-speculation baseline on the SPEED-Bench coding dataset. EAGLE 3.1 delivers 2.03x higher per-user output throughput at concurrency 1, and the speedup stays meaningful as concurrency scales -- 1.71x at C=4, 1.66x at C=16.

Beyond raw speed, EAGLE 3.1 also brings:
- Stronger robustness to different chat templates and system prompts
- Better generalization from training-time to inference-time conditions
- More stable acceptance length across diverse serving environments
How It Was Built
TorchSpec lowers training overhead and simplifies experimentation workflows to accelerate iteration for next-generation speculative decoding research. Based on TorchSpec and vLLM, the team trained and open-sourced an EAGLE 3.1 draft model for Kimi K2.6, available on HuggingFace. The Kimi K2.6 EAGLE 3.1 draft model serves as a reference implementation showing the full pipeline from TorchSpec training to vLLM serving. NVIDIA provided GPU support for large-scale training, benchmarking, and inference validation.
Dropping It Into vLLM
EAGLE 3.1 lands in vLLM as a config-driven extension of the existing EAGLE 3 implementation, with full backward compatibility with existing EAGLE 3 checkpoints preserved. That means upgrading to a 3.1 draft model in production requires no code changes -- just a new model path in your speculative-config.
The support has already been merged into vLLM's main branch and will ship in the upcoming v0.22.0 release (also available via nightly builds now). Here is what serving Kimi K2.6 with EAGLE 3.1 looks like:
vllm serve nvidia/Kimi-K2.6-NVFP4 \
--trust-remote-code \
--tensor-parallel-size 4 \
--speculative-config '{"model":"lightseekorg/kimi-k2.6-eagle3.1-mla","method":"eagle3","num_speculative_tokens":3}' \
--language-model-onlyThe method flag stays "eagle3" -- the 3.1 architecture is detected automatically from the checkpoint. This is a zero-friction upgrade for anyone already running EAGLE 3 in vLLM today.
Why This Matters Beyond EAGLE
The attention drift phenomenon was observed across both EAGLE3 drafters and MTP heads (the multi-token prediction heads used in models like DeepSeek), and the team traced it to the unnormalized residual path between chain steps. This suggests the fix is not EAGLE-specific -- any speculative decoding system that chains draft steps through a residual path may be silently suffering from the same instability. The architectural insight from this work could inform how future draft models are designed across the entire field.
For teams running long-context inference workloads -- coding assistants, document analysis, agentic pipelines -- EAGLE 3.1 is a direct answer to the reliability gap that has made speculative decoding feel fragile in production. The combination of a named root cause, a clean architectural fix, open-source weights, and native vLLM support makes this unusually actionable.