Skip to main content
Back

EAGLE-4 Triples Qwen3 Inference Speed by Adapting to Serving Load

A single adaptive drafter adjusts speculative decoding to batch size and hardware, posting up to 2.99x throughput on vLLM and 3x single-request speedups.

  • EAGLE-4 ships as eagle-harness, a source-available drafter that plugs into SGLang, vLLM, and TokenSpeed.
  • One drafter adapts tree depth and branching to batch size, hardware and workload at runtime.
  • Measured 2.99x on vLLM batch-of-1 and 1.35x on SGLang 64-way, all preserving target distribution.
  • Qwen3.8-27B is the reference target; draft weights are published on Hugging Face.
  • Supports NVIDIA sm_80+ and AMD CDNA2 through CDNA4, with MI355X verified by serving tests.
  • Commercial use capped at 20M USD annual revenue; competing inference services require a paid license.

The EAGLE team has released EAGLE-4, an adaptive speculative-decoding drafter, alongside a pip-installable package called eagle-harness. The drafter measures serving costs and adjusts its speculative tree for the current batch size, hardware, and workload, reducing the need to maintain separate configurations for interactive and high-concurrency inference.

Speculative decoding uses a smaller model to propose several tokens before the target model verifies them in parallel. Correct verification preserves the target model’s output distribution while reducing the number of sequential target-model calls. Performance depends on how quickly the drafter proposes useful candidates and how much work the target spends verifying them.

pip install eagle-harness

The harness integrates with SGLang, vLLM, and the experimental TokenSpeed engine through their registration hooks, with no runtime patching. The reference pairing uses Qwen3.8-27B and a published EAGLE-4 checkpoint.

Runtime-sized speculation

EAGLE-4 profiles draft generation and target verification at the active batch size, then selects the depth and branching factor expected to produce the highest throughput. A top-k value of 1 creates a speculative chain, while larger values create a tree of candidate continuations.

The drafter combines several techniques intended to limit its own compute and memory costs:

  • Sliding-window draft layer: Keeps a bounded window of recent hidden states instead of attending across the full sequence.
  • Low-rank candidate head: Factors token prediction into smaller matrices, reducing the cost of proposing candidates.
  • Append-only draft cache: Extends the draft KV cache between rounds instead of rebuilding it.