LightSeek's TokenSpeed Beats TensorRT-LLM and Gets Adopted by vLLM
LightSeek Foundation's TokenSpeed delivers TensorRT-LLM-beating inference with vLLM-level usability, built in two months for agentic coding workloads
Takeaways
- New inference engine: TokenSpeed by LightSeek Foundation targets TensorRT-LLM-level performance with vLLM-level usability, MIT licensed.
- Beats TensorRT-LLM: ~9% faster min-latency and ~11% higher throughput at 100 TPS/user on Kimi K2.5 on NVIDIA B200.
- 580 TPS record: Achieved 580 tokens/sec on Qwen3.5-397B-A17B for agentic workloads on B200 GPUs.
- MLA kernel adopted by vLLM: TokenSpeed's Multi-head Latent Attention kernel, which nearly halves decode latency vs TensorRT-LLM, was integrated into vLLM as a day-0 launch partner.
- Hybrid model support: Native Mamba/GDN prefix caching achieves 90%+ KV cache hit rates on multi-turn agentic workloads.
- Built in two months: Cross-industry team from NVIDIA, AMD, Qwen, Together AI, and others; compute support from OpenAI and NVIDIA.
TokenSpeed is an open-source LLM inference engine from the LightSeek Foundation, built in roughly two months and released under the MIT license. It targets a specific gap in the inference landscape: existing tools optimize for either maximum performance (TensorRT-LLM) or maximum usability (vLLM). TokenSpeed pursues both, specifically for agentic inference workloads.
The timing is deliberate. Agentic coding systems like Claude Code, Codex, and Cursor now generate an extraordinary volume of tokens, and data centers requiring tens of gigawatts of power are being built to keep up. At that scale, even modest per-GPU efficiency gains compound into enormous capacity savings.
Why agentic inference is its own problem
Standard LLM benchmarks don't capture how coding agents actually behave. Contexts routinely exceed 50K tokens, conversations span dozens of turns, and the performance objective isn't raw throughput alone. The goal is maximizing tokens per minute per GPU while keeping per-user generation fast enough to feel responsive, typically 70 TPS per user and sometimes 200 TPS or higher.
TokenSpeed benchmarks against SWE-smith traces, which are real coding-agent traffic patterns, rather than the synthetic prompts most inference benchmarks use. That's a more honest comparison for the workload that actually matters here.
How the architecture works
TokenSpeed is built around four core subsystems, each solving a distinct problem:
- Modeling layer: A local-SPMD design with a static compiler that generates collective communication from module-boundary placement annotations. SPMD (Single Program, Multiple Data) means the same code runs on every GPU, with the compiler determining where data needs to move between them. Users don't hand-write parallelism logic.