NetKV Cuts AI Response Latency 17.6% by Making Schedulers Network-Aware
NetKV adds a network-cost oracle to LLM inference schedulers, cutting mean TTFT by up to 21% on 64-GPU clusters with zero changes to hardware or inference engines
Takeaways
- NetKV is a network-aware scheduler for disaggregated LLM inference that routes KV cache transfers to decode instances based on topology and congestion, not just compute load.
- Current cache+load-aware schedulers ignore network distance; NetKV proves this becomes arbitrarily suboptimal as context length (and KV cache size) grows.
- On a 64-GPU fat-tree simulator with real Mooncake traces, NetKV cuts mean TTFT by up to 21.2% vs. round-robin and 17.6% vs. the best cache+load baseline.
- SLO attainment improves by up to 20.1 percentage points; Time Between Tokens overhead stays under 0.5 ms in all conditions.
- The approach requires no changes to inference engines, transport, or hardware -- only a lightweight network cost oracle feeding an O(|D|) greedy selector.
- No open-source code has been released; the paper is theoretical and simulation-based, using Mooncake production traces as the workload driver.
Every time a large language model finishes processing your prompt, it has to ship the result -- a dense blob of floating-point data called the KV cache -- across a datacenter network before it can start generating your response. In modern disaggregated inference setups, where prefill (prompt processing) and decode (token generation) run on separate GPU pools, that network hop is unavoidable. And yet, today's schedulers route requests as if the network doesn't exist. A new paper, NetKV, fixes that blind spot.
Why disaggregated inference is the new normal
Prefill-decode (PD) disaggregation separates the prefill phase from the decode phase, assigning them to dedicated prefill engines and decode engines respectively. The two phases exhibit distinct compute and memory patterns: prefill is compute-intensive and batched, while decode is memory-bound and latency-sensitive. Keeping them on the same GPU means one always starves the other.
Instead of forcing prefill and decode to share the same GPU, disaggregated systems split them into different sets of GPUs, which fundamentally eliminates interference between prefill and decode, and allows each to be independently optimized for throughput or latency. Systems like Mooncake -- the serving platform behind Kimi -- have taken this architecture to production scale. Mooncake is currently operational across thousands of nodes, processing over 100 billion tokens daily.
But disaggregation introduces a new bottleneck: after completing the prefill computation for a request, the prefill instance transfers both the request and its corresponding KV cache to a decode instance via high-speed interconnects such as NVLink or InfiniBand. That transfer time lands directly inside the Time to First Token (TTFT) budget -- the latency the user actually feels.
The gap current schedulers leave open
Today's state-of-the-art schedulers make two considerations when routing a request to a decode instance: compute load and prefix-cache locality (whether the decode GPU already holds a cached version of a matching prefix). Both are sensible. Neither accounts for where the decode instance sits in the network relative to the prefill instance that just finished.