Skip to main content
Back

Microsoft's CLSA Slashes Long-Context AI Decoding by 7.6x With One Trick

Microsoft Research's CLSA architecture computes sparse attention routing once and shares it across all decoder layers, hitting 17.1x throughput gains at 128K context with near-zero quality loss.

Paper, Full Access
  • CLSA computes sparse top-k attention routing once per forward pass and shares the result across all cross-decoder layers, eliminating redundant per-layer routing overhead.
  • Built on YOCO's decoder-decoder architecture, CLSA inherits its KV cache memory savings and prefill speedups while adding a new decoding speedup.
  • At 128K context on NVIDIA B200 GPUs, CLSA achieves 7.6x decoding speedup and 17.1x overall throughput vs. a standard Transformer baseline.
  • Quality is nearly lossless: CLSA matches or beats dense baselines on ARC-Challenge, GSM8K, DROP, and HumanEval at 4B scale.
  • A two-stage distillation training process teaches the shared indexer to identify tokens that are consensually important across all decoder layers.
  • No code or model weights have been publicly released; the paper is from Microsoft Research and Tsinghua University.

Long-context inference is quietly becoming one of the most expensive problems in production AI. As reasoning models like o1 and DeepSeek-R1 generate thousands of tokens of intermediate chain-of-thought before producing an answer, the cost of attending over a growing KV cache at every single layer compounds fast. A team from Microsoft Research and Tsinghua University has now published a paper proposing a surprisingly clean fix: compute the expensive routing step once, and reuse it everywhere.

The work, titled "You Only Index Once: Cross-Layer Sparse Attention with Shared Routing" (CLSA), tackles a bottleneck that has quietly undermined most prior sparse attention approaches. The result is a 4B-parameter architecture that achieves 7.6x decoding speedup and 17.1x overall throughput improvement at 128K context, while matching dense attention on nearly every quality benchmark.

The problem no one fully solved

Sparse attention -- the idea of having each token only attend to a small, relevant subset of the KV cache instead of every prior token -- has been an active research area for years. But in practice, it has faced a stubborn tradeoff:

  • Block-sparse methods (like NSA and MoBA) group tokens into chunks and attend to whole blocks at a time. They map well to GPU memory access patterns and deliver real wall-clock speedups, but the coarse granularity introduces quality loss -- nearby tokens in the same block can be semantically unrelated.
  • Token-sparse methods (like DeepSeek-V3's DSA) select individual high-relevance tokens with fine-grained top-k routing. They preserve quality better, but the top-k operation itself is the problem. Top-k routing over the full cache remains expensive because it must be recomputed independently at every layer during decoding -- and top-k is poorly suited to GPU tensor cores, which thrive on dense matrix multiplications.

The result: token sparse methods are usually more accurate yet deliver limited end-to-end speedup because top-k routing over the full cache remains expensive. At 128K context, the paper's latency analysis shows that an unamortized top-k pass can take as long as substantial dense attention work, despite involving far fewer arithmetic operations.