Skip to main content
Back

Latent Sparse Attention Speeds Up 10M-Token AI Contexts by 9x

Coding agents drown in context. LSA groups attention keys like a library, cutting time to first token by 9.2x. (

November 8, 2026

Hey,

AI agents are getting better at working on tasks that span hours and involve entire codebases, document libraries, and long histories of interactions.

But the longer they work, the more expensive it becomes for the underlying models to process their growing context. Even sparse attention, designed to reduce this cost, has scaling problems of its own.

Researchers have developed Latent Sparse Attention (LSA), a technique that could make million-token contexts much more practical. Today, we examine the results, the tradeoffs, and what developers can already test.

Author photo
Today's Author
Ben Dickson is a veteran software engineer and former CTO known as the "Engineer's Journalist" for translating technical AI research into practical, production-focused insights. He is a lead contributor to TechCrunch and VentureBeat.
From Decodo
Most scrapers work until they hit a website with real anti-bot protection. Decodo’s Web Data API routes through 125M+ IPs across 195+ locations and renders JavaScript pages on demand, so your AI agents can scrape tough targets. Try it with up to 10K free monthly requests.
Sunday Deep Dive
How latent sparse attention tackles the search bottleneck in long-context AI

AI agents (particularly coding agents) accumulate millions of tokens of context as they work with large code repositories and document libraries. At that scale, handling the attention values of the underlying model becomes tedious and expensive.

To address this bottleneck, researchers at Prefrontal AI and ETH Zurich have developed Latent Sparse Attention (LSA), a new technique that cuts the compute costs of processing long contexts in LLMs.

LSA is among a growing range of techniques for optimizing the attention mechanism in LLMs. However, it addresses a bottleneck in other sparse attention methods, which limits how many previous tokens a model examines.

LSA organizes attention values into clusters and groups to make it easier to search through them when mapping new queries. For example, in experiments on 10-million-token tasks, it reduced time to first token by more than 9 times a full-attention model and 2.2 times against a model using a popular sparse attention mechanism.

For developers building long-running agents and LLM applications that process large contexts, LSA offers a way to reduce inference latency.

Why sparse attention still becomes expensive

In a transformer, attention determines which parts of the context matter when processing a token. Each token has a query (what is it looking for), a key (what information is relevant), and a value (the information to obtain the answer). In full attention, every new query examines all preceding keys.

The full attention mechanism becomes very expensive as the model’s context grows. Doubling the context roughly quadruples full-attention computation.

This is particularly evident in the “prefill” stage, where the model must process the attention values of the entire input, which can become quite long for tasks involving large documents and codebases.

When generating new tokens, the “decode” stage, the model must also consult the entire context for each new token, which also becomes slow and expensive as the context grows.

alpha_signal_image_1

Sparse attention solves the compute bottleneck by selecting a limited set of keys for each query. Methods such as DeepSeek Sparse Attention (DSA) use a smaller, cheaper indexer to score candidates. Other approaches group neighboring keys into blocks and score the blocks instead.

But if the indexer still examines every previous key or block, its prefill work continues to grow quadratically. For example, in a 10-million-token experiment with GLM-5.3-Flash, which uses a variant of DSA, selecting relevant tokens consumed 98% of an attention layer's prefill time.

How latent sparse attention works

LSA organizes attention values like a library. Rather than checking a description of every book, a query first examines groups of books, then retrieves books from the most promising groups. LSA creates these groups in a learned “latent space,” where compact numerical representations help identify similar keys.

The process has four main stages:

  • Represent: Create compact representations of keys for selection.
  • Organize: Organize keys into clusters based on their similarities, and store a representative vector, called a “centroid,” for each cluster.
  • Search: Compare each query with cluster centroids and select the groups most likely to contain relevant information.
  • Attend: Retrieve the original keys from those groups and perform conventional attention over them, alongside a recent-token window and a few initial tokens that remain accessible.
alpha_signal_image_2

The researchers propose two ways to search the clusters. “Flat search” compares a query with every cluster centroid. This is simple but the search tax grows with the number of clusters. “Tree search” optimizes search by organizing clusters into hierarchies and following promising branches.

There are also two ways to integrate LSA into models. Distilled LSA (D-LSA) adds a trained indexer to an existing model. The indexer learns to reproduce the model's attention preferences while the original model remains frozen.

alpha_signal_image_3

Native LSA (N-LSA) incorporates clustering during training, learning through the usual next-token prediction task without a separate teacher or index projections.

What the experiments show

The researchers evaluated D-LSA on Qwen-3.8-27B, which uses full attention, and GLM-5.3-Flash, which already uses sparse attention.

At 10 million tokens on an NVIDIA H200, D-LSA accelerated one GLM attention layer's prefill by 22.7 times and decoding by 4.6 times against the baseline sparse attention. For Qwen, prefill was 72.4 times faster than full attention.

In end-to-end serving across eight H200 GPUs, GLM's time to first token improved by 2.2 times, and its time per output token by 2.1 times. Qwen's time to first token improved by 9.2 times.

alpha_signal_image_4

On RULER, a benchmark for long-context retrieval and reasoning, Qwen improved by scoring 63.9 at 10 million tokens against 59.9 for full attention. GLM, however, lost up to eight points at contexts of two to five million tokens relative to its sparse-attention baseline.

They tested N-LSA by training a 500-million-parameter hybrid model at up to 128,000 tokens. It matched full attention's validation loss and trained 1.6 times faster per attention layer and 1.24 times faster end to end. It also retained more long-context accuracy than other baseline attention methods such as sliding-window attention (SWA).

What developers can do with LSA

Prefrontal AI has published the code under the Apache 2.0 license, along with distilled indexer weights for Qwen and GLM, native-training checkpoints, evaluation tools, and an SGLang integration.

The first candidates are applications that repeatedly process very large contexts: coding agents with extensive histories, research assistants searching document collections, and systems that must connect facts across many files.

Compare LSA against existing attention methods using representative input lengths, first-token latency, output-token latency, and retrieval accuracy. Note that the reported gains become much smaller in shorter context tasks.

AI models and agents are becoming increasingly better at long horizon tasks. Accordingly, the underlying attention mechanisms must catch up to make sure the AI systems don’t grind to a halt. LSA is one of the techniques that can enable this future where agents work for hours and days at a time.

Sponsor Logo
Give AI agents dependable access to websites that block ordinary scrapers, handling proxy rotation, fingerprinting, and anti-bot bypass for you.
Learn More →
At Alpha Signal, our mission is to build a sharp, engaged community focused on AI, machine learning, and cutting-edge language models, helping over 200,000 developers stay informed and ahead. We’re passionate about curating the best in AI, from top research and trending technical blogs to expert insights and tailored job opportunities. We keep you connected to the breakthroughs and discussions that matter, so you can stay in the loop without endless searching. We also work closely with partners who value the future of AI, including employers and advertisers who want to reach an audience as passionate about AI as we are.

Our partnerships are based on shared values of ethics, responsibility, and a commitment to building a better world through technology.Privacy is a priority at Alpha Signal. Our Privacy Policy clearly explains how we collect, store, and use your personal and non-personal information. By using our website, you accept these terms, which you can review on our website. This policy applies across all Alpha Signal pages, outlining your rights and how to contact us if you want to adjust the use of your information. We’re based in the United States. By using our site, you agree to be governed by U.S. laws.
Looking to promote your company, product, service, or event to 250,000+ AI developers?
Work with us