Flash Linear Attention Beats FlashAttention by 5x, Powers Qwen3.5's Record Throughput
Flash Linear Attention is the open-source kernel library quietly powering production LLMs like Qwen3.5 and Kimi, with 30+ subquadratic architectures in one place
Takeaways
- What it is: Flash Linear Attention (FLA) is an open-source Triton-based library with 30+ hardware-efficient kernels for linear attention, SSMs, and hybrid LLM architectures.
- Production proof: FLA's Gated DeltaNet kernel powers Qwen3.5-397B, achieving 8.6x–19x higher decoding throughput than Qwen3-Max at long contexts.
- Recent additions: Gated DeltaNet 2, Mamba3, Parallax, Wall Attention, Raven, MoBA, YOCO, and AttnRes were all added in the past few months.
- Speed: FLA's chunkwise kernels achieve 2–4x speedup over FlashAttention-2/3 on long sequences; at 16K tokens, GDN forward+backward is ~5.5x faster than FlashAttention2.
- Tradeoff: Linear attention's fixed-size state matrix limits exact long-range retrieval; hybrid 3:1 linear-to-softmax layer mixing is the practical solution now used by Qwen3.5, Kimi, and others.
- Usage: Install with
pip install flash-linear-attention[cuda]; layers are HuggingFace-compatible drop-in replacements for standard attention.
Flash Linear Attention (FLA) is a Triton-based kernel library that has become the standard toolkit for building and researching beyond-softmax sequence models. With over 5,400 GitHub stars and support for 30+ architectures spanning RetNet to Mamba3, it functions as the infrastructure layer underneath several high-profile production LLMs released in the past year.
The core premise: linear attention is theoretically faster than softmax attention on long sequences, but that advantage only materializes with hardware-efficient kernels. FLA provides those kernels, and recent additions including Gated DeltaNet 2, Parallax, Wall Attention, and Mamba3 show the subquadratic architecture race is moving quickly.
The Quadratic Problem
Standard Transformer attention compares every token against every other token, producing O(N²) time and memory complexity as sequence length grows. Linear attention rewrites the underlying math to scale as O(N), but realizing that speedup in practice requires optimized custom kernels. Transformers benefit from the highly tuned FlashAttention stack; until recently, linear attention had no equivalent. FLA fills that gap.
How Chunkwise Parallelism Works
A linear attention layer can be viewed as an RNN with a fixed-size state matrix. Processing tokens one by one is slow and sequential; materializing the full attention matrix costs quadratic memory. FLA's solution is to split the sequence into chunks, process each chunk in parallel, and pass the recurrent state between them. During training this gives you parallelism; during inference you get the cheap recurrent form. The same kernel handles both modes, switching automatically based on context.