Skip to main content
Back

Alibaba's Qwen3.5 Runs 5x Faster on Device With a Clever Math Trick

A MatMul-only matrix inversion trick delivers 5x kernel speedup for Gated DeltaNet linear attention on NPUs, without sacrificing accuracy

  • Core result: A new MatMul-only algorithm replaces sequential forward substitution for matrix inversion in Gated DeltaNet, delivering up to 5x kernel speedup.
  • Method: Uses a truncated Neumann series expansion -- approximating the matrix inverse as a finite sum of matrix powers -- plus structural masking and a parallel residual correction step.
  • NPU focus: Designed specifically for NPUs, which excel at matrix multiplication but handle sequential control flow poorly, making the standard triangular solve a major bottleneck.
  • Quantization-aware: Handles INT4/INT8 inference by adapting approximation order per chunk size to prevent dynamic range overflow from repeated matrix powers.
  • No accuracy loss: Preserves model accuracy under both floating-point and low-precision inference on Qwen3.5-family models, with 20% reduction in decode-layer overhead.
  • Availability: No public code released yet; the paper is available at arxiv.org/abs/2606.06034.

Linear attention is having a moment. Qwen3.5, Alibaba's latest flagship model family, bets heavily on it: roughly 75% of the model's layers use Gated DeltaNet, a linear attention mechanism, while the remaining 25% use conventional full attention -- a 3:1 ratio of linear to quadratic layers. The appeal is obvious. Instead of maintaining an ever-growing attention map, each Gated DeltaNet layer keeps a fixed-size state matrix with dimensions proportional to the head dimension squared, independent of sequence length. New tokens update this state incrementally, and the output for each token is produced by querying the current state -- making the cost linear in sequence length.

But there's a hidden cost buried inside this elegant design. A new paper from researchers at Luoming Zhang et al. targets one of the nastiest remaining bottlenecks: matrix inversion inside chunk-wise parallel linear attention, and proposes a surprisingly clean fix that delivers up to a 5x kernel-level speedup.

The hidden bottleneck nobody talks about

To understand the problem, you need to know how Gated DeltaNet actually runs during training and inference. The sequence is split into chunks of size C. Within a chunk, the recurrence is computed exactly. The state is then passed between chunks. This chunkwise approach is what makes linear attention hardware-friendly -- it maps well to the batched matrix multiplications (GEMMs) that GPUs and NPUs are optimized for.

The problem is that computing the intra-chunk recurrence exactly requires inverting a strictly lower-triangular matrix -- a matrix where all the action is below the diagonal. The standard way to do this is forward substitution, a sequential algorithm where each row depends on the row before it. In AMG smoothers, polynomial Gauss-Seidel and ILU can be formulated via truncated Neumann sums, transforming triangular solves to batched matrix-vector products and providing dramatic speedups on parallel hardware -- but the standard approach in linear attention kernels hasn't made this leap. The result is a kernel that runs mostly sequentially on hardware that is built for parallelism.