Moonshot AI's AttnRes Challenges a Core Transformer Assumption Unchanged Since 2015
Moonshot AI's Kimi team replaces a decade-old Transformer building block with depth-wise attention, delivering a 1.25x compute advantage and +7.5 points on GPQA-Diamond
Takeaways
- What: Moonshot AI's Kimi team releases Attention Residuals (AttnRes), replacing fixed residual connections in Transformers with learned depth-wise softmax attention.
- Compute gain: Block AttnRes matches a baseline trained with 1.25x more compute, cutting training cost by ~20% for equivalent quality.
- Benchmark wins: On a 48B MoE model trained on 1.4T tokens: GPQA-Diamond +7.5, Math +3.6, HumanEval +3.1 points over baseline.
- Overhead: Less than 4% training compute overhead and under 2% inference latency increase; parameter count increase is negligible.
- Architecture shift: AttnRes makes deeper, narrower models optimal -- at 48 layers, optimal width drops from 4096 to 3072 vs. standard residuals.
- Available now: Paper on arXiv and open-source code released; community PyTorch and Rust implementations already available.
Every Transformer ever trained shares one assumption baked in since 2015: the best way to pass information between layers is a simple addition, where every layer contributes with the exact same fixed weight of 1. Residual connections have been essentially unchanged since ResNet in 2015, and they were carried into Transformers as-is. Moonshot AI's Kimi team just published a paper and open-sourced the code challenging that assumption directly.
The flaw hiding in plain sight
In PreNorm architectures, each layer adds its output back into a running hidden state, which keeps optimization stable and allows deep models to train. But Moonshot AI researchers argue that this standard mechanism also introduces a structural problem: all prior layer outputs are accumulated with fixed unit weights, which causes hidden-state magnitude to grow with depth and progressively weakens the contribution of any single layer.
This creates three concrete bottlenecks: no selective access (every layer receives the same blended state, regardless of whether it needs more low-level or high-level features), irreversible loss (once information is merged, later layers cannot retrieve a specific earlier representation), and output growth (deeper layers inflate their magnitude to stay influential, which can destabilise optimisation).
Attention over depth, not just over tokens
Attention Residuals (AttnRes) is a drop-in replacement for standard residual connections in Transformers that enables each layer to selectively aggregate earlier representations via learned, input-dependent attention over depth. The key insight is that instead of adding everything with weight 1, each layer runs a softmax over all preceding layer outputs and picks what it actually needs.
The mechanism is lightweight by design. The only modification to the base architecture is the addition of AttnRes to the residual connections; all other components remain unchanged. AttnRes introduces only one RMSNorm and one pseudo-query vector per layer, amounting to a negligible fraction of the total parameter count. All pseudo-query vectors are initialized to zero, so at the start of training, attention weights are uniform, equivalent to standard residual connections, avoiding early training instability.