Skip to main content
Back

HUST's MoDA Replaces a Decade-Old AI Building Block With 2.11% Accuracy Gains

HUST's hustvl group open-sources MoDA, replacing residual connections with attention across transformer layers for a 2.11% accuracy gain at only 3.7% extra compute.

  • What: HUST's hustvl group open-sources MoDA, a hardware-efficient Triton implementation of Mixture-of-Depths Attention.
  • Core idea: Replaces additive residual connections with attention across the depth dimension, letting any layer directly query KV pairs from all preceding layers.
  • Speed: Custom chunk/group-aware kernel achieves 97.3% of FlashAttention-2 efficiency at 64K sequence length, down from 44,924 ms naive baseline.
  • Results: +2.11% average accuracy on 10 downstream tasks and -0.2 average perplexity over OLMo2, with only 3.7% extra FLOPs.
  • Side effect: The attention-sink phenomenon (models over-attending to fixed tokens) disappears when cross-layer retrieval is available.
  • Access: MIT license, ships with LLM and vision training recipes; requires PyTorch >= 2.5 and Triton >= 3.0.

Since ResNet in 2015, every deep learning model has communicated between layers the same way: add the output of one layer to the next, x + F(x). It is so fundamental that most practitioners never question it. The hustvl research group at Huazhong University of Science and Technology (HUST) just open-sourced MoDA (Mixture-of-Depths Attention), a hardware-efficient implementation of a mechanism that replaces that additive residual with attention across the depth dimension of a transformer, letting any layer directly query the key-value pairs of every layer that came before it.

The problem with depth

Scaling depth is a key driver for large language models, yet as LLMs become deeper they often suffer from signal degradation: informative features formed in shallow layers are gradually diluted by repeated residual updates, making them harder to recover in deeper layers. Think of it as a game of telephone where every person in the chain also hears every prior message simultaneously. By person 152, the original message from person 3 is technically present but buried in a chorus of 151 accumulated additions.

Prior work tried to fix this by blending layer outputs more cleverly. DenseNet kept every layer's output. DenseFormer and LIMe learned static mixing weights. Hyper-Connections widened the residual to multiple channels. But all of these approaches share the same conceptual frame: accumulation. They decide how much to draw from each earlier layer based on the current layer's state alone, without consulting what those earlier layers actually contain. The HUST team argues this is a category error. Inter-layer communication should be retrieval, not accumulation.

Attention in two dimensions at once