Ohio State's ARLoop Beats Huginn-3.5B With a 1.2B Open Model
ARLoop rethinks residual connections in looped transformers, letting each layer pick which past states to reuse via depth attention.
Takeaways
- OSU NLP released ARLoop, a looped transformer using attention over depth instead of standard residuals.
- ARLoop-1.2B trained on 0.5T tokens beats Huginn-3.5B on all evaluated tasks and matches Qwen3.5-2B Base.
- Sliding-window depth attention matches full-history attention because depth attention concentrates on recent residual blocks.
- Gains are strongest on math and coding, consistent across 140M, 370M, 0.6B and 1.2B scales.
- Full artifacts on Hugging Face: weights, PreNorm baseline, intermediate checkpoints, training code, data recipe.
- Models load via standard Transformers with
trust_remote_code=True; vLLM and SGLang serving supported.
Looped transformers reduce weight memory by running one stack of transformer blocks multiple times. An 18-layer stack with recurrence depth R=3 performs 54 effective layer applications while storing one set of 18 layer weights. The model still pays for all 54 applications in sequential compute, but it avoids the parameter cost of 54 independent layers.
Parameter sharing complicates the residual stream, the running state that carries information between layers. Repeated passes combine outputs from different recurrences in one tensor, leaving each shared block without explicit access to the separate states produced during earlier passes.
Researchers at Ohio State University address that problem with ARLoop. Its attention residuals let each sublayer select earlier representations from across model depth. The team has released model weights, intermediate checkpoints, training code, a data recipe, and a matched baseline.
Shared layers muddy the residual stream
A standard transformer adds each block’s output to its current hidden state. Independent layers can specialize around that accumulated representation because each layer has its own parameters and fixed position in the network. A looped model sends the combined state through the same parameterized blocks several times, without exposing each earlier residual source separately.
ARLoop maintains previous block outputs and the token embedding as a bank of residual states. Each sublayer forms a query, scores those states as keys, and combines their values into its next input. The mechanism adapts attention residuals introduced by the Kimi team.
Depth attention and token self-attention operate on different axes. Token self-attention chooses information from positions in the sequence; ARLoop’s residual attention chooses information from earlier points in model depth. That gives each shared sublayer an explicit route to the representations it needs.