Skip to main content
Back

LOOM Pushes MoE Transformers to 12 Loops, Boosting Accuracy by 5.3 Points

LOOM diagnoses why looped Mixture-of-Experts models stall after two iterations and fixes it, scaling stable recurrence to 9-12 loops with real gains.

  • LOOM is a training recipe that scales looped Mixture-of-Experts models stably to 9-12 iterations instead of the usual two.
  • Diagnoses two failure modes: hidden-state variance blow-up and expert selection collapse across loops.
  • Fixes them with residual scaling, embedding re-injection, per-loop routers, and a Looping Residual carrying earlier outputs forward.
  • 700M model at 5 loops cuts perplexity 18.36 to 16.54 under matched FLOPs versus a non-looped baseline.
  • 1.7B model at 9 loops cuts perplexity 9.62 to 7.77 and lifts zero-shot accuracy from 42.4% to 47.7%.
  • Code and config available at github.com/hed-ucas/LOOM alongside the arXiv paper.

Di He and colleagues introduce LOOM, a training recipe that extends useful recurrence in Mixture-of-Experts (MoE) Transformers from roughly two passes to as many as 12, depending on model size and compute regime. In the paper, perplexity continues to fall and average zero-shot accuracy continues to rise as loops are added. Developers gain another scaling control: more sequential inference compute while the dominant model weights remain shared across passes.

The authors test models from 100M to 1.7B parameters and publish the source code. Looped depth has long been proposed as a scaling axis alongside model size and training data, but sparse models have struggled to benefit beyond two iterations.

Breaking the two-pass ceiling

A looped Transformer feeds its hidden state through the same block several times before producing an output. Each pass adds computational depth without duplicating the main attention and expert weights, although it consumes additional FLOPs and increases latency.

An MoE layer contains multiple feed-forward expert networks and a router that sends each token to a small subset of them. The paper identifies two reasons these models stop improving after a few recurrent passes:

  1. Hidden-state drift. Residual updates accumulate across layers and loops, increasing activation variance and destabilizing the representation.
  2. Expert selection collapse. A shared router repeatedly sends tokens to the same experts, so later passes perform similar computations instead of adding useful diversity.