LoopRPT Teaches AI to Think Smarter With Every Recurrent Pass
A new reinforcement pre-training recipe pushes rewards into the hidden iterations of looped transformers, improving accuracy and cutting compute per token.
Takeaways
- LoopRPT is a reinforcement pre-training framework built for looped language models like Ouro.
- It reframes next-token prediction as a per-iteration reasoning task with rewards assigned to latent steps.
- Uses an EMA teacher reference and noisy latent rollouts to supervise intermediate hidden states.
- Instantiated on Ouro-1.4B and 2.6B, it achieves Pareto dominance on accuracy versus compute trade-offs.
- Gains concentrate on hard MATH500 tokens, suggesting better early reasoning rather than premature exits.
- Full paper on arXiv; code not yet released, Ouro backbone is open.
Looped language models reuse one Transformer stack across several recurrent passes, refining hidden states before predicting a token. The LoopRPT paper proposes a reinforcement pre-training objective that scores those intermediate passes. On two Ouro checkpoints, the authors report higher accuracy at a given average loop count and earlier exits on tokens that require less computation.
Token rewards blur latent credit
LoopRPT builds on Ouro, a family of decoder-only language models that repeatedly applies the same stack of Transformer layers. Weight sharing keeps the parameter count fixed across passes, while each additional pass increases computation and latency.
| Property | Ouro-1.4B | Ouro-2.6B |
|---|---|---|
| Reused stack | 24 layers | 48 layers |
| Hidden size | 2,048 | 2,048 |
| Maximum recurrence | Four passes | Four passes |
| Pre-training data | Approximately 7.7 trillion tokens | Approximately 7.7 trillion tokens |
| Core components | Multi-head self-attention, RoPE, SwiGLU, and sandwich RMSNorm | |
Ouro predicts a halting probability during each recurrent pass. That probability determines whether computation stops or continues for a given token, allowing easy predictions to use less depth and difficult predictions to use more.
Conventional next-token training already propagates gradients through recurrent hidden states, but its supervision evaluates the resulting token distribution. Output-level reinforcement learning has the same credit-assignment problem: a final reward can influence the full trajectory without evaluating the contribution of each pass. LoopRPT adds a training signal tied to intermediate latent updates.