Looped Prediction Transformer Cuts AI Memory by 79% With One Shared Cache
A new preprint shows that forcing looped Transformers to share one KV cache across recursions cuts memory 76-79% and improves perplexity.
Takeaways
- New preprint introduces Looped Prediction Transformer (LPT) with a single shared KV cache across recursions.
- Only the first recursion writes the cache; later ones read it and keep a short local window.
- Cuts context memory by 76-79% versus standard Transformers of the same size.
- Lowers FineWeb-Edu validation perplexity by 1.12 to 1.82 at five recursions.
- Shared cache doubles as a gradient highway, stabilizing training without adaptive loss tricks.
- Tested at 150M to 1B parameters; no code or weights released yet.
A new preprint by Giovanni Monea and collaborators describes language models that reuse the key-value cache written during their first pass. Later passes read that shared cache and retain only a short window of pass-specific state. Across models with 150 million to 1 billion parameters, the design reduced context memory and improved validation perplexity over the reported baselines.
One stack, many caches
Looped Transformers run the same stack of layers multiple times, with each traversal called a recursion. Reusing the layers keeps the parameter count fixed while adding computation per token, which can support iterative refinement and reasoning workloads.
A standard looped architecture stores a separate key-value cache for each recursion. These caches hold attention keys and values for previous tokens, allowing autoregressive generation to reuse earlier computations. Their memory cost grows with context length, batch size, model dimensions, layer count, and recursion count. At long contexts, the caches can dominate inference memory.
The first pass writes for every pass
The Looped Prediction Transformer assigns distinct memory roles to its recursions. The first pass writes a full-context KV cache. Every later pass reads that shared state and maintains a bounded sliding window containing its own recent keys and values. The local window preserves recent pass-specific information, while the shared cache supplies the longer history.