Ai2's OlmPool Reveals Four Design Choices That Tank Long Context by 47%
Ai2 released OlmPool, a controlled suite of 26 open 7B models that isolates how four small architecture choices wreck long context extension.
Takeaways
- Ai2 released OlmPool, 26 controlled 7B models isolating architecture effects on long context extension.
- Four attention design choices compound to drop long context scores by up to 47% when combined.
- QK norm, GQA, sliding window attention, and short pretraining context are the culprits identified.
- Short-context metrics and validation loss fail to predict post-extension performance at 32K or 64K.
- All 26 models ship with 38 checkpoints each, plus tech report and code.
- Models without QK norm develop stronger attention sinks, which correlate with better long-range retrieval.
Ai2 has released OlmPool, a controlled suite of 26 open 7-billion-parameter language models built to answer a single question that has been quietly haunting long context research: when you extend a model from short sequences to 64K tokens and it falls apart, is that the fault of the architecture, the training data, or the extension recipe? By training 26 models on the same data with the same extension procedure and varying only the architecture, the team isolated four small design choices that compound to tank long context performance by up to 47%.
Why a controlled architecture study was overdue
Most language models are trained on short sequences and then taught to handle much longer inputs through additional training on longer documents, a process called context extension. A large share of the published work on extending context has been developed and validated on Llama-family models, partly because of Llama's popularity, but also because Llama 3 happens to extend very easily.
That is a shaky foundation for the whole field. Llama 3's pretraining data is proprietary, so nobody outside Meta knows whether its cooperative behavior during extension comes from architectural choices or from the corpus. Teams building on Olmo, Qwen, Gemma, or in-house architectures have been forced to assume that recipes transfer, and often they do not.
Four culprits inside the attention stack
Every design decision the team tested affects attention, the mechanism that decides which tokens the model focuses on. As inputs get longer, the demands on attention increase, and small structural choices start to matter a lot. The four features under the microscope:
- QK normalization. A technique that normalizes query and key vectors inside each attention layer to improve training stability. It shows up in Olmo 3, Qwen 3, and Gemma 3, with a headwise variant used by Qwen 3 and Gemma 3.
- Grouped-query attention (GQA).