ICML Paper Cuts Transformer KV Cache Memory in Half With 3.1% Quality Loss
Researchers show merging K and V projections cuts KV cache by 50% with only 3.1% perplexity loss — and stacks with GQA/MQA for up to 96.9% reduction
Takeaways
- Core finding: Merging K and V into one shared projection (Q-K=V) cuts KV cache by 50% with only 3.1% perplexity degradation at 300M scale.
- Stacks with GQA/MQA: Combining Q-K=V with MQA yields 96.9% KV cache reduction — a 67x reduction at 1.2B scale — with under 5% quality loss.
- Why it works: K and V weight matrices have 0.73 cosine similarity in trained models; Q is genuinely distinct (0.42), so merging K+V is safe but merging Q+K breaks causal directionality.
- Scales well: Perplexity degradation shrinks from 3.1% at 300M to 2.48% at 1.2B, and downstream task accuracy drops only 0.41% on average.
- Theoretical bonus: Full Q=K=V collapse mathematically reduces linear attention to a state-space model — a formal bridge between transformers and SSMs.
- Open source: Code available on GitHub; paper accepted at ICML 2026.
Every transformer you have ever trained carries a quiet assumption baked into its architecture: that queries, keys, and values each need their own dedicated weight matrix. A new paper accepted at ICML 2026, authored by Ali Kayyam, Anusha Madan Gopal, and M. Anthony Lewis, asks whether that assumption is actually necessary , and finds that, at least for keys and values, it largely is not.
The paper, "Do Transformers Need Three Projections?", is a systematic study of what happens when you force two or all three of the QKV projection matrices to share weights. The headline result: merging K and V into a single projection cuts KV cache memory in half with only a 3.1% perplexity increase at 300M scale , and the trick stacks multiplicatively on top of existing techniques like GQA and MQA.
The KV cache is the real bottleneck
To understand why this matters, you need to understand what the KV cache actually is. During autoregressive generation (i.e., every time a model produces a token), the model needs to attend over all previous tokens. Rather than recompute the key and value representations for every past token from scratch at each step, transformers store them in a cache. This cache grows linearly with sequence length and batch size, and at long contexts it dominates GPU memory consumption , often more than the model weights themselves.
The industry's current answer to this problem is head sharing: techniques like Grouped Query Attention (GQA, used in Llama 2 and Mistral) and Multi-Query Attention (MQA, used in PaLM and Falcon) reduce the number of KV heads, so fewer tensors need to be stored. While GQA and MQA reduce the KV cache size by sharing heads across a layer, this paper's approach shares the projection matrices themselves , and these strategies are orthogonal, meaning they can be combined multiplicatively.