Skip to main content
Back

NVIDIA's kvpress Cuts Long-Context GPU Memory 55% With One Line

NVIDIA's open-source kvpress library unifies 20+ training-free KV cache compression methods into a single pip install, slashing long-context LLM memory costs by up to 50%.

  • What it is: NVIDIA's open-source kvpress library unifies 20+ training-free KV cache compression methods into a single Hugging Face-compatible pip package.
  • The problem: KV cache memory grows linearly with context length; 1M tokens on Llama 3.1-70B requires 330GB of memory for the cache alone.
  • Key results: At 128k context on an A100, a 50% compression ratio cuts peak memory from 45GB to 37GB and boosts decoding speed from 11 to 17 tok/s.
  • How it works: "Presses" hook into attention layers at prefill time and prune the lowest-importance KV pairs using scoring strategies (attention weights, key norms, key similarity, etc.).
  • New: decoding compression: The experimental DecodingPress wrapper now compresses the cache periodically during token generation, not just at prefill.
  • Free & open: Apache 2.0, installable via pip install kvpress, with a live Hugging Face demo and public leaderboard.

Long-context LLMs are memory hogs, and the culprit is the KV cache. Every token processed during inference gets its attention keys and values stored in memory so the model doesn't have to recompute them. That's the good news. The bad news: as context windows grow, the KV cache size grows linearly with sequence length, quickly exhausting available GPU memory in long-context scenarios. Handling 1M tokens with Llama 3.1-70B in float16 alone requires up to 330GB of memory. NVIDIA's answer is kvpress, an open-source Python library that makes KV cache compression a one-liner.

The memory wall no one talks about

The KV cache is often treated as a solved problem, but it quietly dominates GPU memory at scale. The KV cache can consume up to 70% of total GPU memory during inference when processing long texts. Memory consumption grows multi-linearly with sequence length, layer depth, and head width, and KV cache requirements can quickly surpass that of model parameters, especially when handling batches or serving long-context models. The research community has produced dozens of compression strategies to fight this, but they're scattered across papers, each with its own implementation quirks.

That fragmentation is exactly what kvpress is designed to fix. kvpress implements multiple KV cache compression methods and benchmarks using Hugging Face Transformers, aiming to simplify the development of new methods for researchers and developers in this field.

One abstraction to rule them all

The core concept in kvpress is a "press" -- a compression algorithm that hooks into each attention layer during the prefilling phase (the initial pass where the model processes your input prompt) and prunes the least important KV pairs before they're stored. kvpress provides a set of presses that compress the KV cache during the prefilling phase, each associated with a