Skip to main content
Back

vLLM 0.34 Reclaims GPU Memory and Boosts Cache Capacity by 9.5%

vLLM v0.34.0 now sizes the KV cache from memory it measures after warmup, removes the old V1 runner, adds 10 models and speeds up Blackwell decode by as much as 1.95×.

Repo, Full Access
  • vLLM v0.34.0 ships with 1,061 commits from 412 contributors, plus 10 new models including Mistral Large 4.
  • The extensible KV cache is now on by default and sizes memory after warmup, giving most models 4–10% more cache.
  • The gpu_memory_utilization default is now 1.0, so an explicit 0.92 now gives you a smaller cache than before.
  • Model Runner V1 is removed, and setting VLLM_USE_V2_MODEL_RUNNER=0 now fails at startup.
  • Step5 NVFP4 decode is up to 1.95× faster on 4×GB200, and GLM-5.3-Flash throughput is 1.6× higher.
  • Breaking changes: default block size is 64, ROCm 7.2 is dropped, and huggingface_hub 2.0+ is required. Install via PyPI.

vLLM 0.34.0 replaces conservative KV cache estimates with post-warmup memory measurements, allowing dedicated inference servers to use GPU capacity previously held in reserve. The release contains 1,061 commits from 412 contributors, enables the extensible KV cache by default, removes Model Runner V1, and changes the default cache block size. Operators should review old memory flags because an explicit limit such as --gpu-memory-utilization 0.92 can erase part of the capacity gain.

The allocator waits for real numbers

The KV cache stores attention keys and values for tokens being processed. Its capacity determines how many tokens and concurrent requests the engine can hold without recomputing or preempting work. Earlier vLLM releases created this cache before CUDA graph capture, so they had to estimate the memory that graphs, workspaces, and samplers would later consume. The default gpu_memory_utilization of 0.92 left 8% of GPU memory outside the engine’s budget to absorb estimation errors.

Version 0.34.0 measures available memory after graph capture and warmup. CUDA graphs record GPU operations for efficient replay, which requires memory addresses to remain stable. The new allocator preserves those addresses through virtual memory:

  1. vLLM reserves the KV cache’s complete virtual address range without immediately backing it with physical memory. NVIDIA GPUs use the CUDA driver’s cuMem* APIs; AMD GPUs use hipMem*.
  2. Graph capture and model warmup run, with physical memory committed only for pages they touch.
  3. Each worker records its peak temporary allocation and measures the memory still available.
  4. The engine keeps a safety margin of max(1 GiB, 0.75 × transient peak), then maps the remaining physical memory into the reserved cache range.

Stable virtual addresses keep captured graphs valid while the post-warmup measurement gives each model a cache sized around its observed memory use. The implementation is documented in the KV cache PR.

Capacity gains depend on the workload

Benchmarks from the pull request covered eight models on NVIDIA GB200 systems. Every tested configuration gained KV cache capacity, although the increase varied with graph and workspace requirements. TP4 denotes tensor parallelism across four GPUs.

Model Configuration Reported gain
Qwen3-32B GB200 103.1 GiB to 113.0 GiB, or 9.5% more cache tokens
Qwen3-8B GB200 7.5% more cache tokens
Qwen3-235B-A22B-FP8 GB200, TP4 8.2% more cache tokens
DeepSeek-V4-Flash GB200, TP4 4.1% more cache tokens

DeepSeek-V4-Flash gained less because its graphs and workspaces already consumed most of the old reserve. The vLLM 0.34 release gives a larger example: MiniMax-M3 on H200 increased from 433,000 to 672,000 cache tokens.

Engine initialization excluding weight loading improved by 21% to 34% because vLLM no longer runs a disposable graph-capture pass for memory estimation. Qwen3-235B initialization fell from 76 seconds to 50 seconds, reducing the time required to add replicas during autoscaling.

Old flags can erase the gain

The larger cache can increase decode latency per token when the scheduler fills the added capacity with more sequences. Release benchmarks report 6% to 18% higher per-token decode latency on some H200 models at concurrency 128. GPU memory dashboards will also remain near full because the allocator deliberately assigns available memory to the cache.

An explicit --gpu-memory-utilization 0.92 produced caches 0.6% to 3.7% smaller than before in the project’s tests. The new allocator subtracts measured graph memory from a budget already capped at 92%, and vLLM now prints a startup warning when an explicit value withholds memory.

Operational goal Setting Effect
Use available memory on a dedicated GPU Omit the memory flag Lets the extensible cache size itself after warmup
Reserve memory for another process --gpu-memory-utilization 0.9 Keeps the extensible allocator while limiting its budget
Restore the previous allocation path --no-enable-extensible-kv-cache Disables the extensible cache
Limit decode contention --max-num-seqs N Caps concurrent sequences without shrinking the cache

Explicit memory limits remain appropriate when several engines share one GPU, another process allocates GPU memory after startup, or KV connectors and reinforcement-learning weight-transfer jobs maintain separate buffers. Unified-memory systems such as DGX Spark also need room for the host operating system and CPU workloads because they share the same physical memory pool.

# Dedicated GPU: remove old memory limits
vllm serve "$MODEL"

# Preserve the previous cache block size
vllm serve "$MODEL" --block-size 16

Version 0.34.0 raises the default --block-size from 16 tokens to 64. Prefix caching reuses only complete blocks, so a shared 40-token system prompt yields no reusable block at size 64. A 16-token block size can reuse 32 tokens from the same prompt. The pull request does not include a 16-versus-64 comparison on a production serving workload, so deployments with short shared prefixes should benchmark both settings.

V2 becomes the only GPU runner

Model Runner V2 now handles all GPU execution. Setting VLLM_USE_V2_MODEL_RUNNER=0 causes startup to fail, and custom logits processors written for V1 must migrate to the V2 sampler module. Dual-batch overlap, which lets two micro-batches share GPU execution, now works on V2 alongside speculative decoding.

Suffix decoding also runs directly on V2 without the separate Arctic Inference package. This speculative technique drafts tokens by matching the current sequence against earlier output, then verifies those candidates with the model.

Faster kernels favor newer accelerators

Reported kernel improvements concentrate on NVIDIA Blackwell GPUs and AMD MI355X accelerators. Results vary by model shape, batch size, context length, and parallelism configuration.

Workload Hardware or path Reported result
Step5 NVFP4 MoE 4× GB200, TRTLLM-Gen kernels Up to 1.95× faster decoding
GLM-5.3-Flash DeepGEMM FP8 MegaMoE 1.6× higher output throughput
GLM-5.3 long prompts GB200, 32K to 128K context 1.24× to 1.40× faster time to first token
Kimi-K3 GB300, opt-in KV-balanced MLA decoding 1.34× to 1.59× faster decode attention
Qwen3.5 Breakable CUDA graphs 3% to 8.8% lower decode latency

Model support expands to Mistral Large 4, MiniMax-M3.1-Flash, Nemotron-3.5-Transcribe, Liquid AI’s LFM3 family, Zyphra ZAYA1, and several speech models. Mistral Large 4 is a mixture-of-experts model with 1 trillion total parameters and 52 billion active parameters per token. Preliminary Kimi K3.1 code landed in the release, but K3.1 cannot yet be served.

Upgrade checklist and removals

Version 0.34.0 is available under the Apache 2.0 license from PyPI. The main Docker image uses CUDA 13.0, with separate ROCm 10.0, CPU, and XPU variants.

python -m pip install "vllm==0.34.0"
docker pull vllm/vllm-openai:v0.34.0

Several removals and dependency changes can block an in-place upgrade:

  • ROCm 7.2 images have been removed; deployments must move to ROCm 10.0 artifacts.
  • huggingface_hub >= 2.0.0, tokenizers >= 0.23.3, and timm >= 1.0.30 are required.
  • Marlin W4A8 support has been removed. Use --quantization humming with VLLM_HUMMING_INPUT_QUANT_CONFIG instead of VLLM_MARLIN_INPUT_DTYPE.
  • Support for QuTLASS fp_quant checkpoints and Keye-VL models has been removed.

Community Windows builds report that the extensible KV cache works on Windows. A GPU that also drives the desktop may expose less free memory during warmup, producing a smaller cache than earlier releases. Dedicated inference GPUs avoid that source of variability.

On a dedicated GPU, remove legacy memory limits, deploy a canary with representative prompts, and compare throughput, time to first token, inter-token latency, vllm:kv_cache_usage_perc, and preemption counts. High raw GPU memory use is expected under the new allocator. Deployments with strict latency targets can use --max-num-seqs to control concurrency while retaining the larger cache.