Skip to main content
Back

vLLM v0.23.0 Ships DeepSeek-V4 Production Hardening and 56% Throughput Boost

vLLM v0.23.0 ships 408 commits from 200 contributors, maturing DeepSeek-V4 support, expanding Model Runner V2, and adding multi-tier KV cache offloading to object stores.

Repo, Full Access
  • 408 commits, 200 contributors: vLLM v0.23.0 is a major consolidation release with broad model, hardware, and infrastructure improvements.
  • DeepSeek-V4 hardened: Sparse MLA decoupled from V3.2, TRTLLM-gen attention kernel, EPLB for Mega-MoE, and detached from torch.compile for production stability.
  • Model Runner V2 now default for Llama + Mistral: The rewritten execution core (previously default only for Qwen3) expands, bringing GPU-native input prep and async scheduling to more deployments.
  • Multi-tier KV cache offloading: New object-store secondary tier (S3-compatible) added beyond CPU DRAM, with per-request offloading policy via on_new_request hook.
  • Unified parser: Reasoning tokens and tool-call tokens now share a single Parser.parse() interface, simplifying agentic pipeline code.
  • Rust frontend matures: Streaming generate, dynamic LoRA endpoints, /version + /server_info, and new tool parsers added; full release notes on GitHub.

vLLM v0.23.0 is out, and it reads less like a patch release and more like a platform consolidation. The release packs 408 commits from 200 contributors, 63 of whom are first-timers. The headline themes are clear: DeepSeek-V4 gets a serious production hardening pass, the new Model Runner V2 execution core expands to cover Llama and Mistral, KV cache offloading grows a third storage tier, and the experimental Rust frontend keeps adding features at a steady clip.

DeepSeek-V4 Gets a Real Production Pass

DeepSeek-V4 was introduced in v0.22.0, but this release is where it starts to feel production-ready. Following its introduction, DeepSeek-V4 received another large hardening and optimization pass: its sparse MLA metadata is now decoupled from DeepSeek-V3.2, it gained a TRTLLM-gen attention kernel, EPLB support for the Mega-MoE, and selective prefix-cache retention for sliding-window KV cache.

The MLA (Multi-head Latent Attention) decoupling from V3.2 matters because it means the two models no longer share internal state that can cause subtle correctness bugs when running them side by side. The TRTLLM-gen attention kernel is a TensorRT-LLM-derived kernel that replaces a more generic path with one tuned specifically for DeepSeek's attention pattern. The model was also detached from torch.compile, and its attention and RoPE paths were refactored. Detaching from torch.compile trades some potential graph-optimization upside for more predictable startup times and easier debugging in production.

EPLB (Expert Parallel Load Balancer) is worth understanding if you run large MoE models at scale. While MoE models are typically trained so that each expert receives a similar number of tokens, in practice the distribution of tokens across experts can be highly skewed. vLLM's EPLB redistributes expert mappings across EP ranks, evening the load. To implement EPLB, each MoE forward pass records per-token load, and a sliding window aggregates these statistics across EP ranks. In v0.23.0, this mechanism now covers DeepSeek-V4's larger "Mega-MoE" configuration, which has more experts and a wider routing fan-out than V3.

Model Runner V2 Becomes the Default for Dense Models

Model Runner V2 (MRv2) is vLLM's ground-up rewrite of its core execution engine, and it's been rolling out model-by-model since it launched. The team revisited persistent batching, async scheduling, input preparation, and sampling, then rebuilt the model runner around three core principles: be modular (isolate model-specific logic from the common execution path), be GPU-native (move bookkeeping off the CPU and onto the GPU), and be async-first (treat overlapped CPU/GPU execution as a design constraint, not a retrofit).

The practical payoff is significant. On Qwen3-0.6B running on a single GB200, the team reported a 56.2% throughput increase, from 16K to 25K output tokens per second, after moving input preparation to the GPU. In a separate speculative-decoding test using GLM-4.7-FP8 with MTP=1 on 4xGB200, MRv2 lowered time per output token by 6.3% by eliminating CPU-GPU synchronization.

MRv2 is now selected by default for Llama and Mistral dense models, in addition to Qwen3. This release also adds several new capabilities to MRv2 itself:

  • FlashInfer sampler: replaces the previous sampling path with a Triton-native kernel that avoids materializing the full softmax distribution in memory
  • Breakable CUDA graphs: CUDA graphs (pre-recorded GPU command sequences for fast replay) can now be interrupted mid-execution, which unblocks certain dynamic workloads that previously had to fall back to eager mode
  • Pipeline-parallel bubble elimination: when running a model split across multiple GPUs in sequence (pipeline parallelism), there are idle gaps between stages; this change reduces those gaps
  • Gemma 4 MTP: Multi-Token Prediction support for Gemma 4, allowing the model to draft multiple tokens per step

The code structure is cleaner too: the old gpu_model_runner.py had grown past 6,700 lines, while the largest MRv2 file is now under 1,300 lines. That's not just aesthetics. Smaller, more focused files mean faster iteration and fewer latent bugs when adding new model support.

KV Cache Grows a Third Tier

KV cache management has always been one of vLLM's core differentiators, and v0.23.0 pushes it further. Previously, vLLM offered native KV offloading to CPU memory but did not support further offloading from CPU memory to other tiers such as storage. That changes now.

The offloading framework gained an object-store secondary tier, HMA enabled by default for capable connectors, tiering support for HMA models, and a per-request offloading policy via the on_new_request lifecycle hook. The object-store tier means you can now route evicted KV blocks to S3-compatible storage or similar systems, not just CPU DRAM. The on_new_request hook gives you per-request control: you can decide at request time whether a given conversation's KV cache should be offloaded aggressively, kept hot in GPU memory, or somewhere in between.

The 2026 inflection point is context length: 128K-token and 1M-token windows are production workloads now, not research benchmarks. Multi-tier offloading is the direct response to that pressure. A three-tier hierarchy (GPU HBM → CPU DRAM → object store) lets you serve far more concurrent long-context sessions on the same hardware.

The Rust Frontend Keeps Growing Up

The experimental Rust frontend, introduced in v0.22.0, is quietly becoming a real alternative to the Python serving layer. In this release it gained:

  • A streaming generate endpoint
  • Dynamic LoRA loading endpoints
  • /version and /server_info introspection endpoints
  • New tool parsers for InternLM2, Phi-4-mini, and Gemma4
  • An --enable-request-id-headers flag for tracing requests end-to-end
  • A --language-model-only flag that skips the multimodal processor entirely for pure-text workloads

The Rust frontend matters because Python's GIL and interpreter overhead become real bottlenecks at high request rates. A Rust layer handling HTTP routing, request parsing, and tool-call streaming can keep the Python engine focused on model execution. The streaming endpoint and dynamic LoRA support are the two features most likely to unlock production use cases that were previously awkward to implement.

Unified Parser: One Interface for Reasoning and Tool Calls

If you've built agentic pipelines on top of vLLM, you've probably hit the friction of reasoning tokens and tool-call tokens being handled by separate code paths. Reasoning and tool-call parsing are now unified behind a single Parser.parse() interface, with the Responses parser migrated to it. This means less conditional logic in your serving code and a more consistent streaming experience when a model interleaves chain-of-thought with tool invocations.

Hardware Coverage Expands Across the Board

The hardware story in this release is unusually broad. Key wins by platform:

  • NVIDIA: CUTLASS FP8 scaled-mm padding bypass delivers a +20% throughput gain on certain matrix shapes; MoE-permute buffer pre-allocation adds +9-14%; Triton MoE backend is now default on Hopper (H100/H200); NUMA auto-binding on DGX B300
  • AMD ROCm: Updated to ROCm 7.2.3, native W4A16 and fused-MoE W4A16 kernels for RDNA3 (RX 7900 series), AITER top-k/top-p sampler now default
  • Intel XPU: Block FP8 MoE, a DeepSeek-V4 attention decode path, and transparent sleep mode (the GPU can power down between requests without a full reload)
  • CPU and other architectures: zentorch-accelerated W8A8/W4A16 on AMD Zen CPUs, RISC-V RVV WNA16 helpers, a PowerPC SHM communicator, and an arm64 CI image

New Models and Security Fixes

Six new models land in this release: Step-3.7-Flash, Cosmos3 Reasoner, JetBrains Mellum v2, Granite Speech Plus, Cohere Mini Code, and Gemma 4 Unified (the encoder-free variant). The Gemma 4 Unified support is notable because it removes the vision encoder from the serving path for text-only use, reducing memory footprint and startup time.

On the security side, two fixes stand out. First, out-of-vocabulary token IDs are now rejected before they reach the GPU logprob path, preventing a class of request-triggered crashes. Second, SSL/TLS support was added for the data-parallel supervisor, which matters for multi-node deployments where the supervisor communicates across a network boundary.

What to Watch Before Upgrading

A few things worth checking before you upgrade:

  • MRv2 is now default for Llama and Mistral. If you rely on features not yet covered by MRv2 (certain LoRA configurations, some logits processors), vLLM will fall back to MRv1 automatically, but it's worth verifying your setup
  • The NixlConnector kv_both role has entered its deprecation cycle. If you use disaggregated serving with Nixl, plan to migrate
  • Functions previously deprecated in v0.21.0 have now been removed. Check the release notes if you use any internal APIs
  • Minimax M3 is explicitly not supported in this version

vLLM is available now via pip install vllm==0.23.0. It's free and open-source under the Apache 2.0 license. The full release notes are on GitHub.