vLLM v0.22.0 Ships a Rust Frontend and Cuts Latency by 28.9%
vLLM v0.22.0 lands a Rust frontend, 28.9% latency gains from batch-invariant FP8, and production-grade DeepSeek V4 support across NVIDIA, AMD, and CPU hardware
Takeaways
- vLLM v0.22.0 ships 459 commits from 230 contributors with a Rust frontend, 28.9% latency gains, and production-grade DeepSeek V4 support.
- Experimental Rust frontend replaces Python's FastAPI server to eliminate GIL and asyncio bottlenecks at high concurrency; enabled via
VLLM_USE_RUST_FRONTEND=1. - Batch-invariant Cutlass FP8 delivers 28.9% end-to-end latency improvement and 13.5% TTFT gain, making deterministic inference nearly free.
- DeepSeek V4 reorganized into a dedicated package with NVFP4 fused MoE, full CUDA graphs, and MTP speculative decoding on both NVIDIA and AMD ROCm.
- Multi-tier KV cache offloading now cascades from GPU to CPU RAM to NVMe disk, extending context length and batch size on existing hardware.
- Breaking changes: removed
get_tokenizerandresolve_hf_chat_templateold import paths, and deprecated MLA prefill arguments.
vLLM v0.22.0 is out, and it is one of the most architecturally ambitious releases the project has shipped. Batch-invariant inference now gains Cutlass FP8 support for a 28.9% end-to-end latency improvement, and an experimental Rust frontend has moved in-tree. Under the hood, this release is really about three converging bets: a new language for the serving layer, a maturing execution pipeline, and a hardware expansion that now stretches from NVIDIA Blackwell to AMD MI300X to RISC-V CPUs.
459 commits, 230 contributors, one big question
The scale of this release is notable. This release features 459 commits from 230 contributors, 63 of them new. But raw commit count is not the story. The real question is: what does this release change about how you run inference at scale? The answer is: quite a lot, and in ways that will compound over the next several releases.
Why Rust? The Python bottleneck is real
The most architecturally significant move in v0.22.0 is the experimental Rust frontend landing in-tree. This is not a cosmetic change. vLLM has always had a strong Python bias to make it accessible to contributors and to exploit ML libraries like PyTorch and Triton. But Python carries inherent performance downsides, including garbage collection and restricted parallelism due to the GIL.
As GPU latency continues to fall, request concurrency grows, and large-scale deployments become the norm, the CPU parts of the system have become a bottleneck. This is often seen in the frontend process where the asyncio event loop cannot keep up. In other words: GPUs got so fast that Python is now the slow part.
The Rust frontend is an experimental, high-performance alternative to the Python-based FastAPI server. It provides an OpenAI-compatible HTTP interface while leveraging Rust's concurrency model and memory safety, communicating with the vLLM Python engine via a ZeroMQ transport layer.
You can try it today by setting a single environment variable:
VLLM_USE_RUST_FRONTEND=1 vllm serve meta-llama/Llama-3.1-8B-Instruct
The Rust frontend is still experimental and not yet feature-complete relative to the Python frontend. A roadmap issue tracks the remaining gaps in feature parity. Think of it as a preview of where vLLM's serving layer is heading, not a drop-in replacement for production today.
Batch invariance: determinism without the performance tax
Batch invariance is a property that guarantees identical model outputs regardless of how many requests are batched together. Normally, LLM inference is non-deterministic across batch sizes because GPU reduction operations (like summing activations) execute in different orders depending on how many tokens are being processed simultaneously. Batch-invariant computation constrains a GPU kernel to use a single, universal reduction strategy for all tokens, eliminating batch-dependent reductions.
This matters more than it sounds. Batch invariance is crucial for framework debugging (deterministic outputs make it easier to isolate issues), model debugging (consistent behavior across batch configurations), and reinforcement learning training, which often requires deterministic rollouts for reproducibility and stable training.
The previous challenge was that enabling batch invariance came with a performance penalty. v0.22.0 closes that gap significantly. The batch-invariant Cutlass FP8 path delivers a 28.9% end-to-end latency improvement, CutlassFP8 padding pre-processing adds another 13.5% improvement in time-to-first-token, and the padded NVFP4 quantization kernel contributes a 2.4 to 5.7% end-to-end gain.
To enable it:
import os
os.environ["VLLM_BATCH_INVARIANT"] = "1"
from vllm import LLM, SamplingParams
llm = LLM(model="deepseek-ai/DeepSeek-V3", tensor_parallel_size=8)
# Outputs are now identical regardless of batch size
outputs = llm.generate(prompts, SamplingParams(temperature=0.7, seed=42))
DeepSeek V4 gets a proper home
DeepSeek V4 received what the team calls a "hardening pass." The model was reorganized into a dedicated vllm/models/deepseek_v4/ package, gained NVFP4 fused MoE support, full and piecewise CUDA graph support, and MTP speculative decoding. A large set of fused kernels covering MegaMoE, multi-head compression, Q-norm, indexer, and sparse MLA landed alongside accuracy fixes.
MTP (Multi-Token Prediction) speculative decoding is worth calling out specifically. It lets DeepSeek V4 predict multiple tokens per forward pass using a lightweight draft head, then verify them in parallel. This reduces the number of sequential decoding steps without changing output quality. The feature now works on both NVIDIA and AMD ROCm.
Model Runner V2 marches toward default
Model Runner V2 (MRv2) is vLLM's refactored execution pipeline, designed to eliminate the complex bookkeeping that accumulated in the original runner. MRv2 is now the default for Qwen3 dense models, and vLLM will automatically fall back to MRv1 for features not yet supported in MRv2.
New capabilities in MRv2 this cycle:
- Sleep-mode weight reload: unload weights to free GPU memory, then reload on demand without restarting the server
update_config: change serving configuration at runtime without a full restart- Shared KV-cache layers: models that share transformer layers can now share their KV cache entries, reducing memory pressure
- Auto-fallback: if a KV connector is present (e.g., for disaggregated serving), MRv2 gracefully falls back to MRv1 rather than failing
KV cache offloading goes multi-tier
KV cache (the memory that stores intermediate attention states across tokens) is the primary GPU memory consumer during inference. When it fills up, requests stall or get preempted. A new multi-tier KV cache offloading framework extends offloading beyond CPU memory, adding a Python filesystem secondary tier, DeepSeek V4 support, and Mooncake disk offloading.
The practical implication: you can now serve longer contexts and larger concurrent batches on the same hardware by spilling KV cache to CPU RAM and then to NVMe disk, in a tiered cascade. This is especially relevant for long-document or multi-turn workloads where KV cache growth is the binding constraint.
Hardware coverage: Blackwell, ROCm, and RISC-V
vLLM's hardware story keeps expanding. Key additions this cycle:
- NVIDIA Blackwell (SM120/121): FlashInfer MoE and FP4 GEMM support for SM120/121, plus per-tensor FP8 CUTLASS on SM12.1.
- AMD ROCm: DeepSeek V4 functionality gains, flash sparse MLA Triton kernels, and gluon paged MQA logits on the MI355X.
- CPU / RISC-V: RVV-optimized attention kernels for the RISC-V Vector Extension, fused GDN for Intel AMX, and experimental Triton plus MRv2 on CPU.
- Intel XPU: GPTQ int4 support, MXFP8 MoE, and FP8 block-scaled quantization.
What else landed
A few more items worth knowing before you upgrade:
- New model architectures: MiniCPM-V 4.6, InternS2 Preview, and OpenVLA are now supported.
- Speculative decoding: custom callable proposer backends let you plug in your own draft model logic; Gemma3 and Gemma4 get multi-GPU fixes and batched vision encoders.
- Tool calling: improvements across Qwen3Coder, MiniMax-M2, DeepSeek-V3.2, and Seed-OSS parsers, with shared schema coercion logic.
- Quantization: MXFP4 linear layers with compressed-tensors integration; AutoRound W4A16 support; GPTQ consolidation merging
gptq_marlinintoauto_gptq.
Breaking changes to know before upgrading
Three removals that may affect existing code:
- The old import locations for
get_tokenizerandresolve_hf_chat_templateare gone. Update any direct imports to the new paths. - Deprecated MLA prefill arguments have been removed. Check your DeepSeek serving configs.
- Dead CUDA kernels and associated dead code have been cleaned up, which should not affect behavior but may surface if you were relying on undocumented internals.
The bigger picture
vLLM is quietly making a bet that the inference serving stack needs to be rebuilt from the ground up, not just patched. The Rust frontend is the clearest signal of this: the Python frontend code has become increasingly complex and fragile as it evolved organically, and with the advent of coding agents, the Python versus Rust contributor accessibility issue is minimized. The team is betting that Rust's performance characteristics are worth the migration cost, and that AI-assisted development lowers the barrier to contributing in a new language.
Meanwhile, the batch-invariance work is closing one of the last gaps between research reproducibility and production serving. RL training requires deterministic rollouts for reproducibility and stable training, and large-scale inference systems benefit from deterministic behavior for testing, validation, and consistency guarantees. Getting that for free at 28.9% lower latency is a meaningful step toward making vLLM the default choice for both research and production workloads simultaneously. Install it now via pip install vllm==0.22.0 and it is free and open source.