vLLM-Omni Cuts Multimodal AI Serving Time by 91% With Open-Source Framework
vLLM-Omni brings production-grade serving to any-to-any multimodal models, cutting job completion time by up to 91% with a fully disaggregated pipeline.
Takeaways
- vLLM-Omni is an open-source, Apache 2.0 framework from the vLLM project for serving any-to-any multimodal models (text, image, audio, video, actions).
- Its fully disaggregated pipeline decomposes models into independent stages (encoders, LLM core, diffusion generators), reducing job completion time by up to 91.4% vs. sequential baselines.
- The OmniConnector routes tensors between stages via shared memory (single-node) or RDMA (multi-node), with direct KV cache transfer to avoid redundant recomputation.
- Supports a broad model zoo: Qwen3-Omni, BAGEL, FLUX, Wan2.2, Cosmos3, MiniCPM-o 4.5, FLUX, robot-policy models, and more.
- Exposes an OpenAI-compatible API, runs on CUDA/ROCm/NPU/XPU, and supports TP/PP/DP/EP distributed inference.
- Free on PyPI; docs at vllm-omni.readthedocs.io; paper at arXiv:2602.02204.
Serving a model that can take in a video, reason about it, and respond with synthesized speech is a fundamentally different engineering problem than serving a text-only LLM. The compute stages are heterogeneous, the output types are mixed, and existing inference engines were never designed to handle it. vLLM-Omni is the vLLM project's answer to that problem: a fully disaggregated serving framework built from the ground up for any-to-any multimodal models.
The gap that needed filling
Modern any-to-any models combine multiple autoregressive LLMs, diffusion transformers, and other specialized components -- but existing serving systems are mainly tailored to a single paradigm, like autoregressive LLMs for text or diffusion transformers for visual generation, and lack support for pipelines that involve multiple interconnected model components. As a result, developers have had to manually handle cross-stage interactions, leading to significant performance degradation.
Traditional serving engines were optimized for text-based autoregressive tasks. As models evolve into "omni" agents -- capable of seeing, hearing, and speaking -- the serving infrastructure must evolve with them. vLLM-Omni is the infrastructure catching up to the models.
What it actually does
vLLM-Omni is a fully disaggregated serving system for any-to-any models. It features a novel stage abstraction that enables users to decompose complex any-to-any architectures into interconnected stages represented as a graph. Think of it as a DAG (directed acyclic graph) scheduler for model components -- each stage runs independently, and the system routes tensors between them efficiently.