Skip to main content
Back

vLLM-Omni Cuts Multimodal AI Serving Time by 91% With Open-Source Framework

vLLM-Omni brings production-grade serving to any-to-any multimodal models, cutting job completion time by up to 91% with a fully disaggregated pipeline.

  • vLLM-Omni is an open-source, Apache 2.0 framework from the vLLM project for serving any-to-any multimodal models (text, image, audio, video, actions).
  • Its fully disaggregated pipeline decomposes models into independent stages (encoders, LLM core, diffusion generators), reducing job completion time by up to 91.4% vs. sequential baselines.
  • The OmniConnector routes tensors between stages via shared memory (single-node) or RDMA (multi-node), with direct KV cache transfer to avoid redundant recomputation.
  • Supports a broad model zoo: Qwen3-Omni, BAGEL, FLUX, Wan2.2, Cosmos3, MiniCPM-o 4.5, FLUX, robot-policy models, and more.
  • Exposes an OpenAI-compatible API, runs on CUDA/ROCm/NPU/XPU, and supports TP/PP/DP/EP distributed inference.
  • Free on PyPI; docs at vllm-omni.readthedocs.io; paper at arXiv:2602.02204.

Serving a model that can take in a video, reason about it, and respond with synthesized speech is a fundamentally different engineering problem than serving a text-only LLM. The compute stages are heterogeneous, the output types are mixed, and existing inference engines were never designed to handle it. vLLM-Omni is the vLLM project's answer to that problem: a fully disaggregated serving framework built from the ground up for any-to-any multimodal models.

The gap that needed filling

Modern any-to-any models combine multiple autoregressive LLMs, diffusion transformers, and other specialized components -- but existing serving systems are mainly tailored to a single paradigm, like autoregressive LLMs for text or diffusion transformers for visual generation, and lack support for pipelines that involve multiple interconnected model components. As a result, developers have had to manually handle cross-stage interactions, leading to significant performance degradation.

Traditional serving engines were optimized for text-based autoregressive tasks. As models evolve into "omni" agents -- capable of seeing, hearing, and speaking -- the serving infrastructure must evolve with them. vLLM-Omni is the infrastructure catching up to the models.

What it actually does

vLLM-Omni is a fully disaggregated serving system for any-to-any models. It features a novel stage abstraction that enables users to decompose complex any-to-any architectures into interconnected stages represented as a graph. Think of it as a DAG (directed acyclic graph) scheduler for model components -- each stage runs independently, and the system routes tensors between them efficiently.