vLLM Omni-Modal Support: Running Multimodal Models with vLLM

vLLM omni is not a separate product. It is shorthand for using vLLM — the high-throughput LLM inference engine — to serve omni-modal models, meaning models that accept text, images, audio, and video in a single conversation. In practice this almost always means Alibaba’s Qwen2.5-Omni series, though vLLM’s multimodal stack also handles vision-only and audio-only models through the same API.

This guide covers what “omni” means inside vLLM, which models are supported, how to install and serve them, what the request format looks like, and where the current limitations are.

What “Omni” Means in the vLLM Context

vLLM’s multimodal subsystem groups models by the modalities they accept on the input side. The relevant categories are:

Omni-modal models share a single language backbone with separate encoders for each modality (a ViT for images and video frames, an audio encoder derived from Whisper-style architectures, and so on). vLLM’s job is to schedule these encoders, cache their outputs, and interleave them with the text token stream in the KV cache so throughput stays high.

Output-side generation in vLLM is text. If you want the audio-out capability that Qwen2.5-Omni supports natively (speech synthesis), you currently need the reference implementation from the model authors — vLLM will give you the text transcript, not the audio waveform. This is the single biggest thing to understand before choosing vLLM for an omni workload.

Supported Omni-Modal Models

The authoritative list lives in the vLLM docs under Supported Models → Multimodal Language Models. As a stable reference point, these families have had upstream support for some time:

• Qwen2.5-Omni (3B, 7B) — the canonical “omni” model, text + image + audio + video in, text out.
• MiniCPM-o 2.6 — 8B-class omni model from OpenBMB.
• Qwen2-VL / Qwen2.5-VL — vision + video only, but often lumped in with “omni” workflows.
• Qwen2-Audio — audio-only companion.

Because model support is added per-release, always check the current docs and the model card for the minimum vLLM version. Trying to serve a new omni model on an old vLLM build is the most common failure mode. Browse an up-to-date models database if you are still choosing.

Hardware Requirements

Omni-modal models are heavier than their text-only siblings at the same parameter count because they carry additional encoders and because vision/audio inputs consume many tokens after tokenization. A single image at native resolution can expand to 1,000–4,000 tokens; a minute of audio to several hundred.

These are practical ranges, not spec-sheet minimums. For an exact figure at your context length and batch size, plug the model into the VRAM calculator or consult the VRAM requirements reference. If you have not chosen hardware yet, the best GPUs for local LLMs guide covers the trade-offs at the 24 GB, 48 GB, and multi-GPU tiers.

Installation

vLLM is Linux-first. Windows is not officially supported; use WSL2 or a Linux container. macOS has a CPU-only build that will technically load small models but is not viable for omni-modal serving in production.

Linux (recommended)

Requirements: a CUDA-capable GPU (compute capability 7.0+), CUDA 12.x drivers, Python 3.9–3.12.

# Create an isolated environment
python -m venv vllm-env
source vllm-env/bin/activate

# Install vLLM (this pulls a matching PyTorch build)
pip install vllm

# Extra dependencies commonly needed for omni models
pip install librosa soundfile decord

vllm --version

The librosa, soundfile, and decord packages handle audio decoding and video frame extraction. Some omni models pull these in automatically via trust_remote_code; installing them upfront avoids first-request failures.

Windows (via WSL2)

Install WSL2 with an Ubuntu 22.04 or 24.04 distribution, install the NVIDIA driver on Windows (the WSL side uses the Windows driver directly), then follow the Linux steps inside WSL. Do not install a separate Linux NVIDIA driver inside WSL — that will break CUDA.

macOS

There is no CUDA on macOS and vLLM does not target Metal. For local multimodal inference on Apple Silicon, use LM Studio or MLX-based runtimes instead. vLLM is the wrong tool here.

Serving an Omni Model

Launch an OpenAI-compatible server:

vllm serve Qwen/Qwen2.5-Omni-7B 
  --trust-remote-code 
  --dtype bfloat16 
  --max-model-len 32768 
  --limit-mm-per-prompt image=4,audio=2,video=1 
  --port 8000

Key flags:

• --trust-remote-code is required because omni models ship custom preprocessing code in their Hugging Face repos.
• --limit-mm-per-prompt caps the number of each modality per request. Raising these increases the multimodal input budget but also VRAM pressure.
• --max-model-len should be set explicitly. Vision and audio tokens count against it.
• --tensor-parallel-size N shards across N GPUs if a single card is too small.

Making Multimodal Requests

vLLM exposes the OpenAI Chat Completions API. Multimodal parts follow the OpenAI content-array convention:

curl http://localhost:8000/v1/chat/completions 
  -H "Content-Type: application/json" 
  -d '{
    "model": "Qwen/Qwen2.5-Omni-7B",
    "messages": [{
      "role": "user",
      "content": [
        {"type": "text", "text": "Describe what you see and hear."},
        {"type": "image_url", "image_url": {"url": "https://example.com/scene.jpg"}},
        {"type": "audio_url", "audio_url": {"url": "https://example.com/clip.wav"}}
      ]
    }]
  }'

The Python client is identical to OpenAI’s:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

resp = client.chat.completions.create(
    model="Qwen/Qwen2.5-Omni-7B",
    messages=[{"role": "user", "content": [
        {"type": "text", "text": "Transcribe and summarise."},
        {"type": "audio_url", "audio_url": {"url": "file:///data/meeting.wav"}},
    ]}],
)
print(resp.choices[0].message.content)

Data URLs (data:image/png;base64,...) and local file:// paths both work; the exact set of accepted schemes has expanded across releases, so consult the version’s docs if a scheme fails.

Performance Notes

• Prefill dominates. Encoding a 1-minute audio clip or a 720p image is CPU/GPU-heavy and happens before the first token. Batching helps throughput but not per-request latency.
• KV cache pressure. A single image can add thousands of tokens to the cache. Reduce --max-model-len or lower --limit-mm-per-prompt if you hit OOM under load.
• Quantisation. AWQ and GPTQ 4-bit variants of Qwen2.5-Omni-7B fit on a 16 GB card and lose relatively little quality on vision/audio understanding tasks. Availability depends on community uploads on Hugging Face.
• Chunked prefill (enabled by default in recent versions) smooths latency when mixing multimodal and text-only requests.

When to Use vLLM Omni vs Alternatives

If you are still deciding whether to self-host at all, run the numbers through the self-hosting vs API break-even calculator. Omni-modal workloads tilt the answer because per-request token counts are much higher than text-only, which makes usage-based API pricing more expensive per session.

Frequently Asked Questions

Does vLLM support Qwen2.5-Omni’s speech output?

No. vLLM handles text generation from the language backbone. The optional audio-decoder head that produces spoken responses in the reference Qwen2.5-Omni implementation is not wired into vLLM. You get the model’s text response, and you would need a separate TTS step or the original inference code to synthesise speech.

Can I run vLLM omni models on a 24 GB card like an RTX 4090?

Yes for the 3B and 7B variants, especially at bf16 with modest context, or comfortably with AWQ/GPTQ quantisation at longer contexts. You will need to tune --max-model-len and --limit-mm-per-prompt to stay under the memory ceiling. Verify with the VRAM calculator before committing.

Why does my request fail with a “trust_remote_code” error?

Omni-modal models ship custom Python preprocessors in their Hugging Face repos. vLLM will not execute this code unless you pass --trust-remote-code at server startup. Only enable this for model repositories you trust.

How do I send video to a vLLM omni endpoint?

Use a content part with "type": "video_url" pointing at an accessible URL or local file. vLLM samples frames using decord; the exact frame-count and sampling policy is model-specific and documented on the model card. Video eats tokens quickly, so keep clips short and set --limit-mm-per-prompt video=1 unless you have plenty of VRAM.

Is there a Docker image for vLLM omni?

The official vllm/vllm-openai image on Docker Hub supports multimodal models out of the box for whichever vLLM version it is tagged with. Pin to a specific tag rather than latest so your omni model’s required version does not drift underneath you.

Can Ollama run these omni models instead?

Ollama supports some vision-language models but its omni-modal coverage lags vLLM’s, and audio/video inputs are limited or absent for most models. For a desktop workflow, check the best local models for Ollama and the Ollama models list to see what is currently available; for full omni features on a server, stick with vLLM.