- Pull
vllm/vllm-openai:latestand run it with--runtime nvidia --gpus all --ipc=hostto get a GPU-backed OpenAI-compatible server. - Mount
~/.cache/huggingfaceinto the container so model weights survive container restarts. - The server exposes an OpenAI-compatible API on port 8000; test it with
curl http://localhost:8000/v1/models. - The three most important tuning flags are
--tensor-parallel-size,--max-model-len, and--gpu-memory-utilization.
vLLM publishes an official Docker image, vllm/vllm-openai, that ships a ready-to-run OpenAI-compatible inference server. The fastest path: install the NVIDIA Container Toolkit on the host, then run the image with --gpus all and a Hugging Face model ID. Once the model downloads, the server is live on port 8000 and accepts the same requests as the OpenAI API.
Prerequisites
- Docker Engine 20.10+ — Docker Desktop on Windows and macOS works via the WSL2 backend.
- NVIDIA GPU with a driver that supports CUDA 12.x. Run
nvidia-smito confirm; the “CUDA Version” shown is the maximum your driver supports. - NVIDIA Container Toolkit — the bridge that lets Docker see the GPU. See the next section.
- Enough VRAM for your target model. Use the VRAM calculator to check before committing to a model download.
Installing the NVIDIA Container Toolkit
Skip this section if docker run --gpus all nvidia/cuda:12.0-base nvidia-smi already works on your machine.
Linux (Ubuntu/Debian):
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey |
sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list |
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' |
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart dockerRHEL/CentOS: Replace the deb repository URL with the rpm equivalent from NVIDIA’s documentation and use dnf in place of apt-get.
Windows (WSL2): Install the NVIDIA driver for Windows on the host — no separate container toolkit install is needed inside WSL2. Docker Desktop handles the passthrough automatically.
macOS: NVIDIA GPUs are not supported on macOS. vLLM does not run on Apple Silicon via Docker with GPU acceleration. For local inference on Apple hardware, consider a CPU-only build or a different runtime.
The Minimal Run Command
docker run --runtime nvidia --gpus all
--ipc=host
-v ~/.cache/huggingface:/root/.cache/huggingface
-p 8000:8000
vllm/vllm-openai:latest
--model meta-llama/Meta-Llama-3-8B-Instruct| Flag | Why it matters |
|---|---|
--runtime nvidia | Routes GPU calls through the NVIDIA container runtime. |
--gpus all | Exposes all host GPUs. Use "device=0,1" to target specific GPUs. |
--ipc=host | Shares the host IPC namespace. Required for PyTorch shared memory; omitting it causes a Bus error or shared-memory crash. |
-v ~/.cache/huggingface:… | Mounts the host HF cache so weights survive container restarts. |
-p 8000:8000 | Exposes the OpenAI-compatible server on the host. |
--model | Any Hugging Face model ID or a local path mounted into the container. |
To pull a gated model (Llama 3, Mistral, etc.), also pass -e HUGGING_FACE_HUB_TOKEN=hf_yourtoken. Store the token in a .env file and pass it with --env-file .env rather than inlining it in your shell history.
Mounting the Hugging Face Cache
vLLM downloads model weights to /root/.cache/huggingface inside the container. Without a volume mount, every docker run re-downloads the full model — often 5–80 GB. The mount line is:
-v ~/.cache/huggingface:/root/.cache/huggingfaceIf your models live in a non-default location, set -e HF_HOME=/your/path and mount that path instead. For air-gapped environments, download the model with huggingface-cli download first, then pass --model /path/in/container alongside a volume mount of the weights directory.
Key vLLM Server Flags
These flags are passed after the image name — they are arguments to the vLLM server process, not to Docker.
| Flag | Default | When to change it |
|---|---|---|
--tensor-parallel-size N | 1 | Set to the number of GPUs for multi-GPU serving. The model’s attention heads must be divisible by N. Pair with --gpus "device=0,1,..." listing exactly N devices. |
--gpu-memory-utilization 0.X | 0.90 | Lower to 0.75–0.80 if you see OOM errors or share the GPU with other processes. |
--max-model-len N | Model config | Caps the KV cache size. Useful when a model’s default context (e.g. 128 k) would exhaust VRAM. Try --max-model-len 8192 as a first reduction. |
--dtype auto | auto | Override with bfloat16 or float16 if auto-detection picks an unexpected precision. |
--quantization awq / gptq | none | Enable for pre-quantized model variants. Roughly halves VRAM at some quality cost. |
--port | 8000 | Change if 8000 is already occupied on the host. |
Not sure whether your GPU has enough VRAM for a given model? The VRAM requirements guide lists common models, and the VRAM calculator lets you plug in quantization and batch size. For hardware purchasing decisions, see the GPU recommendations guide.
Exposing and Testing the OpenAI-Compatible Endpoint
Once the container prints INFO: Application startup complete, the API is live.
# List loaded models
curl http://localhost:8000/v1/models
# Text completion
curl http://localhost:8000/v1/completions
-H "Content-Type: application/json"
-d '{"model": "meta-llama/Meta-Llama-3-8B-Instruct", "prompt": "The capital of France is", "max_tokens": 20}'
# Chat completion
curl http://localhost:8000/v1/chat/completions
-H "Content-Type: application/json"
-d '{"model": "meta-llama/Meta-Llama-3-8B-Instruct", "messages": [{"role": "user", "content": "Hello"}]}'Any OpenAI-compatible client — the Python openai SDK, LangChain, LlamaIndex — works by setting base_url="http://localhost:8000/v1" and providing any non-empty string as the api_key.
Docker Compose Example
For persistent deployments, a Compose file is easier to manage than a long docker run command:
services:
vllm:
image: vllm/vllm-openai:latest
runtime: nvidia
environment:
- HUGGING_FACE_HUB_TOKEN=${HF_TOKEN}
volumes:
- ~/.cache/huggingface:/root/.cache/huggingface
ports:
- "8000:8000"
ipc: host
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]
command: >
--model meta-llama/Meta-Llama-3-8B-Instruct
--gpu-memory-utilization 0.90
--max-model-len 8192Start with docker compose up -d. The deploy.resources block is the Compose v3 equivalent of --gpus all.
Common Failures and Fixes
| Error | Cause | Fix |
|---|---|---|
| Bus error or /dev/shm too small | Docker’s default /dev/shm is 64 MB — too small for PyTorch. | Add --ipc=host to the run command. Alternatively use --shm-size=8g if you cannot share the host IPC namespace. |
| CUDA error: no kernel image is available | The CUDA version compiled into the vLLM image exceeds what your driver supports. | Run nvidia-smi to find your max supported CUDA version, then pin to a matching versioned image tag (e.g. vllm/vllm-openai:v0.5.5). |
| torch.cuda.OutOfMemoryError | Model weights plus KV cache exceed available VRAM. | Try --max-model-len 4096 first. Then lower --gpu-memory-utilization to 0.80. If still OOM, use a quantized variant or a larger GPU. |
| permission denied on cache directory | Container runs as root; host directory owned by another user. | Run chmod -R a+rw ~/.cache/huggingface on the host, or use a named Docker volume instead of a bind mount. |
Container starts but curl returns connection refused | Model still loading, or -p 8000:8000 missing. | Wait for the Application startup complete log line. Verify the port mapping is present in your run command. |
Frequently Asked Questions
Which vLLM Docker image tag should I use?
vllm/vllm-openai:latest tracks the most recent release and is fine for experimentation. For production, pin to a specific version tag (e.g. v0.6.0) so builds are reproducible. Each release tag on Docker Hub indicates the CUDA version it was compiled against, which must be less than or equal to the version your driver supports.
Can I run vLLM in Docker without a GPU?
The standard image requires an NVIDIA GPU. CPU-only inference is possible by building vLLM from source with VLLM_TARGET_DEVICE=cpu, but throughput is orders of magnitude slower and not practical for serving. For CPU-only local inference, llama.cpp or Ollama are more suitable alternatives — see the Ollama guide for comparison.
How do I run a gated model that requires a Hugging Face token?
Pass the token as an environment variable: -e HUGGING_FACE_HUB_TOKEN=hf_yourtoken. Store it in a .env file and reference it with --env-file .env to avoid leaking it in shell history. The server uses it during the initial model download; it is not required after the weights are cached locally.
What does –tensor-parallel-size do and when do I need it?
Tensor parallelism shards model weight matrices across multiple GPUs, enabling models too large for a single card. Set it to the number of GPUs you want to use (2 or 4 are common). The number must match the GPU count passed to --gpus, and the model’s attention heads must be divisible by that number.
Is running vLLM in Docker cost-effective compared with a managed API?
It depends entirely on your request volume. Self-hosting has high fixed costs (GPU instance or hardware) but near-zero marginal cost per request. Managed APIs have zero fixed cost but charge per token. Use the self-hosting vs API calculator to find your break-even point before committing to infrastructure.
How do I serve multiple models at once?
Run one container per model, each mapped to a different host port (e.g. 8000, 8001). vLLM does not currently support multi-model serving from a single process. Place a reverse proxy such as nginx or Caddy in front of the containers to route requests by model name to the correct port.
