How to Install vLLM on Linux, Windows and macOS

To install vLLM on Linux with an NVIDIA GPU, create a clean Python 3.12 environment and run pip install vllm, then start a server with vllm serve Qwen/Qwen3-8B. That is the entire happy path. Windows requires WSL2 or Docker because vLLM publishes Linux-only wheels, and macOS requires a build from source that runs CPU-only inference.

Before you install: what vLLM actually requires

Requirement	What works
Operating system	Linux (x86_64 is the primary target; some releases also publish aarch64 wheels). Windows only via WSL2 or Docker. macOS via source build.
Python	3.9–3.12 covers releases through most of 2025, with 3.13 added in later versions. The supported range shifts between releases — check the release notes for the version you install.
GPU	NVIDIA GPUs with compute capability 7.0 or higher (V100, T4, RTX 20-series and newer, A10, L4, A100, H100, H200). AMD cards need the separate ROCm build.
CUDA	A current NVIDIA driver. The default PyPI wheel ships with the CUDA runtime that its bundled PyTorch build needs, so you do not need a system CUDA toolkit unless you compile from source.
Disk	The package itself is a few GB. Model weights dominate — they land in ~/.cache/huggingface and range from a few GB to hundreds of GB.

The authoritative and continuously updated list lives in the project’s own documentation at docs.vllm.ai, and version-specific changes are recorded on the vLLM releases page. Published wheels and their supported Python versions are listed on PyPI.

Install vLLM on Linux with an NVIDIA GPU

Option 1: uv (fastest, and what the vLLM docs now recommend)

curl -LsSf https://astral.sh/uv/install.sh | sh
uv venv vllm-env --python 3.12 --seed
source vllm-env/bin/activate
uv pip install vllm --torch-backend=auto

The --torch-backend=auto flag lets uv detect your driver and pick a matching PyTorch/CUDA build. If your driver is older than the wheel expects, replace auto with an explicit backend such as cu126. Which CUDA builds are available changes with each release, so consult the installation page rather than assuming a specific tag exists.

Option 2: plain pip and venv

python3.12 -m venv ~/vllm-env
source ~/vllm-env/bin/activate
pip install --upgrade pip
pip install vllm

Option 3: conda

conda create -n vllm python=3.12 -y
conda activate vllm
pip install vllm

Note that vLLM should be installed with pip even inside a conda environment. Do not install PyTorch separately first — vLLM will pull the exact torch build it was compiled against.

Verify the install

vllm --version
python -c "import torch; print(torch.__version__, torch.cuda.is_available())"
nvidia-smi

If torch.cuda.is_available() prints False, stop here — the problem is your driver or environment, not vLLM, and no serve command will work until that reads True.

Install with Docker (the most reproducible option)

The project publishes an official OpenAI-compatible server image. This skips Python environment problems entirely and is the route worth taking on a shared or production box:

docker run --runtime nvidia --gpus all 
  -v ~/.cache/huggingface:/root/.cache/huggingface 
  --env "HF_TOKEN=$HF_TOKEN" 
  -p 8000:8000 
  --ipc=host 
  vllm/vllm-openai:latest 
  --model Qwen/Qwen3-8B

The --ipc=host flag matters: vLLM uses shared memory between processes, and Docker’s small default /dev/shm causes crashes with tensor parallelism. If you cannot use host IPC, pass --shm-size=8g instead. Mounting the Hugging Face cache means you download each model once rather than once per container. This requires the NVIDIA Container Toolkit on the host.

Install vLLM on Windows (WSL2)

vLLM has no native Windows build. WSL2 is the supported path and works well:

1. Install the standard NVIDIA Windows driver. Do not install a Linux display driver inside WSL — the WSL CUDA stack maps through the Windows driver. NVIDIA documents this in the CUDA on WSL user guide.
2. In PowerShell: wsl --install -d Ubuntu-24.04, then reboot if prompted.
3. Inside Ubuntu, run nvidia-smi. If your GPU is not listed, fix that before continuing.
4. Install python3.12-venv, create a venv, and follow the Linux instructions above.

Two WSL-specific gotchas: WSL caps RAM by default, so add a [wsl2] section with memory= to C:Users<you>.wslconfig if model loading gets killed; and model weights stored on the Windows filesystem (/mnt/c/...) load noticeably slower than weights inside the WSL filesystem.

Install vLLM on macOS

There is no macOS wheel on PyPI. Apple silicon support is a CPU-only source build:

xcode-select --install
git clone https://github.com/vllm-project/vllm.git
cd vllm
pip install -r requirements/cpu.txt
pip install -e .

The requirements file path has moved between versions (it was requirements-cpu.txt in older releases), so check the repository tree for the tag you check out. Critically, this build does not use Metal or the Apple GPU — inference runs on CPU and is far slower than a CUDA machine. vLLM’s design goal is high-throughput batched serving on server GPUs, which is not what a Mac laptop is. If your goal is running a model locally on macOS, use the Ollama route or LM Studio, both of which use Metal properly.

AMD, Intel and CPU-only Linux

ROCm (AMD), Intel GPU/XPU and CPU-only x86 all have their own installation paths, generally either a prebuilt Docker image or a source build with a target-device environment variable such as VLLM_TARGET_DEVICE=cpu. These backends move faster than the CUDA path and the exact commands change between releases, so follow the hardware-specific page in the current docs rather than copying a command from a tutorial.

Start the server and test it

vllm serve Qwen/Qwen3-8B 
  --max-model-len 8192 
  --gpu-memory-utilization 0.90 
  --port 8000

Qwen3-8B is a good first target because it is ungated, downloads quickly, and fits a single 24 GB card at bf16. Then test:

curl http://localhost:8000/v1/models

curl http://localhost:8000/v1/chat/completions 
  -H "Content-Type: application/json" 
  -d '{"model": "Qwen/Qwen3-8B", "messages": [{"role": "user", "content": "Say hi"}]}'

Security note: vllm serve starts with no authentication. Pass --api-key and keep port 8000 behind a firewall or reverse proxy — an exposed vLLM endpoint is an open, uncapped inference bill on your own hardware.

Useful flags on first run: --tensor-parallel-size N to shard across N GPUs, --max-model-len to cap context (the single most effective lever against startup OOM), --quantization for pre-quantized checkpoints, --served-model-name to expose a shorter alias to clients, and --enforce-eager to skip CUDA graph capture when you want faster startup while debugging.

Will the model fit? Size it before you install

vLLM loads weights in bf16 by default, so budget roughly 2 GB of VRAM per billion parameters, plus KV cache — and vLLM preallocates that cache aggressively (90% of the card at the default --gpu-memory-utilization). An 8B model is about 16 GB of weights at bf16. Quantized to 4 bits, the same model drops to roughly 5 GB, per the Convly models database:

Model	Context	~VRAM at 4-bit	Realistic single-node setup
Qwen3 8B	128K	~5 GB	One 12–24 GB consumer card
Llama 3.1 8B	128K	~5 GB	One 12–24 GB consumer card
Gemma 3 27B	128K	~16 GB	One 24 GB card at 4-bit
Qwen3 32B	128K	~20 GB	One 24 GB card at 4-bit, tight on KV cache
Llama 3.3 70B	128K	~40 GB	2× 24 GB with --tensor-parallel-size 2, or one 48 GB card
DeepSeek R1	128K	~400 GB	Multi-GPU server, not a workstation

For a figure specific to your context length and batch size, use the VRAM calculator; the fuller per-model breakdown is in the VRAM requirements guide. If you are still choosing hardware, see best GPUs for local LLMs. And before you buy anything, it is worth running the numbers with the self-hosting vs API calculator — Llama 3.3 70B costs $0.10 in / $0.32 out per 1M tokens from a hosted provider, which is hard to beat on your own electricity below fairly high sustained utilization.

Common vLLM install and startup errors

Symptom	Cause and fix
ImportError on vllm._C, or an ABI/symbol error from torch	vLLM was installed over an incompatible PyTorch. Delete the environment, recreate it clean, install vLLM first.
“The model’s max seq len is larger than the maximum number of tokens that can be stored in KV cache”	Not enough free VRAM for the requested context. Lower --max-model-len, raise --gpu-memory-utilization, or use a quantized checkpoint.
CUDA out of memory while loading weights	The weights themselves do not fit. Shard with --tensor-parallel-size or pick a smaller model.
Found no NVIDIA driver	No GPU visible to the process. In WSL, the driver belongs on the Windows side; in Docker, you are missing --gpus all.
401/403 downloading a model	Gated repository. Accept the licence on Hugging Face, then authenticate (hf auth login in current versions of the Hugging Face CLI, huggingface-cli login in older ones) or set HF_TOKEN.
Long pause before the server accepts requests	Normal: CUDA graph capture and compilation. Use --enforce-eager to skip it during debugging.

Frequently asked questions

Can I install vLLM natively on Windows?

No. vLLM publishes Linux wheels only, and pip install vllm in Windows Python will not give you a working GPU server. Use WSL2 with an Ubuntu distribution, or run the official Docker image. Both are fully supported and give near-native performance on the same hardware.

Do I need to install the CUDA toolkit first?

Not for the default wheel. It bundles the CUDA runtime through its pinned PyTorch build, so a reasonably current NVIDIA driver is enough. You only need a full toolkit with nvcc if you compile vLLM from source or build custom kernels.

How do I install a specific vLLM version or the nightly build?

Pin it like any package: pip install vllm==<version>, choosing from the versions listed on PyPI. Nightly and per-commit wheels are published separately by the project and installed with an extra index URL — the current address is documented on the installation page, and it has changed before, so read it there rather than copying an old command.

Why does vLLM take up my whole GPU?

By design. It preallocates a large KV cache block pool at startup — controlled by --gpu-memory-utilization, which defaults to 0.9 — because paged attention is what makes its throughput high under concurrency. Lower the value if you need to share the card, and expect fewer concurrent requests as a result.

Should I use vLLM or Ollama?

Ollama is a single cross-platform installer aimed at one user on one machine; see the Ollama install guide if that describes you. vLLM is a serving engine built for many concurrent requests per GPU, with continuous batching, tensor parallelism and an OpenAI-compatible API. Install vLLM when you are serving an application, not when you are chatting locally.

Which model should I serve first?

Start with something small and ungated so you are debugging the install rather than the download — an 8B-class model at around 5 GB in 4-bit is ideal. Once the server responds to /v1/models, move up. The LLM leaderboard is a reasonable way to shortlist candidates by capability, price and context length before you commit VRAM to one.