Nano-vLLM: A Minimal vLLM Implementation for Local Inference

Nano-vLLM is an open-source, from-scratch reimplementation of the vLLM inference server, written in roughly 1,200 lines of Python. It was released in mid-2025 by Xingkai Yu, a DeepSeek engineer, under the MIT license at github.com/GeeeekExplorer/nano-vllm. It is a teaching-grade codebase and a fast batch inference tool — not a drop-in replacement for the full vLLM server.

What Nano-vLLM Actually Is

The upstream vllm-project/vllm repository is a large, production-grade inference server with hundreds of contributors, an OpenAI-compatible HTTP API, distributed workers, and support for dozens of model architectures and quantization schemes. Nano-vLLM strips that back to the core loop: model loading, a KV cache manager, a batching scheduler, and a sampler.

According to the project’s README, nano-vllm keeps the key optimizations that make vLLM fast:

• Prefix caching — reuses KV cache blocks across requests that share a prompt prefix.
• Tensor parallelism — splits a model across multiple GPUs on one node.
• Torch compilation — uses torch.compile for kernel fusion.
• CUDA graphs — reduces per-step launch overhead during decoding.

What it deliberately omits: the OpenAI-compatible HTTP server, continuous streaming APIs, most quantization backends (AWQ, GPTQ, FP8), speculative decoding, LoRA hot-swapping, multi-node clustering, and the broad model zoo. It ships primarily tested against Qwen3-class dense models.

Installing Nano-vLLM

Nano-vLLM is a Python package. There is no Windows-native CUDA support path in the upstream repo; on Windows use WSL2 with an NVIDIA driver. Linux and WSL2 are the primary platforms. macOS is not supported because the code paths assume CUDA.

Linux and WSL2

python -m venv .venv
source .venv/bin/activate
pip install torch --index-url https://download.pytorch.org/whl/cu121
pip install git+https://github.com/GeeeekExplorer/nano-vllm.git

Match the CUDA wheel (cu121, cu124, etc.) to your installed NVIDIA driver. Verify with nvidia-smi before installing.

Windows (via WSL2)

Install the NVIDIA Windows driver, enable WSL2 with an Ubuntu distribution (wsl --install -d Ubuntu), then follow the Linux steps inside WSL. Do not install CUDA toolkit inside WSL — the Windows driver exposes the GPU.

macOS

Not supported. Nano-vLLM depends on CUDA kernels and paged-attention primitives that have no Metal backend. On Apple Silicon use Ollama or LM Studio instead, both of which run llama.cpp under the hood.

Downloading a Model

Nano-vLLM loads standard Hugging Face model directories — the same config.json, tokenizer.json and safetensors layout that transformers and vLLM use. Fetch a model with the official huggingface_hub CLI:

pip install -U "huggingface_hub[cli]"
hf download Qwen/Qwen3-8B --local-dir ~/models/Qwen3-8B

See the Hugging Face Hub CLI docs for authentication and gated models. Note that the older huggingface-cli entry point still ships with the package, but Hugging Face now recommends the hf command.

Running Inference

The API mirrors vLLM’s offline batch interface closely. A minimal script:

from nanovllm import LLM, SamplingParams

llm = LLM("/home/user/models/Qwen3-8B", enforce_eager=False, tensor_parallel_size=1)
sp = SamplingParams(temperature=0.7, max_tokens=256)

prompts = ["Explain paged attention in one paragraph."]
outputs = llm.generate(prompts, sp)
print(outputs[0]["text"])

The exact class names and return shape may drift across commits — check example.py in the repo root, which is the canonical usage reference.

Hardware Requirements by Model

Because nano-vllm currently runs models in bf16/fp16 (no built-in 4-bit quantization at the time of writing), VRAM requirements are roughly double the 4-bit figures below. Use the VRAM calculator for a per-precision estimate. The following are 4-bit reference numbers from the Convly models database — for nano-vllm in bf16, budget approximately 2x these values plus KV cache headroom.

Model	Context	VRAM (4-bit ref.)	Realistic nano-vllm GPU
Qwen3 8B	128K	~5 GB	Single RTX 4090 (24 GB) at bf16
Qwen3 14B	128K	~9 GB	Single RTX 4090 at bf16 with modest context
Qwen3 32B	128K	~20 GB	2× RTX 4090 with tensor_parallel_size=2
Llama 3.1 8B	128K	~5 GB	Single RTX 4090
Llama 3.3 70B	128K	~40 GB	2× A100 80GB or 4× RTX 4090

For a broader picture of what fits on which card, see best GPUs for local LLMs and the VRAM requirements table.

Nano-vLLM vs vLLM vs Ollama

Feature	Nano-vLLM	vLLM	Ollama
Line count	~1,200 Python	~100k+ Python/C++/CUDA	Go wrapper over llama.cpp
OpenAI-compatible server	No	Yes	Yes (via /v1)
Backend	PyTorch + CUDA	PyTorch + custom kernels	llama.cpp (GGUF)
Quantization	Minimal	AWQ, GPTQ, FP8, INT4	Q2–Q8 GGUF
Multi-GPU	Tensor parallel	Tensor + pipeline + expert	Limited
Primary use	Learning, embedding	Production serving	Desktop / dev

If your goal is to serve a customer-facing endpoint, use full vLLM. If you want to embed a batch inference loop inside a larger Python program with minimal dependencies, nano-vllm is reasonable. If you want a one-command local chatbot, use Ollama.

When Nano-vLLM Makes Sense

• Learning the internals. The scheduler and block manager fit on one screen. Reading nano-vllm is the fastest way to understand paged attention in real code.
• Research forks. Modifying a 1,200-line codebase to test a new sampler or cache policy is tractable; forking upstream vLLM is not.
• Batch offline inference. Grading, synthetic data generation, evaluation loops over a fixed prompt set.

When it does not make sense: production APIs, multi-tenant serving, tight quantization budgets, or any non-NVIDIA hardware.

Self-Hosting vs API

Running nano-vllm locally has real costs — GPU capital, electricity, and engineering time. Frontier hosted models are often cheaper per token than amortized local inference at low volume. Compare with the self-hosting vs API calculator and the API cost calculator. For reference, Qwen3 8B on a hosted endpoint runs at $0.04 in / $0.14 out per 1M tokens per the Convly models database, while a 24 GB GPU capable of running it locally costs well over $1,500.

Frequently Asked Questions

Who wrote nano-vllm?

The repository is maintained by Xingkai Yu (GitHub handle GeeeekExplorer), an engineer at DeepSeek. It is a personal project, not an official DeepSeek release. The code is MIT-licensed and lives at github.com/GeeeekExplorer/nano-vllm.

Is nano-vllm faster than vLLM?

The README reports throughput close to vLLM on small dense models like Qwen3-0.6B on a single RTX 4070-class GPU, and in some short-benchmark configurations slightly faster because there is less scheduler overhead. On larger models, longer contexts, or multi-request serving, upstream vLLM’s optimizations pull ahead. Treat parity as “in the same ballpark for offline batch,” not “a strict replacement.”

Can nano-vllm serve an OpenAI-compatible API?

Not out of the box. The project exposes a Python LLM.generate() method for offline batch use. If you need an HTTP server with /v1/chat/completions, wrap it yourself with FastAPI, or use full vLLM’s OpenAI-compatible server or Ollama.

Does nano-vllm support quantized models like GGUF or AWQ?

No. The codebase loads standard Hugging Face safetensors weights in bf16/fp16. For GGUF (Q4_K_M etc.) use llama.cpp-based tools; for AWQ or GPTQ use upstream vLLM. This is one reason nano-vllm’s VRAM footprint is higher per parameter than Ollama’s for the same model.

Which models are known to work?

Qwen3 dense models are the primary target of the reference implementation. Other Llama-architecture models often work with minor adjustments to the model loader, but exotic architectures (mixture-of-experts, hybrid state-space) generally do not. Check the repo’s nanovllm/models/ directory for the current supported list.

Can I run nano-vllm on AMD or Apple Silicon?

Not currently. The kernels assume CUDA. ROCm may work with a custom PyTorch build but is untested upstream. On Apple Silicon there is no path — use MLX-based tools or llama.cpp. For a survey of alternatives, see the LLM leaderboard and pick a model that matches your hardware profile.