Running Ollama in Docker is the cleanest way to keep a local model server reproducible: the same command works on a laptop, a home server and a rented GPU box, and nothing leaks into the host system. The setup is short, but two details — GPU passthrough and volume persistence — cause most of the wasted time.
Quick answer
Pull and run the official image with a named volume so models survive restarts: docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama. For an NVIDIA GPU, install the NVIDIA Container Toolkit on the host and add --gpus=all. Then pull models inside the container with docker exec -it ollama ollama pull llama3.1. The API is available on port 11434 exactly as it is in a native install.
The three commands that matter
| Goal | Command |
|---|---|
| CPU only | docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama |
| With NVIDIA GPU | docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama |
| Pull a model | docker exec -it ollama ollama pull llama3.1 |
GPU passthrough: the part that fails
A container cannot see your GPU by default. On Linux you install the NVIDIA Container Toolkit on the host, restart Docker, and add --gpus=all. Verify it worked from inside the container with docker exec -it ollama nvidia-smi — if that command fails, Ollama will run, but silently on the CPU, and you will conclude your GPU is slow when it simply is not being used. On Windows the path is Docker Desktop with the WSL2 backend and current NVIDIA drivers. AMD users need the ROCm-tagged image rather than the default one, and Apple Silicon GPUs cannot be passed into Docker at all — on a Mac, run Ollama natively.
Keep your models between restarts
Model files are large and slow to re-download, and a container without a volume discards them the moment it is removed. The -v ollama:/root/.ollama flag in every command above creates a named volume that persists across restarts, upgrades and image changes. If you would rather keep the files somewhere you can see, bind-mount a host directory instead: -v /srv/ollama:/root/.ollama.
docker compose, for a setup you keep
For anything permanent, a compose file beats a long run command: it records the GPU reservation, the volume and the port in one place, restarts the service automatically, and lets you add a web UI alongside it later. The container exposes the same OpenAI-compatible endpoint on http://localhost:11434, so applications cannot tell the difference between the containerised and native installs.
Convly’s take
Containerise Ollama when the machine is a server, and skip Docker when the machine is your laptop. On a home server or a rented GPU box, the volume-plus-compose setup is genuinely better: reproducible, upgradeable and easy to move. On a personal machine — especially a Mac, where GPU passthrough is impossible — Docker adds a layer that costs performance and gains you little. The most common self-inflicted problem we see is a container quietly running on CPU because the toolkit was never installed; run nvidia-smi inside the container before you benchmark anything.
Frequently asked questions
Does Ollama in Docker support GPUs?
Yes on NVIDIA with the NVIDIA Container Toolkit and --gpus=all, and on AMD with the ROCm image. Apple Silicon GPUs cannot be exposed to Docker — run Ollama natively on macOS.
Where are models stored in the Docker setup?
Inside the container at /root/.ollama. Map that path to a named volume or host directory, or the models disappear when the container is removed.
Is Ollama slower in Docker?
Negligibly on Linux with proper GPU passthrough. Perceived slowness almost always means the container is on CPU because the GPU was not passed through correctly.
Can I run multiple models in one container?
Yes — pull as many as you like; Ollama loads them on demand. The limit is memory: only the models actively loaded consume RAM or VRAM.
New to the tool? Start with the Ollama install guide, check hardware needs in Ollama system requirements, or size a model with the VRAM calculator.
