Ollama Not Using GPU: Diagnose and Fix CPU Fallback

When Ollama is not using your GPU, the most common cause is a missing or mismatched driver. Run ollama ps — if the PROCESSOR column shows 100% CPU, Ollama has fallen back entirely. The fix depends on your platform: NVIDIA needs the correct CUDA runtime, AMD needs ROCm, and Docker needs explicit GPU passthrough flags.

Step 1: Confirm Whether Ollama Is Using the GPU

Before changing anything, verify what Ollama is actually doing. Load a model and, while it is running, open a second terminal:

ollama ps

Sample output:

NAME              ID              SIZE      PROCESSOR    UNTIL
llama3.2:8b       abc123def456    5.0 GB    100% GPU     4 minutes from now

The PROCESSOR column is the definitive signal:

On NVIDIA systems, cross-check with nvidia-smi while inference is running. The Memory-Usage column should increase as the model loads. If it stays flat, the GPU is idle regardless of what other tools report.

Step 2: NVIDIA — Drivers and CUDA Runtime

A missing or outdated NVIDIA driver is the single most common cause of full CPU fallback. Ollama bundles its own CUDA libraries, but it still requires a host driver that supports CUDA 11.3 or later.

Check the driver first

nvidia-smi

If the command is not found, no driver is installed. If it runs, note the Driver Version and CUDA Version in the header. The CUDA Version shown there is the maximum the driver supports — it does not mean a separate CUDA toolkit is installed, and Ollama does not need one.

Install or update the driver

Ubuntu / Debian:

sudo apt install nvidia-driver-550
sudo reboot

Windows: Download from nvidia.com/Download or use GeForce Experience, then reboot. A full reboot — not just a service restart — is required after a driver install or update.

After the reboot, confirm nvidia-smi shows your card, then restart Ollama:

# Linux (systemd)
sudo systemctl restart ollama

# macOS
launchctl kickstart -k gui/$(id -u)/com.ollama.ollama

# Windows — restart the Ollama tray app, or reboot

Run a model and check ollama ps again. If PROCESSOR still shows 100% CPU, continue to the next steps.

Step 3: Model Too Large for VRAM

When a model’s weights exceed available VRAM, Ollama does not refuse to run — it splits inference. As many transformer layers as possible go to the GPU; the rest run on CPU. This appears as a split percentage in ollama ps and is intentional behavior, not a bug.

A 70B parameter model at Q4_K_M quantization typically needs around 40 GB of VRAM, which exceeds any single consumer GPU. Before downloading a large model, check whether it fits using the VRAM calculator. For per-model figures across the most popular LLMs, see VRAM requirements for every major LLM.

If the model does not fit your VRAM, your options are:

• Use a lower quantization (e.g., Q2_K or Q3_K_S) — reduces VRAM usage at a small quality cost.
• Switch to a smaller model variant (e.g., 8B instead of 70B). The best local models for Ollama page covers which models run well on consumer hardware.
• Accept the partial offload — throughput will be between full-GPU and full-CPU speeds.
• Upgrade your GPU. The best GPUs for local LLMs guide compares current options by VRAM and price.

Step 4: AMD GPU and ROCm

AMD support in Ollama is built on ROCm, AMD’s GPU compute platform. Ollama officially supports RDNA 2 (RX 6000 series) and RDNA 3 (RX 7000 series) cards on Linux. Windows AMD support is limited — check the release notes for your installed Ollama version before expecting it to work on Windows.

Verify ROCm sees the card

rocm-smi

If rocm-smi is not found, the ROCm stack is not installed. Check amd.com/en/developer/rocm for the current installation instructions for your distribution, as package names and version requirements change between ROCm releases.

If the card is absent from rocm-smi output after installation:

sudo usermod -aG render,video $USER
# Log out and back in for the group change to take effect

To target a specific GPU when multiple are present:

HIP_VISIBLE_DEVICES=0 ollama serve

Step 5: Docker — Missing GPU Passthrough

Docker containers have no GPU access unless you explicitly pass the device through. This is the most common reason Ollama falls back to CPU in containerized deployments — the host GPU is fine, but the container cannot see it.

NVIDIA in Docker

Install the NVIDIA Container Toolkit on the host:

sudo apt install nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Then pass the GPU when starting the container:

docker run -d --gpus all 
  -v ollama:/root/.ollama 
  -p 11434:11434 
  ollama/ollama

Without --gpus all (or --gpus device=0 for a specific card), the container has no GPU access regardless of the host configuration.

AMD in Docker

docker run -d 
  --device /dev/kfd --device /dev/dri 
  -v ollama:/root/.ollama 
  -p 11434:11434 
  ollama/ollama:rocm

The :rocm image tag is required. The default ollama/ollama image does not include ROCm and will fall back to CPU on AMD hardware.

Step 6: WSL2 on Windows

WSL2 can share the NVIDIA GPU with the Windows host, but one rule is absolute: the NVIDIA driver must be installed on Windows, not inside WSL2. Installing a Linux NVIDIA driver inside the WSL2 environment breaks GPU passthrough entirely.

• Install or update the NVIDIA driver on Windows, then reboot Windows.
• WSL2 requires kernel 5.10.43 or later. Check with uname -r inside WSL2; update with wsl --update in a Windows terminal.
• The CUDA toolkit can be installed inside WSL2 if your workflow needs it — that is separate from the driver and does not conflict.

Verify GPU visibility from inside WSL2:

nvidia-smi

If this shows your card, Ollama running inside WSL2 will use it automatically. If it fails, update the Windows-side driver and re-run wsl --update.

macOS — Metal (Apple Silicon and AMD)

On macOS, Ollama uses Apple Metal for GPU acceleration — no drivers to install and no environment variables to set. If you are on Apple Silicon and Ollama is running slowly, confirm you installed the native ARM build from ollama.com rather than the x86 version running under Rosetta. Apple Silicon Macs use unified memory, so the full system RAM is available to models — enter your total RAM as the VRAM limit when using the VRAM calculator.

Verifying the Fix and Expected Throughput

After making changes, run a model and check two signals at the same time:

# Terminal 1 — start a generation
ollama run llama3.2:8b "What is the capital of France?"

# Terminal 2 — while it is generating
ollama ps

# NVIDIA: also watch GPU utilization per second
nvidia-smi dmon -s u

Reference throughput (approximate — varies by quantization, driver version, and PCIe bandwidth):

Single-digit tokens per second with a capable GPU present is the clearest sign the fix has not taken effect.

Frequently Asked Questions

Why does ollama ps show a GPU/CPU split instead of 100% GPU?

The model is larger than your available VRAM. Ollama places as many layers as possible on the GPU and runs the remainder on CPU. The result is faster than full-CPU but slower than full-GPU. Load a smaller model or switch to a lower quantization to push more layers onto the GPU.

Ollama was using the GPU before and suddenly stopped — what changed?

A driver update, OS update, or new Ollama version can break GPU detection. Verify nvidia-smi still shows the card, then do a full service restart. If you updated the NVIDIA driver recently, a complete system reboot is sometimes necessary rather than just restarting the service.

Can Ollama use multiple GPUs at once?

Yes. Ollama distributes model layers across all visible GPUs automatically when more than one is present. To restrict which GPUs Ollama uses, set CUDA_VISIBLE_DEVICES (NVIDIA) or HIP_VISIBLE_DEVICES (AMD) before starting ollama serve.

Does Ollama work with consumer GeForce cards, or do I need a data-center GPU?

GeForce GTX 10xx and newer are fully supported — no data-center GPU required. The practical limit for most users is VRAM: an 8 GB card comfortably runs 7–8B parameter models at Q4 quantization. Use the VRAM calculator to verify a specific model fits before downloading it.

My GPU has enough VRAM but Ollama still uses CPU. Why?

Another process may be occupying VRAM — check nvidia-smi for other consumers such as a browser with hardware acceleration or another running model. The model may also have been loaded before the GPU driver was ready. Stop the loaded model with ollama stop <name>, free VRAM in other applications, and reload.

Where can I learn more about configuring Ollama and choosing models?

The Ollama complete guide covers environment variables, model management, and API usage in depth. For choosing which model to run on your specific hardware, see the best local models for Ollama.