Llama Cpp Python: Install, GPU Build, and Parameters

llama-cpp-python is the Python binding for llama.cpp. It loads GGUF models in-process, exposes a low-level ctypes wrapper plus a high-level Llama class, and ships an OpenAI-compatible HTTP server. The default pip install compiles a CPU-only build. To use a GPU you install a GPU wheel or rebuild from source with CMAKE_ARGS, then pass n_gpu_layers.

Why the default install is CPU-only

The package is a thin binding around a C++ library that has to be compiled with backend support baked in at build time. There is no runtime flag that turns CUDA on after the fact. When pip builds the sdist with no CMAKE_ARGS set, CMake configures the generic CPU backend and that is what you get, permanently, until you rebuild. On macOS arm64 this is less of a problem because recent releases enable the Metal backend by default, but on Linux and Windows a bare install will run entirely on your CPU.

Two consequences worth internalising: first, n_gpu_layers=-1 on a CPU-only build silently does nothing useful, so people often conclude their GPU is “too slow” when it was never touched. Second, pip caches built wheels. Re-running the install with different CMAKE_ARGS can hand you the cached CPU wheel again, which is why every rebuild command below includes --no-cache-dir --force-reinstall.

Installing llama-cpp-python with GPU support

Option 1: prebuilt wheels (no compiler needed)

The project publishes wheel indexes, including a CPU index at https://abetlen.github.io/llama-cpp-python/whl/cpu and CUDA variants whose path segment encodes the CUDA version, for example .../whl/cu124. Install with:

pip install llama-cpp-python --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cu124

Which CUDA tags and Python versions are published changes release to release, and the indexes sometimes lag the newest PyPI version. Check the project README for the tags that currently exist rather than assuming one — a wrong tag returns a 404 and pip quietly falls back to building from source.

Option 2: build from source (Linux, CUDA)

You need a C++ toolchain, CMake, and the CUDA toolkit with nvcc on your PATH.

nvcc --version   # must print a version, not "command not found"

CMAKE_ARGS="-DGGML_CUDA=on" 
  pip install llama-cpp-python --no-cache-dir --force-reinstall --upgrade

The flag name has changed over the project’s life: very old guides use -DLLAMA_CUBLAS=on, mid-2024 guides use -DLLAMA_CUDA=on, and current upstream uses the GGML_ prefix. If a build errors out on an unknown CMake option, that mismatch is usually why. Other backends follow the same pattern — Vulkan is -DGGML_VULKAN=on, SYCL is -DGGML_SYCL=on, and the AMD/ROCm option has been renamed more than once, so read the README for your installed version instead of copying a flag from a forum post.

You can cut compile time substantially by building for only your GPU’s compute capability, for example -DCMAKE_CUDA_ARCHITECTURES=89 for an Ada card such as the RTX 4090, or 86 for a 3090. Look your card’s compute capability up in NVIDIA’s official list; if you are still choosing hardware, our guide to the best GPUs for running LLMs locally covers the VRAM-per-dollar tradeoffs.

Option 3: macOS with Metal

xcode-select --install

CMAKE_ARGS="-DGGML_METAL=on" 
  pip install llama-cpp-python --no-cache-dir --force-reinstall

On Apple Silicon, verify you are not running a Rosetta x86 Python: python -c "import platform; print(platform.machine())" must print arm64. An x86_64 interpreter produces a build with no Metal backend no matter what CMAKE_ARGS you pass. Because the GPU and CPU share memory on Apple Silicon, n_gpu_layers=-1 is almost always the right setting there.

Option 4: Windows with CUDA

Install Visual Studio Build Tools 2022 with the “Desktop development with C++” workload first, then the CUDA Toolkit, so CUDA installs its MSBuild integration into an existing Visual Studio. Then, in PowerShell:

$env:CMAKE_ARGS = "-DGGML_CUDA=on"
pip install llama-cpp-python --no-cache-dir --force-reinstall --upgrade

In cmd.exe the equivalent is set CMAKE_ARGS=-DGGML_CUDA=on on its own line. Windows source builds are the most failure-prone path of the three platforms; if you only want inference and not a custom build, the prebuilt CUDA wheels or WSL2 are both less painful.

How to tell which build you have

The most reliable check is the verbose loader output, which is version-agnostic:

from llama_cpp import Llama
llm = Llama(model_path="./models/model.gguf", n_gpu_layers=-1, verbose=True)

On a CUDA build you will see backend initialisation lines mentioning CUDA and a device name, plus a tensor-loading line reporting how many of the model’s layers were offloaded to GPU. On Metal you will see Metal device lines instead. A CPU-only build prints neither and reports zero offloaded layers. Cross-check with nvidia-smi during generation: if your Python process is not holding VRAM, nothing is running on the GPU.

Recent versions also expose a direct capability check:

from llama_cpp import llama_cpp, __version__
print(__version__)
print(llama_cpp.llama_supports_gpu_offload())

If that attribute raises AttributeError, your build predates it — fall back to the verbose-log method.

Loading a GGUF and running a first completion

Point model_path at any GGUF file. If you would rather pull from Hugging Face, Llama.from_pretrained(repo_id=..., filename="*Q4_K_M.gguf", ...) does the download for you when huggingface-hub is installed.

from llama_cpp import Llama

llm = Llama(
    model_path="./models/qwen2.5-7b-instruct-q4_k_m.gguf",
    n_gpu_layers=-1,
    n_ctx=4096,
    n_batch=512,
    verbose=False,
)

out = llm.create_chat_completion(
    messages=[{"role": "user", "content": "Explain a KV cache in two sentences."}],
    max_tokens=256,
    temperature=0.7,
)
print(out["choices"][0]["message"]["content"])

For raw text continuation, call the object directly: llm("Q: What is a GGUF file? A:", max_tokens=128, stop=["Q:"]) and read out["choices"][0]["text"].

Streaming tokens

Pass stream=True and iterate. The response shape mirrors the OpenAI streaming format, so the first chunk typically carries only the role and later chunks carry content deltas:

stream = llm.create_chat_completion(
    messages=[{"role": "user", "content": "Write a haiku about GGUF files."}],
    stream=True,
)
for chunk in stream:
    delta = chunk["choices"][0]["delta"]
    if "content" in delta:
        print(delta["content"], end="", flush=True)

Most GGUF files embed a chat template that llama-cpp-python applies automatically. When output looks garbled or the model never stops, the template is the first suspect — override it with the chat_format argument. Browse quantised options and their sizes in our AI models database.

The parameters that matter

Note that flash attention and KV-cache quantization (type_k / type_v) have moved around between releases — the flash-attention switch has been a boolean in some versions and a three-way auto/on/off setting in others. Run help(Llama) against your installed version rather than trusting a flag name from a blog post.

Tuning in practice

The two settings that interact are n_gpu_layers and n_ctx. Weights and KV cache compete for the same VRAM, and the KV cache scales roughly linearly with context length. Halving n_ctx from 8192 to 4096 often frees enough memory to offload several more layers, which is usually the better trade. Work out the budget before you start guessing with our VRAM calculator, or check per-model figures in the VRAM requirements reference.

Partial offload works — it is llama.cpp’s signature feature — but expect a steep drop once any layer stays on CPU, because every token must cross the PCIe bus. If you can fit all layers, do.

OpenAI-compatible server mode

pip install "llama-cpp-python[server]"

python -m llama_cpp.server 
  --model ./models/qwen2.5-7b-instruct-q4_k_m.gguf 
  --n_gpu_layers -1 
  --n_ctx 4096 
  --host 0.0.0.0 --port 8000

Server flags mirror the constructor arguments, underscores included. You get /v1/chat/completions, /v1/completions, /v1/models, and interactive docs at /docs. Any OpenAI client works, and streaming is supported over SSE:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")
for event in client.chat.completions.create(
    model="gpt-3.5-turbo",  # ignored unless you set model aliases
    messages=[{"role": "user", "content": "Hello"}],
    stream=True,
):
    print(event.choices[0].delta.content or "", end="", flush=True)

For more than one model, pass --config_file config.json with a models array, giving each entry a model path, a model_alias clients can request by name, and its own n_gpu_layers / n_ctx. Add --api_key if the port is reachable beyond localhost. Weighing this against a hosted endpoint? The self-hosting vs API break-even calculator puts numbers on it.

Common build failures and fixes

Frequently asked questions

Is llama-cpp-python the same as llama.cpp?

No. llama.cpp is the C/C++ inference engine; llama-cpp-python vendors a specific commit of it and wraps it for Python. Because the vendored version is pinned per release, the bindings can trail upstream by days or weeks — which matters when a brand-new model architecture has just landed in llama.cpp but not yet in a published binding release.

How do I confirm the GPU is really being used?

Load with verbose=True and look for backend initialisation lines and a report of layers offloaded to GPU. Then watch nvidia-smi (or Activity Monitor’s GPU history on macOS) during a generation. If VRAM usage does not rise and tokens per second look like CPU numbers, you are on a CPU-only build.

Can I avoid compiling entirely?

Often, yes — use the project’s prebuilt wheel indexes with --extra-index-url, matching the CUDA tag to your toolkit. When no matching wheel exists for your Python version and platform, pip falls back to a source build, which typically takes several minutes with CUDA enabled.

Should I use llama-cpp-python, Ollama, or LM Studio?

Use llama-cpp-python when you want the model inside your own Python process, with direct control over sampling, logits, and grammars. Prefer Ollama for a managed daemon with model pulls and automatic memory handling, or LM Studio for a GUI. All three sit on llama.cpp, so quality is comparable; the difference is ergonomics.

Can I run a model larger than my VRAM?

Yes. Set n_gpu_layers to a value below the model’s layer count and the remainder runs on CPU with system RAM. It works reliably but the speed penalty is severe once a meaningful share of layers stays on CPU, so a smaller model at a higher quantization usually beats a large model half-offloaded.

Does GPU support work on Windows without WSL?

It does. Either install a prebuilt CUDA wheel, or build from source with Visual Studio Build Tools 2022 (Desktop development with C++) installed before the CUDA Toolkit, setting $env:CMAKE_ARGS in PowerShell. WSL2 remains the smoother path if you are comfortable with it, since the Linux build instructions are better trodden.