KoboldCpp: Complete Guide to the Single-Binary Local LLM Runtime

KoboldCpp is a single-file local LLM runtime built on top of llama.cpp. Download one binary, point it at a GGUF model, and you immediately get a browser-based chat UI and an OpenAI-compatible REST API — no package manager, no daemon to configure, no install step required. It runs on Windows, macOS, and Linux with optional GPU acceleration via CUDA, Metal, Vulkan, or OpenCL.

Downloading KoboldCpp

Releases are published on the KoboldCpp GitHub releases page. Each release ships platform-specific binaries; pick the one that matches your hardware.

Windows

Download koboldcpp.exe for NVIDIA GPU support (CUDA libraries are bundled — no separate CUDA toolkit install required, only the standard NVIDIA display driver). If you have no NVIDIA GPU, download koboldcpp_nocuda.exe instead. Double-clicking the .exe opens a graphical launcher where you can browse for your model file and configure settings before starting the server. To skip the launcher and start from the command line, pass the --skiplauncher flag.

macOS

Download the macOS binary from the releases page (typically named koboldcpp_mac or distributed as a .dmg). Metal GPU acceleration is included automatically — no extra flag is needed; KoboldCpp detects Apple Silicon and uses Metal by default. On first launch, macOS may warn that the binary is from an unidentified developer; right-click → Open to bypass Gatekeeper.

Linux

Download the Linux binary and make it executable:

chmod +x koboldcpp
./koboldcpp --model /path/to/model.gguf

Prebuilt Linux binaries include CPU and Vulkan support. For CUDA on NVIDIA cards, look for a release asset with a cu suffix in the filename, or compile from source with make LLAMA_CUDA=1. If your driver is too old for the bundled CUDA version, the Vulkan build is a reliable fallback.

Getting a GGUF Model

KoboldCpp loads GGUF files directly — the same format used by llama.cpp and Ollama. The primary source is Hugging Face; search for a model name plus “GGUF”. Before downloading, use the VRAM calculator to confirm the model will fit your GPU at your chosen context size. Quantization tiers to know:

Launching KoboldCpp

The minimal command on any platform:

./koboldcpp --model /path/to/model.gguf

This starts the server on http://localhost:5001. Open that URL in your browser to reach the web UI.

Windows — GUI Launcher

Double-click koboldcpp.exe. The launcher window lets you browse for a model file, set GPU layers, context size, and backend without touching the command line. Click Launch when done; a terminal window opens showing the server log and the browser UI launches automatically.

Command-Line (all platforms)

A typical launch command with GPU offload, custom context, and explicit port:

./koboldcpp 
  --model ./models/llama3-8b-q4_k_m.gguf 
  --gpulayers 32 
  --contextsize 8192 
  --port 5001

Key flags reference:

The Web UI and OpenAI-Compatible API

Once running, KoboldCpp exposes two interfaces from the same port:

• Browser UI — http://localhost:5001: A full-featured text generation interface with story, chat, and instruct modes. Supports prompt templates, memory, author’s notes, and world info fields inherited from the KoboldAI project.
• KoboldAI API — http://localhost:5001/api/v1: Used by frontends like SillyTavern and Agnaistic.
• OpenAI-compatible API — http://localhost:5001/v1: Implements /v1/chat/completions and /v1/completions. Any client that accepts a custom base URL works, including LangChain, the OpenAI Python SDK, and most open-source chat apps.

To point the OpenAI Python SDK at KoboldCpp:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:5001/v1",
    api_key="unused"  # KoboldCpp does not require an API key by default
)
response = client.chat.completions.create(
    model="koboldcpp",
    messages=[{"role": "user", "content": "Hello"}]
)

GPU Offload: Picking the Right –gpulayers Value

Each transformer layer offloaded to GPU moves processing off the CPU and dramatically increases tokens-per-second. The tradeoff is VRAM. A model’s total layer count is fixed by its architecture — common values are 32 for 7B/8B models, 40 for 13B models, and 80 for 70B models. Each layer consumes a roughly equal share of the model’s total GPU memory.

Practical approach:

1. Use the VRAM calculator to estimate how many layers fit at your target context size.
2. Start with --gpulayers 999 to attempt full offload. KoboldCpp clamps this to the model’s actual layer count automatically.
3. If you get an out-of-memory error on startup, reduce the value and retry. The server log prints per-layer allocation to help you calibrate.

Partial offload is supported and useful — even offloading half the layers of a large model gives a significant speed improvement over pure CPU inference. If you are unsure which GPU to pair with a model, see the best GPUs for local LLMs guide and the VRAM requirements breakdown by model.

Context Size and Performance Settings

Context size (--contextsize) is the single biggest driver of VRAM beyond model weights. A 7B model at Q4_K_M uses roughly 4 GB for weights; extending context from 4096 to 32768 tokens can add several more gigabytes to the KV cache. Enable --flashattention to reduce KV cache footprint — this is particularly effective at very long contexts and costs nothing in output quality.

Other settings that affect speed:

• --threads: for CPU-only inference, set this close to your physical core count, not your logical (hyperthreaded) count.
• --batchsize: larger values (e.g., 512) improve prompt-processing speed at the cost of peak VRAM during prefill.
• --smartcontext: when the context fills up, KoboldCpp shifts the oldest tokens out instead of stopping generation — useful for long interactive sessions.

KoboldCpp vs Ollama vs llama.cpp

All three are built on the same llama.cpp engine and support GGUF models. The differences are in workflow and interface.

Choose KoboldCpp if you want zero-install setup, the built-in story/chat UI, or compatibility with KoboldAI frontends like SillyTavern.
Choose Ollama if you want a managed model library, a systemd service, or tighter CLI integration — see the Ollama complete guide for a full walkthrough.
Choose llama.cpp directly if you are building a custom integration or need the absolute latest upstream features before they reach downstream wrappers.

If you are still deciding whether to self-host at all versus calling a hosted API, the self-hosting vs API break-even calculator can help you model the cost crossover point.

Frequently Asked Questions

Does KoboldCpp require installing CUDA drivers separately?

On Windows, koboldcpp.exe bundles the CUDA runtime libraries, so you only need the standard NVIDIA display driver — no separate CUDA toolkit installation. On Linux, CUDA builds typically link against the installed CUDA runtime, so driver version compatibility matters; if your driver is too old, the Vulkan build is the easiest fallback.

What does –gpulayers 0 mean?

Zero GPU layers means all computation runs on CPU. This is the default when no GPU flag is set. CPU inference is much slower — typically 2–10 tokens/second on a modern CPU versus 40–100+ tokens/second on a mid-range GPU — but works on any machine regardless of GPU availability.

Can I use KoboldCpp as an OpenAI API drop-in for my application?

Yes. Set your OpenAI client’s base_url to http://localhost:5001/v1 and any non-empty string as the api_key (it is not validated by default). The model field is accepted but ignored — whichever GGUF is loaded is always used. Chat completions and text completions both work; embeddings and image endpoints are not supported.

How do I run two different models at the same time?

Each KoboldCpp process handles one model. Launch a second instance with a different --port value (e.g., 5002) pointing at a different GGUF file. There is no built-in load balancer; route between instances at the application layer.

Why is generation slower than expected even with a GPU?

The most common cause is partial CPU offload: if --gpulayers is lower than the model’s total layer count, the remaining layers run on CPU and create a bottleneck. Check the startup log — KoboldCpp prints exactly how many layers went to GPU versus CPU. Also confirm that the correct backend (CUDA/Metal/Vulkan) appears in the startup output rather than a CPU fallback.

Is it safe to expose KoboldCpp on a network?

By default KoboldCpp binds to 127.0.0.1 (localhost only). To expose it on a LAN, add --host 0.0.0.0. There is no built-in authentication, so exposing it to untrusted networks or the public internet is not recommended without a reverse proxy with authentication in front of it.