What Is GGUF? The LLM File Format Behind Ollama and llama.cpp, Explained

GGUF (usually expanded as GPT-Generated Unified Format) is a binary file format for storing large language models: the weights, the tokenizer, and all configuration metadata in one self-contained file. It is the native format of llama.cpp, and it is what Ollama, LM Studio, KoboldCpp, Jan and most other local-LLM applications actually run. If you have ever downloaded a file ending in .gguf, or pulled a model with ollama run, you were using this format.

What a GGUF file actually contains

A GGUF file starts with the four ASCII bytes GGUF, followed by a version number, a key-value metadata section, and then the tensors themselves. The metadata is the important design decision: it stores the model architecture, context length, tokenizer vocabulary, chat template and quantisation details as named keys, so a runtime can load any GGUF file without needing separate config.json or tokenizer files alongside it.

Three practical consequences follow from this design:

• One file is the whole model. You can copy a .gguf file between machines and it just works. (Very large models are sometimes split into numbered parts like -00001-of-00002.gguf, but that is still one logical model.)
• It is memory-mappable. Runtimes can mmap the file and page weights in on demand, which makes loading fast and lets a model larger than free RAM start at all.
• Quantisation is built into the format. A GGUF file stores weights already compressed to 2–8 bits per weight, which is why an 8-billion-parameter model can be a 4.9 GB file instead of a 16 GB one.

Why GGUF replaced GGML

Before August 2023, llama.cpp used a format called GGML (named after the tensor library underneath it). GGML worked, but it had a structural flaw: the file layout was rigid and unversioned. Every time llama.cpp added a feature — new architectures, better quantisation methods, rope scaling parameters — the format had to change, and every existing model file on every hard drive broke. Users had to re-download or re-convert their entire model collections several times in a few months.

GGUF, introduced by the llama.cpp project in August 2023, fixed this with two changes:

• Versioning. The file declares its format version, so readers know exactly how to parse it.
• Extensible key-value metadata. New information (a new architecture field, a chat template, an extra hyperparameter) is just a new key. Old files without that key still load; new files with it work in updated runtimes. Nothing breaks retroactively.

llama.cpp dropped GGML support shortly after, and the ecosystem followed. Today GGML survives only as the name of the underlying tensor library; if you find a .ggml or .bin model file from 2023, it will not load in any current tool and you should download a GGUF version instead.

How to read the quantisation suffixes

Every GGUF filename carries a suffix like Q4_K_M that describes how aggressively the weights were compressed. The pattern:

• The number after Q is the approximate bits per weight: Q4 ≈ 4-bit, Q8 ≈ 8-bit. Fewer bits means a smaller file and lower quality.
• _K marks the newer “k-quant” methods, which spend extra bits on the tensors that matter most for output quality. At the same size, a K quant beats the older style.
• _S / _M / _L (small/medium/large) are variants within a K family — _M keeps a few more sensitive tensors at higher precision than _S.
• _0 (as in Q8_0, Q4_0) marks the older, simpler block quantisation. Q8_0 is still widely used because at 8 bits the simple method is already near-lossless; Q4_0 is mostly obsolete — prefer Q4_K_M.
• IQ prefixes (IQ2_M, IQ3_XS…) are “i-quants” built with an importance matrix, used to squeeze models below 4 bits with less damage than the alternatives.
• F16 / BF16 / F32 means unquantised 16- or 32-bit weights — the reference quality, at maximum size.

Here is what the common options cost in practice, using Llama-class 8B instruct models as the example (sizes are approximate and vary slightly by model):

Two useful rules: quantisation hurts small models more than large ones (a 70B at Q3 holds up far better than an 8B at Q3), and below Q4 the quality curve drops steeply. When in doubt, Q4_K_M is the community default for a reason.

Picking a quantisation for your VRAM

Total memory needed is roughly file size + context cache + overhead. Budget 1–2 GB on top of the file size for a few thousand tokens of context, more if you run long contexts. The model runs fastest when all of that fits in GPU VRAM; llama.cpp-based tools can offload the remainder to system RAM, which still works but gets much slower the more layers spill over.

For exact numbers on a specific model and context length, the VRAM calculator does the arithmetic for you, and there is a per-model breakdown in VRAM requirements for every major LLM. If you are choosing hardware rather than a quant, start with the best GPUs for local LLMs.

GGUF vs safetensors

These two formats coexist because they serve different runtimes, not because one is winning.

The decision is really about which software you run: desktop tools and Ollama want GGUF; GPU serving stacks and anything involving fine-tuning want safetensors. Model publishers typically release safetensors first, and the community converts to GGUF within days.

Where GGUF files come from

Almost all GGUF files live on Hugging Face. Labs occasionally publish official GGUFs, but most come from community quantisers — accounts like bartowski, unsloth, lmstudio-community and ggml-org — who convert each new release into the full range of quantisations. A repo named something like Meta-Llama-3.1-8B-Instruct-GGUF will contain one file per quant level.

You can also convert a model yourself with llama.cpp’s tooling: convert_hf_to_gguf.py turns a Hugging Face model into an F16 GGUF, and the llama-quantize binary (built when you compile llama.cpp) compresses it to your chosen quant, e.g. llama-quantize model-f16.gguf model-Q4_K_M.gguf Q4_K_M.

Loading a GGUF in Ollama

Ollama can pull GGUF repos directly from Hugging Face, quant specified after the colon:

ollama run hf.co/bartowski/Meta-Llama-3.1-8B-Instruct-GGUF:Q4_K_M

For a file you already have on disk, create a plain-text file named Modelfile containing:

FROM ./my-model.gguf

then register and run it:

ollama create my-model -f Modelfile
ollama run my-model

Current Ollama versions read the chat template embedded in the GGUF metadata; if an imported model produces garbled output, the fix is usually adding an explicit TEMPLATE line to the Modelfile. The Ollama complete guide covers Modelfiles in depth.

Loading a GGUF in LM Studio

The normal path is the built-in downloader: open the Discover tab, search for a model, and LM Studio lists the available GGUF quants, flagging which ones fit your hardware. To import a file you already have, use the CLI (lms import path/to/model.gguf) or place it in LM Studio’s models directory — the exact default path varies by version and OS, but the My Models tab shows and lets you change it. Files must sit in a publisher/model-name/ subfolder structure to be recognised. See the LM Studio complete guide for the full walkthrough.

Frequently asked questions

Is GGUF CPU-only, or does it use the GPU?

Both. llama.cpp was born as a CPU inference project, but it offloads model layers to the GPU (CUDA, Metal, Vulkan, ROCm) and runs fully on GPU when the model fits in VRAM. Ollama and LM Studio handle the CPU/GPU split automatically; with llama.cpp directly you control it with the -ngl (number of GPU layers) flag.

What is the difference between Q4_K_M and Q4_K_S?

Both are ~4-bit k-quants; the letter is the size variant. _M (medium) keeps more of the quality-sensitive tensors at higher precision than _S (small), so it is slightly larger and slightly better. The difference is small — pick _M unless the last few hundred megabytes decide whether the model fits.

How much quality do I actually lose to quantisation?

At Q8_0 and Q6_K, effectively none — blind tests struggle to separate them from F16. Q4_K_M shows a small measurable loss that most users never notice in chat, though it can matter for precise tasks like code generation or maths. Below Q4 the degradation becomes obvious, and small models suffer more than large ones.

Can I use GGUF with transformers or vLLM?

Support exists but is limited and not the native path — transformers can dequantise some GGUF files on load, and vLLM has experimental GGUF support that varies by architecture. If you are building on those stacks, use safetensors; GGUF is best treated as the format for the llama.cpp family of runtimes.

Why is my model split into multiple .gguf files?

Very large models are sharded into parts named like model-00001-of-00002.gguf, mainly to stay under file-size limits on hosting platforms. Keep all parts in the same folder and point your runtime at the first one — llama.cpp and the tools built on it load the rest automatically.

Once you know which quant fits your hardware, the practical next question is which model to put in it — the models database lists specs, VRAM needs and pricing for 37 current models, and the best local models for Ollama is a good shortlist to start from.