- GGUF is a single-file container format for quantized models, created for llama.cpp. One
.gguffile holds the weights, the tokenizer, the chat template and the metadata — no config folder, no separate tokenizer files. - GGUF models are what Ollama, LM Studio, KoboldCpp, Jan and llama.cpp actually load. If you run a model locally on consumer hardware, you are almost certainly running GGUF.
- Default choice:
Q4_K_M. Roughly 0.6 GB of file per billion parameters — an 8B model lands near 5 GB, which matches the ~5 GB 4-bit figure the Convly models database lists for Llama 3.1 8B. - GGUF is for single-user local inference. For concurrent serving use safetensors with vLLM or an API instead.
GGUF (GPT-Generated Unified Format) is the file format the llama.cpp/ggml ecosystem uses to store a model ready for inference. A single .gguf file contains the quantized tensors plus everything needed to use them: vocabulary, tokenizer settings, the chat template, and architecture metadata. It replaced the older GGML format in August 2023 and is now the de facto standard for running models on laptops, desktops and CPUs.
What is actually inside a .gguf file
A GGUF file is a header, a key-value metadata block, then the tensor data. The metadata is the part that matters practically — it is why a GGUF model needs no sidecar files. Hugging Face’s GGUF documentation describes the layout and the Hub’s built-in GGUF metadata viewer, which lets you inspect the quantization type, context length and chat template of a file before downloading several gigabytes of it.
Typical metadata keys include the architecture (llama, qwen3, gemma3, phi3), the training context length, RoPE settings, the full vocabulary, and the Jinja chat template. A loader reads that block and configures itself; you do not pass a tokenizer or a prompt format by hand.
GGUF vs safetensors
| GGUF | safetensors | |
|---|---|---|
| Primary runtime | llama.cpp, Ollama, LM Studio | PyTorch, vLLM, Transformers, TGI |
| Files per model | One (or numbered shards) | Weights plus config/tokenizer folder |
| Quantization | Baked in (Q4_K_M, Q8_0, IQ…) | Usually FP16/BF16, or GPTQ/AWQ variants |
| CPU + partial GPU offload | Yes, core design goal | Limited and slow |
| Concurrent batched serving | Weak | Strong |
| Fine-tuning | No (convert back first) | Yes |
How to read a GGUF filename
Files are usually named Model-Name-8B-Instruct-Q4_K_M.gguf. The suffix is the quantization mix. The number is the nominal bit width; _K means k-quants, which keep attention and embedding tensors at higher precision than the bulk feed-forward weights; S/M/L are small/medium/large variants of that mix.
| Quant | Approx. bits/weight | Approx. size, 8B model | When to use it |
|---|---|---|---|
| Q8_0 | ~8.5 | ~8.5 GB | Near-lossless reference; small models only |
| Q6_K | ~6.6 | ~6.6 GB | You have VRAM to spare |
| Q5_K_M | ~5.7 | ~5.7 GB | Quality-leaning default |
| Q4_K_M | ~4.9 | ~4.9 GB | The usual default |
| Q4_0 | ~4.5 | ~4.5 GB | Legacy; some accelerators prefer it |
| Q3_K_M | ~3.9 | ~4.0 GB | Squeezing a bigger model into small VRAM |
| Q2_K / IQ2 | ~2.5–3.4 | ~2.5–3.4 GB | Last resort; noticeable degradation |
Treat the bits-per-weight column as approximate. Actual sizes shift with model architecture (a large vocabulary inflates the embedding tensors) and with the llama.cpp version, because the quantization mixes are tuned over time. The IQ family (IQ2_XXS through IQ4_NL) uses importance-matrix calibration to hold up better at very low bit widths, and quantizers such as Bartowski and Unsloth publish IQ variants alongside the standard K-quants.
Community consensus, not a Convly measurement, is that Q4_K_M is the sweet spot and that quality falls off sharply below roughly 3 bits per weight — small models degrade faster than large ones at the same quant.
Sizing: which GGUF models fit your GPU
The file size is the floor, not the total. Add the KV cache for your context length plus roughly 500 MB–1 GB of overhead. These 4-bit figures come from the Convly models database:
| Model | VRAM at 4-bit | Context | Realistic GPU |
|---|---|---|---|
| Gemma 3 4B | ~3 GB | 128K | Any 6 GB card, integrated GPU |
| Mistral 7B | ~4.5 GB | 32K | 8 GB card |
| Llama 3.1 8B | ~5 GB | 128K | 8 GB card |
| Qwen3 14B | ~9 GB | 128K | 12 GB card |
| Phi-4 | ~9 GB | 16K | 12 GB card |
| Gemma 3 27B | ~16 GB | 128K | 24 GB card |
| Qwen3 32B | ~20 GB | 128K | 24 GB card |
| Llama 3.3 70B | ~40 GB | 128K | 2×24 GB, or 48 GB unified memory |
| DeepSeek R1 (full) | ~400 GB | 128K | Multi-GPU server only |
Mixture-of-experts models are the trap here. Qwen3 30B-A3B activates only ~3B parameters per token but still needs all ~18 GB resident. At the extreme, the database puts Kimi K3 at ~1.4 TB at 4-bit — a GGUF exists in principle, but no single machine you own will hold it. For an exact figure at your context length, use the VRAM calculator or the per-model breakdown in VRAM requirements for every major LLM. If you are still choosing hardware, best GPUs for local LLMs covers the VRAM-per-dollar tradeoff.
Where to download GGUF models
- Hugging Face — filter the model list with library=gguf. Most repos ship every quant in one repo; download only the file you need, not the whole repo.
- Ollama library — ollama.com/library serves pre-packaged GGUF with the chat template already wired up. See the Ollama models list.
- LM Studio — the in-app Discover tab searches Hugging Face GGUF repos and flags which quants fit your detected RAM/VRAM. Walkthrough: LM Studio complete guide.
Large models arrive sharded as model-00001-of-00003.gguf and so on. Download every shard into the same folder and point the loader at shard 00001 — it finds the rest.
Running GGUF models on Linux
Build llama.cpp with CUDA support:
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -jBinaries land in build/bin/. Start an OpenAI-compatible server:
./build/bin/llama-server -m ~/models/model-Q4_K_M.gguf -c 8192 -ngl 99 --port 8080-ngl (--n-gpu-layers) is the offload dial: 99 means “all layers on GPU”, 0 means CPU only, and intermediate values split the model when it does not fit. -c sets context; leaving it at the model’s full trained context can cost more VRAM than the weights. The endpoint is then http://localhost:8080/v1/chat/completions.
Two version caveats: the CMake flag was LLAMA_CUBLAS in older releases, and the binaries were renamed (main → llama-cli, server → llama-server) in 2024. Check the repo README for the build you cloned. For AMD or Intel GPUs, substitute the ROCm/Vulkan/SYCL backend flag documented there rather than guessing.
If you use Ollama instead, models live in /usr/share/ollama/.ollama/models when it runs as a system service, or ~/.ollama/models for a user install. See how to install Ollama.
Running GGUF models on macOS
Apple Silicon is unusually good at GGUF because unified memory means the GPU can address most of system RAM — a 64 GB Mac can hold a ~40 GB 4-bit 70B model that would need two 24 GB PC cards.
brew install llama.cpp
llama-server -m ~/models/model-Q4_K_M.gguf -c 8192 --port 8080Metal offload is enabled in the Homebrew build, so you generally do not need -ngl. macOS caps how much RAM the GPU may claim (adjustable via an iogpu sysctl, which varies by macOS release), so leave headroom for the OS. Ollama stores models in ~/.ollama/models; LM Studio uses ~/.lmstudio/models on current versions and ~/.cache/lm-studio/models on older ones — the My Models tab shows and changes the real path.
Running GGUF models on Windows
Three routes, easiest first:
- LM Studio — GUI installer, in-app model search, a local server toggle. Best default for non-developers.
- Ollama — native Windows installer; models go to
C:Users<you>.ollamamodels. Set theOLLAMA_MODELSenvironment variable to move them off the system drive. Background: what is Ollama. - Prebuilt llama.cpp binaries — the GitHub Releases page publishes zipped Windows builds per backend (CUDA, Vulkan, CPU). Unzip and run
llama-server.exe -m model.gguf -ngl 99from PowerShell. Pick the CUDA build for NVIDIA; Vulkan is the safe cross-vendor fallback for AMD and Intel Arc.
Windows-specific gotchas: Defender real-time scanning slows first load of a multi-gigabyte file, and running llama.cpp inside WSL2 costs you a slice of RAM to the VM. Native Windows builds are usually the simpler path.
Loading an arbitrary GGUF into Ollama
Recent Ollama versions can pull straight from Hugging Face:
ollama run hf.co/<user>/<repo>:Q4_K_MFor a local file, write a Modelfile in the same directory:
FROM ./model-Q4_K_M.ggufthen ollama create my-model -f Modelfile and ollama run my-model. If the GGUF lacks a usable chat template, add a TEMPLATE and PARAMETER stop line — the exact Modelfile directives have grown over releases, so check ollama help create for your version. Curated picks: best local LLMs for Ollama.
When GGUF is the wrong answer
GGUF optimizes one user at a time. Serving many concurrent requests, or needing tensor parallelism across GPUs, points to safetensors with vLLM or SGLang. And running locally is not automatically cheaper: hosted Llama 3.3 70B is $0.10 in / $0.32 out per 1M tokens, while frontier hosted models like Claude Sonnet 5 sit at $2.00 in / $10.00 out per 1M tokens for 1M context. Run your own volume through the API cost calculator and the self-hosting vs API break-even calculator before buying a GPU.
Frequently asked questions
What does GGUF stand for, and how is it different from GGML?
GGUF stands for GPT-Generated Unified Format. GGML was the earlier format from the same project; it broke compatibility whenever new metadata was needed. GGUF added an extensible key-value metadata block so new architectures and options can be added without invalidating old files. Pre-GGUF .bin GGML files no longer load in current llama.cpp.
Which quantization should I download?
Start with Q4_K_M. If the model still leaves several GB of VRAM free, move up to Q5_K_M or Q6_K. If it does not fit, prefer a smaller model at Q4_K_M over the same model at Q2_K — a 14B at 4-bit generally beats a 32B mangled down to 2-bit. Check fit with the VRAM calculator at your intended context length.
Can I run GGUF models without a GPU?
Yes — CPU-only inference is what GGUF was built for. Speed is bound by memory bandwidth, so expect single-digit tokens per second for a 7B–8B model at Q4_K_M on typical dual-channel desktop RAM, and slower for anything larger. Small models like Gemma 3 4B (~3 GB at 4-bit) are the practical CPU-only choice.
Can I convert a Hugging Face model to GGUF myself?
Yes. llama.cpp ships a conversion script (convert_hf_to_gguf.py in current versions; the filename used hyphens in older ones) that produces an F16 GGUF, then the llama-quantize binary compresses it to a target quant. Check the script’s --help in the copy you cloned, and confirm your architecture is supported before starting — unsupported architectures fail at conversion.
Do GGUF models support vision or tool calling?
Tool calling works where the model’s chat template defines it and your loader honours the template. Multimodal models need a second file — an mmproj GGUF holding the vision projector — loaded alongside the text weights. The multimodal tooling in llama.cpp has been renamed and reorganised more than once, so follow the current repo docs rather than an old tutorial.
Does quantization change the context window?
No. Context is a property of the model, not the quant: Llama 3.3 70B is 128K and Llama 4 Scout is 10M whether you run F16 or Q4_K_M. What changes is whether you can afford the KV cache for that context in memory. Compare context and pricing across models on the LLM leaderboard.