GGUF Model Format: What It Is and How to Run One

A GGUF model is a large language model packaged in the GGUF file format — a single-file container that holds the quantized weights, tokenizer, hyperparameters and prompt template for a model. It was created by Georgi Gerganov and the llama.cpp project in August 2023 to replace the older GGML format. GGUF is what makes it possible to download one file, point a runtime at it, and get a working local LLM on a laptop or workstation.

What GGUF Actually Is

GGUF stands for GPT-Generated Unified Format. Structurally it is a binary file with a header, a key-value metadata block, a tensor index and the tensor data itself. The official GGUF specification in the ggml repository defines the layout.

Two properties matter for users:

• Self-contained. The file carries the tokenizer, special tokens, EOS/BOS IDs, chat template and architecture metadata. You do not need a separate tokenizer.json or config.json next to it.
• Memory-mappable. Runtimes mmap() the file, so startup is near-instant and the OS pages weights in on demand. This is why a 40 GB GGUF opens in seconds even on a slow SSD.

GGUF replaced GGML because GGML had no versioning, mixed metadata with weights, and broke every time a new architecture landed. GGUF is versioned and extensible.

Quantization: The Naming Scheme

Most GGUF files you download are quantized — weights stored in fewer bits than the original FP16. The suffix in the filename tells you which scheme was used. The current families are documented in the llama.cpp quantize README.

Quant	Bits/weight (approx)	Typical use
Q2_K	~2.6	Smallest, noticeable quality loss
Q3_K_M	~3.9	Aggressive compression
Q4_K_M	~4.8	Most common default — good size/quality trade
Q5_K_M	~5.7	Higher quality, larger file
Q6_K	~6.6	Near-lossless vs FP16
Q8_0	~8.5	Effectively lossless, ~2× the size of Q4
F16 / BF16	16	Unquantized reference

The _K variants are k-quants, which use variable precision per tensor block. The _M / _S / _L suffixes indicate medium, small or large block-mix presets. Newer IQ variants (e.g. IQ4_XS) use importance-matrix calibration for better quality at the same bit budget.

As a rule of thumb for VRAM, take the file size on disk and add roughly 1–3 GB for KV cache at short contexts, more for long ones. From the Convly models database: Llama 3.1 8B needs around 5 GB at 4-bit, Llama 3.3 70B around 40 GB, and Mistral 7B around 4.5 GB. For anything larger, the VRAM requirements table is the fastest lookup.

Where to Get GGUF Files

Almost all GGUF distribution happens on Hugging Face. Search the model name plus “GGUF”. Reliable re-quantizers include bartowski, Qwen (official for Qwen3), lmstudio-community, unsloth and MaziyarPanahi. Meta, Google, Mistral and Microsoft publish some official GGUF builds; for others the community quantizes from the released safetensors.

A repo typically contains one file per quant level, e.g. Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf. For models over ~50 GB the file is split into shards named -00001-of-00003.gguf; runtimes load them from the first shard automatically.

Running a GGUF Model

Linux

Build llama.cpp from source or install the prebuilt binary. With CUDA:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j
./build/bin/llama-cli -m ~/models/llama-3.1-8b-instruct-Q4_K_M.gguf 
    -p "Explain GGUF in one paragraph." -n 256 -ngl 99

The -ngl (n-gpu-layers) flag decides how many transformer layers move to the GPU. Set it high enough that the whole model fits in VRAM; if you run out, lower it and the rest stays on CPU. To serve an OpenAI-compatible HTTP API, use llama-server on port 8080 by default.

macOS

On Apple Silicon, llama.cpp uses the Metal backend automatically. Install via Homebrew:

brew install llama.cpp
llama-cli -m ~/models/qwen3-8b-Q5_K_M.gguf -p "Hello" -ngl 99

Unified memory means the -ngl ceiling is your total RAM minus the OS overhead. A 32 GB M-series Mac comfortably runs Qwen3 14B or Gemma 3 12B at Q4_K_M — the database lists those at ~9 GB and ~8 GB of VRAM respectively.

For a GUI, LM Studio is the usual pick on Mac. It downloads GGUF files from Hugging Face directly and exposes a local OpenAI-compatible server.

Windows

Three practical routes:

• Ollama. Install from ollama.com/download, then ollama run llama3.1:8b. Ollama pulls its own curated GGUFs. To load an arbitrary file, write a Modelfile with a FROM ./mymodel.gguf line and run ollama create mymodel -f Modelfile. See our Ollama install guide.
• LM Studio. One-click install, browse and download GGUF from a UI, GPU offload slider.
• llama.cpp release binaries. The project ships prebuilt Windows zips for CPU, CUDA and Vulkan on the GitHub releases page. Unzip and run llama-cli.exe the same way as on Linux.

For GPU choice, the local LLM GPU guide covers the current price/VRAM options.

GGUF vs Safetensors vs AWQ vs GPTQ

Format	Primary runtime	Precision	Best for
GGUF	llama.cpp, Ollama, LM Studio	2–8 bit + F16	CPU, mixed CPU/GPU, Apple Silicon, single-user
Safetensors (FP16/BF16)	Transformers, vLLM	16-bit	Training, research, full-precision inference
AWQ	vLLM, AutoAWQ	4-bit	High-throughput GPU serving
GPTQ	vLLM, ExLlamaV2	3–8 bit	GPU-only inference with batching
MLX	MLX (Apple)	4–16 bit	Apple Silicon only

Pick GGUF when the target is a single user on mixed hardware or when you want the model to run at all on CPU. Pick vLLM with AWQ/GPTQ safetensors when you are serving many concurrent users on a datacentre GPU.

When GGUF Is Not the Right Answer

GGUF assumes single-stream, batch-size-1 inference. Throughput is much lower than vLLM or TensorRT-LLM under concurrent load, and there is no paged attention. If you are serving an API to real traffic, an FP16 or AWQ deployment on vLLM will beat GGUF on tokens/sec/dollar even before you count the developer time.

It also breaks down at the top end of model size. Kimi K3 needs ~1.4 TB of VRAM at 4-bit and DeepSeek V4-Pro around 800 GB — those are multi-node deployments, not desktop GGUF workloads. And for frontier hosted models like Claude Sonnet 4.6 at $3 in / $15 out per 1M tokens or Gemini 3.6 Flash at $1.50 in / $7.50 out, no local weights exist to convert. Run the numbers on the self-hosting vs API calculator before committing to a GPU purchase.

Converting a Model to GGUF

If a model exists on Hugging Face as safetensors but no one has quantized it yet, the workflow is:

# in the llama.cpp repo
pip install -r requirements.txt
python convert_hf_to_gguf.py /path/to/hf-model --outfile model-f16.gguf
./build/bin/llama-quantize model-f16.gguf model-Q4_K_M.gguf Q4_K_M

The conversion script only supports architectures llama.cpp has been taught — llama, mistral, qwen2/3, gemma, phi, and a growing list of others. Novel architectures need a code change first. Filenames and flags in the script have shifted between llama.cpp versions, so check python convert_hf_to_gguf.py --help against the commit you built.

Frequently Asked Questions

Is GGUF the same as GGML?

No. GGML was the earlier format used by llama.cpp through mid-2023. GGUF replaced it in August 2023 with a versioned header, embedded metadata and a stable tensor layout. Old .bin GGML files no longer load in current llama.cpp; you have to find a re-quantized GGUF version.

Which quantization should I download?

Start with Q4_K_M. It fits the widest range of hardware and quality loss versus FP16 is small for most models above 7B parameters. Move up to Q5_K_M or Q6_K if you have spare VRAM and care about coding or reasoning accuracy. Drop to Q3_K_M or an IQ3 variant only if the model does not otherwise fit.

Can GGUF models use my GPU?

Yes. llama.cpp supports CUDA (NVIDIA), ROCm (AMD), Metal (Apple), Vulkan (cross-vendor) and SYCL (Intel). The -ngl flag offloads layers to the GPU. If the model fits entirely in VRAM, throughput is close to dedicated GPU runtimes; partial offload runs at the speed of whichever tier holds the slowest layer.

Does GGUF work with vision or audio models?

Partially. llama.cpp supports multimodal LLaVA-style vision models via a paired mmproj GGUF file that holds the vision projector. Support for Qwen-VL, Gemma 3 vision and similar has been added over time but lags behind text-only releases. Audio models like Whisper have their own related format handled by whisper.cpp, not the main llama.cpp binary.

How do I pick a GGUF model that fits my hardware?

Look up the model in the Convly models database for its 4-bit VRAM figure, then subtract 1–3 GB of headroom for context and OS overhead. For a 24 GB card, Qwen3 32B (~20 GB) or Gemma 3 27B (~16 GB) at Q4 are comfortable. The VRAM calculator handles longer contexts, which grow the KV cache significantly.

Are GGUF files safe to run?

The weights themselves are just numbers — unlike Python pickle-based checkpoints, GGUF does not execute code on load. The risk is a maliciously crafted file triggering a parser bug in the runtime. Stick to known Hugging Face uploaders (bartowski, unsloth, lmstudio-community, official vendor accounts) and keep llama.cpp, Ollama or LM Studio updated.