{"id":2605,"date":"2026-09-09T20:14:45","date_gmt":"2026-09-09T20:14:45","guid":{"rendered":"https:\/\/convly.ai\/?p=2605"},"modified":"2026-09-09T20:14:45","modified_gmt":"2026-09-09T20:14:45","slug":"gguf-models","status":"publish","type":"post","link":"https:\/\/convly.ai\/ar\/gguf-models\/","title":{"rendered":"GGUF Models: What They Are and How to Run Them"},"content":{"rendered":"<div class=\"convly-tldr\">\n<ul>\n<li><strong>GGUF is a single-file container format for quantized models<\/strong>, created for <a href=\"https:\/\/github.com\/ggml-org\/llama.cpp\" rel=\"noopener\" target=\"_blank\">llama.cpp<\/a>. One <code>.gguf<\/code> file holds the weights, the tokenizer, the chat template and the metadata \u2014 no config folder, no separate tokenizer files.<\/li>\n<li><strong>GGUF models are what Ollama, LM Studio, KoboldCpp, Jan and llama.cpp actually load.<\/strong> If you run a model locally on consumer hardware, you are almost certainly running GGUF.<\/li>\n<li><strong>Default choice: <code>Q4_K_M<\/code>.<\/strong> Roughly 0.6 GB of file per billion parameters \u2014 an 8B model lands near 5 GB, which matches the ~5 GB 4-bit figure the <a href=\"https:\/\/convly.ai\/ar\/models\/\">Convly<\/a> lists for Llama 3.1 8B.<\/li>\n<li><strong>GGUF is for single-user local inference.<\/strong> For concurrent serving use safetensors with vLLM or an API instead.<\/li>\n<\/ul>\n<\/div>\n<p>GGUF (GPT-Generated Unified Format) is the file format the llama.cpp\/ggml ecosystem uses to store a model ready for inference. A single <code>.gguf<\/code> file contains the quantized tensors plus everything needed to use them: vocabulary, tokenizer settings, the chat template, and architecture metadata. It replaced the older GGML format in August 2023 and is now the de facto standard for running models on laptops, desktops and CPUs.<\/p>\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_87_1 counter-flat ez-toc-counter ez-toc-container-direction\">\n<label for=\"ez-toc-cssicon-toggle-item-6aa1d082a54e6\" class=\"ez-toc-cssicon-toggle-label\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">\u062a\u0628\u062f\u064a\u0644<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #000000;color:#000000\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewbox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #000000;color:#000000\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewbox=\"0 0 24 24\" version=\"1.2\" baseprofile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/label><input type=\"checkbox\"  id=\"ez-toc-cssicon-toggle-item-6aa1d082a54e6\"  aria-label=\"\u062a\u0628\u062f\u064a\u0644\" \/><nav><ul class='ez-toc-list ez-toc-list-level-1' ><li class='ez-toc-page-1'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/convly.ai\/ar\/gguf-models\/#What_is_actually_inside_a_gguf_file\" >What is actually inside a .gguf file<\/a><\/li><li class='ez-toc-page-1'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/convly.ai\/ar\/gguf-models\/#How_to_read_a_GGUF_filename\" >How to read a GGUF filename<\/a><\/li><li class='ez-toc-page-1'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/convly.ai\/ar\/gguf-models\/#Sizing_which_GGUF_models_fit_your_GPU\" >Sizing: which GGUF models fit your GPU<\/a><\/li><li class='ez-toc-page-1'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/convly.ai\/ar\/gguf-models\/#Where_to_download_GGUF_models\" >Where to download GGUF models<\/a><\/li><li class='ez-toc-page-1'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/convly.ai\/ar\/gguf-models\/#Running_GGUF_models_on_Linux\" >Running GGUF models on Linux<\/a><\/li><li class='ez-toc-page-1'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/convly.ai\/ar\/gguf-models\/#Running_GGUF_models_on_macOS\" >Running GGUF models on macOS<\/a><\/li><li class='ez-toc-page-1'><a class=\"ez-toc-link ez-toc-heading-7\" href=\"https:\/\/convly.ai\/ar\/gguf-models\/#Running_GGUF_models_on_Windows\" >Running GGUF models on Windows<\/a><\/li><li class='ez-toc-page-1'><a class=\"ez-toc-link ez-toc-heading-8\" href=\"https:\/\/convly.ai\/ar\/gguf-models\/#Loading_an_arbitrary_GGUF_into_Ollama\" >Loading an arbitrary GGUF into Ollama<\/a><\/li><li class='ez-toc-page-1'><a class=\"ez-toc-link ez-toc-heading-9\" href=\"https:\/\/convly.ai\/ar\/gguf-models\/#When_GGUF_is_the_wrong_answer\" >When GGUF is the wrong answer<\/a><\/li><li class='ez-toc-page-1'><a class=\"ez-toc-link ez-toc-heading-10\" href=\"https:\/\/convly.ai\/ar\/gguf-models\/#Frequently_asked_questions\" >\u0627\u0644\u0623\u0633\u0626\u0644\u0629 \u0627\u0644\u0634\u0627\u0626\u0639\u0629<\/a><\/li><\/ul><\/nav><\/div>\n<h2><span class=\"ez-toc-section\" id=\"What_is_actually_inside_a_gguf_file\"><\/span>What is actually inside a .gguf file<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>A GGUF file is a header, a key-value metadata block, then the tensor data. The metadata is the part that matters practically \u2014 it is why a GGUF model needs no sidecar files. Hugging Face&#8217;s <a href=\"https:\/\/huggingface.co\/docs\/hub\/en\/gguf\" rel=\"noopener\" target=\"_blank\">GGUF documentation<\/a> describes the layout and the Hub&#8217;s built-in GGUF metadata viewer, which lets you inspect the quantization type, context length and chat template of a file before downloading several gigabytes of it.<\/p>\n<p>Typical metadata keys include the architecture (<code>llama<\/code>, <code>qwen3<\/code>, <code>gemma3<\/code>, <code>phi3<\/code>), the training context length, RoPE settings, the full vocabulary, and the Jinja chat template. A loader reads that block and configures itself; you do not pass a tokenizer or a prompt format by hand.<\/p>\n<h3>\u0645\u0642\u0627\u0631\u0646\u0629 \u0628\u064a\u0646 GGUF \u0648safetensors<\/h3>\n<table>\n<thead>\n<tr>\n<th><\/th>\n<th>GGUF<\/th>\n<th>safetensors<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>Primary runtime<\/td>\n<td>llama.cpp, Ollama, LM Studio<\/td>\n<td>PyTorch, vLLM, Transformers, TGI<\/td>\n<\/tr>\n<tr>\n<td>Files per model<\/td>\n<td>One (or numbered shards)<\/td>\n<td>Weights plus config\/tokenizer folder<\/td>\n<\/tr>\n<tr>\n<td>\u0627\u0644\u062a\u0643\u0645\u064a\u0629<\/td>\n<td>Baked in (Q4_K_M, Q8_0, IQ\u2026)<\/td>\n<td>Usually FP16\/BF16, or GPTQ\/AWQ variants<\/td>\n<\/tr>\n<tr>\n<td>CPU + partial GPU offload<\/td>\n<td>Yes, core design goal<\/td>\n<td>Limited and slow<\/td>\n<\/tr>\n<tr>\n<td>Concurrent batched serving<\/td>\n<td>\u0636\u0639\u064a\u0641<\/td>\n<td>\u0642\u0648\u064a<\/td>\n<\/tr>\n<tr>\n<td>\u0627\u0644\u0636\u0628\u0637 \u0627\u0644\u062f\u0642\u064a\u0642<\/td>\n<td>No (convert back first)<\/td>\n<td>\u0646\u0639\u0645<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<h2><span class=\"ez-toc-section\" id=\"How_to_read_a_GGUF_filename\"><\/span>How to read a GGUF filename<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Files are usually named <code>Model-Name-8B-Instruct-Q4_K_M.gguf<\/code>. The suffix is the quantization mix. The number is the nominal bit width; <code>_K<\/code> means k-quants, which keep attention and embedding tensors at higher precision than the bulk feed-forward weights; <code>S<\/code>\/<code>M<\/code>\/<code>L<\/code> are small\/medium\/large variants of that mix.<\/p>\n<table>\n<thead>\n<tr>\n<th>\u0627\u0644\u0643\u0645\u0651<\/th>\n<th>Approx. bits\/weight<\/th>\n<th>Approx. size, 8B model<\/th>\n<th>When to use it<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>Q8_0<\/td>\n<td>~8.5<\/td>\n<td>~\u0668,\u0665 \u063a\u064a\u063a\u0627\u0628\u0627\u064a\u062a<\/td>\n<td>Near-lossless reference; small models only<\/td>\n<\/tr>\n<tr>\n<td>Q6_K<\/td>\n<td>~6.6<\/td>\n<td>~\u0666,\u0666 \u063a\u064a\u063a\u0627\u0628\u0627\u064a\u062a<\/td>\n<td>You have VRAM to spare<\/td>\n<\/tr>\n<tr>\n<td>Q5_K_M<\/td>\n<td>~5.7<\/td>\n<td>~5.7 \u063a\u064a\u063a\u0627\u0628\u0627\u064a\u062a<\/td>\n<td>Quality-leaning default<\/td>\n<\/tr>\n<tr>\n<td>Q4_K_M<\/td>\n<td>~4.9<\/td>\n<td>~4.9 \u063a\u064a\u063a\u0627\u0628\u0627\u064a\u062a<\/td>\n<td><strong>The usual default<\/strong><\/td>\n<\/tr>\n<tr>\n<td>Q4_0<\/td>\n<td>~4.5<\/td>\n<td>~4.5 \u063a\u064a\u063a\u0627\u0628\u0627\u064a\u062a<\/td>\n<td>Legacy; some accelerators prefer it<\/td>\n<\/tr>\n<tr>\n<td>Q3_K_M<\/td>\n<td>~3.9<\/td>\n<td>~4.0 \u063a\u064a\u063a\u0627\u0628\u0627\u064a\u062a<\/td>\n<td>Squeezing a bigger model into small VRAM<\/td>\n<\/tr>\n<tr>\n<td>Q2_K \/ IQ2<\/td>\n<td>~2.5\u20133.4<\/td>\n<td>~2.5\u20133.4 GB<\/td>\n<td>Last resort; noticeable degradation<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>Treat the bits-per-weight column as approximate. Actual sizes shift with model architecture (a large vocabulary inflates the embedding tensors) and with the llama.cpp version, because the quantization mixes are tuned over time. The <code>\u0628\u0627\u062f\u0626\u0627\u062a IQ<\/code> family (IQ2_XXS through IQ4_NL) uses importance-matrix calibration to hold up better at very low bit widths, and quantizers such as Bartowski and Unsloth publish IQ variants alongside the standard K-quants.<\/p>\n<p>Community consensus, not a Convly measurement, is that Q4_K_M is the sweet spot and that quality falls off sharply below roughly 3 bits per weight \u2014 small models degrade faster than large ones at the same quant.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Sizing_which_GGUF_models_fit_your_GPU\"><\/span>Sizing: which GGUF models fit your GPU<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>The file size is the floor, not the total. Add the KV cache for your context length plus roughly 500 MB\u20131 GB of overhead. These 4-bit figures come from the <a href=\"https:\/\/convly.ai\/ar\/models\/\">Convly<\/a>:<\/p>\n<table>\n<thead>\n<tr>\n<th>\u0627\u0644\u0646\u0645\u0648\u0630\u062c<\/th>\n<th>VRAM \u0639\u0646\u062f \u062a\u0643\u0645\u064a\u0645 \u0664 \u0628\u062a<\/th>\n<th>\u0627\u0644\u0633\u064a\u0627\u0642<\/th>\n<th>Realistic GPU<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>Gemma 3 4B<\/td>\n<td>~3 \u062c\u064a\u062c\u0627\u0628\u0627\u064a\u062a<\/td>\n<td>128 \u0623\u0644\u0641 \u0631\u0645\u0632<\/td>\n<td>Any 6 GB card, integrated GPU<\/td>\n<\/tr>\n<tr>\n<td>Mistral 7B<\/td>\n<td>~4.5 \u063a\u064a\u063a\u0627\u0628\u0627\u064a\u062a<\/td>\n<td>32K<\/td>\n<td>8 GB card<\/td>\n<\/tr>\n<tr>\n<td>Llama 3.1 8B<\/td>\n<td>~5 \u063a\u064a\u063a\u0627\u0628\u0627\u064a\u062a<\/td>\n<td>128 \u0623\u0644\u0641 \u0631\u0645\u0632<\/td>\n<td>8 GB card<\/td>\n<\/tr>\n<tr>\n<td>Qwen3 14B<\/td>\n<td>~\u0669 \u062c\u064a\u062c\u0627\u0628\u0627\u064a\u062a<\/td>\n<td>128 \u0623\u0644\u0641 \u0631\u0645\u0632<\/td>\n<td>12 GB card<\/td>\n<\/tr>\n<tr>\n<td>Phi-4<\/td>\n<td>~\u0669 \u062c\u064a\u062c\u0627\u0628\u0627\u064a\u062a<\/td>\n<td>16 \u0623\u0644\u0641 \u0631\u0645\u0632<\/td>\n<td>12 GB card<\/td>\n<\/tr>\n<tr>\n<td>Gemma 3 27B<\/td>\n<td>\u062d\u0648\u0627\u0644\u064a \u0661\u0666 \u062c\u064a\u062c\u0627\u0628\u0627\u064a\u062a<\/td>\n<td>128 \u0623\u0644\u0641 \u0631\u0645\u0632<\/td>\n<td>24 GB card<\/td>\n<\/tr>\n<tr>\n<td>Qwen3 32B<\/td>\n<td>~20 \u062c\u064a\u062c\u0627\u0628\u0627\u064a\u062a<\/td>\n<td>128 \u0623\u0644\u0641 \u0631\u0645\u0632<\/td>\n<td>24 GB card<\/td>\n<\/tr>\n<tr>\n<td>Llama 3.3 70B<\/td>\n<td>~40 \u062c\u064a\u062c\u0627\u0628\u0627\u064a\u062a<\/td>\n<td>128 \u0623\u0644\u0641 \u0631\u0645\u0632<\/td>\n<td>2\u00d724 GB, or 48 GB unified memory<\/td>\n<\/tr>\n<tr>\n<td>DeepSeek R1 (full)<\/td>\n<td>\u0646\u062d\u0648 400 \u062c\u064a\u062c\u0627\u0628\u0627\u064a\u062a<\/td>\n<td>128 \u0623\u0644\u0641 \u0631\u0645\u0632<\/td>\n<td>Multi-GPU server only<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>Mixture-of-experts models are the trap here. Qwen3 30B-A3B activates only ~3B parameters per token but still needs all ~18 GB resident. At the extreme, the database puts Kimi K3 at ~1.4 TB at 4-bit \u2014 a GGUF exists in principle, but no single machine you own will hold it. For an exact figure at your context length, use the <a href=\"https:\/\/convly.ai\/ar\/llm-vram-calculator\/\">\u062d\u0627\u0633\u0628\u0629 \u0627\u0644\u0630\u0627\u0643\u0631\u0629 VRAM<\/a> or the per-model breakdown in <a href=\"https:\/\/convly.ai\/ar\/vram-requirements-every-major-llm-2026\/\">\u0645\u062a\u0637\u0644\u0628\u0627\u062a \u0630\u0627\u0643\u0631\u0629 VRAM \u0644\u0643\u0644 \u0646\u0645\u0648\u0630\u062c \u0644\u063a\u0648\u064a \u0631\u0626\u064a\u0633\u064a<\/a>. If you are still choosing hardware, <a href=\"https:\/\/convly.ai\/ar\/best-gpus-for-local-llms-2026\/\">\u0623\u0641\u0636\u0644 \u0648\u062d\u062f\u0627\u062a \u0645\u0639\u0627\u0644\u062c\u0629 \u0627\u0644\u0631\u0633\u0648\u0645\u064a\u0627\u062a \u0644\u062a\u0634\u063a\u064a\u0644 \u0646\u0645\u0627\u0630\u062c \u0627\u0644\u0644\u063a\u0629 \u0627\u0644\u0643\u0628\u064a\u0631\u0629 \u0645\u062d\u0644\u064a\u064b\u0651\u0627<\/a> covers the VRAM-per-dollar tradeoff.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Where_to_download_GGUF_models\"><\/span>Where to download GGUF models<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<ul>\n<li><strong>Hugging Face<\/strong> \u2014 filter the model list with <a href=\"https:\/\/huggingface.co\/models?library=gguf\" rel=\"noopener\" target=\"_blank\">library=gguf<\/a>. Most repos ship every quant in one repo; download only the file you need, not the whole repo.<\/li>\n<li><strong>\u0645\u0643\u062a\u0628\u0629 Ollama<\/strong> \u2014 <a href=\"https:\/\/ollama.com\/library\" rel=\"noopener\" target=\"_blank\">ollama.com\/library<\/a> serves pre-packaged GGUF with the chat template already wired up. See the <a href=\"https:\/\/convly.ai\/ar\/ollama-models-list-2026\/\">\u0642\u0627\u0626\u0645\u0629 \u0646\u0645\u0627\u0630\u062c Ollama<\/a>.<\/li>\n<li><strong>LM Studio<\/strong> \u2014 the in-app Discover tab searches Hugging Face GGUF repos and flags which quants fit your detected RAM\/VRAM. Walkthrough: <a href=\"https:\/\/convly.ai\/ar\/lm-studio-complete-guide-2026\/\">\u0627\u0644\u062f\u0644\u064a\u0644 \u0627\u0644\u0643\u0627\u0645\u0644 \u0644\u0640 LM Studio<\/a>.<\/li>\n<\/ul>\n<p>Large models arrive sharded as <code>model-00001-of-00003.gguf<\/code> and so on. Download every shard into the same folder and point the loader at shard 00001 \u2014 it finds the rest.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Running_GGUF_models_on_Linux\"><\/span>Running GGUF models on Linux<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Build llama.cpp with CUDA support:<\/p>\n<pre><code>git clone https:\/\/github.com\/ggml-org\/llama.cpp\ncd llama.cpp\ncmake -B build -DGGML_CUDA=ON\ncmake --build build --config Release -j<\/code><\/pre>\n<p>Binaries land in <code>build\/bin\/<\/code>. Start an OpenAI-compatible server:<\/p>\n<pre><code>.\/build\/bin\/llama-server -m ~\/models\/model-Q4_K_M.gguf -c 8192 -ngl 99 --port 8080<\/code><\/pre>\n<p><code>(\u0639\u062f\u062f \u0637\u0628\u0642\u0627\u062a \u0648\u062d\u062f\u0629 \u0645\u0639\u0627\u0644\u062c\u0629 \u0627\u0644\u0631\u0633\u0648\u0645\u0627\u062a).<\/code> (<code>--n-gpu-layers<\/code>) is the offload dial: <code>99<\/code> means &#8220;all layers on GPU&#8221;, <code>0<\/code> means CPU only, and intermediate values split the model when it does not fit. <code>-c<\/code> sets context; leaving it at the model&#8217;s full trained context can cost more VRAM than the weights. The endpoint is then <code>http:\/\/localhost:8080\/v1\/chat\/completions<\/code>.<\/p>\n<p>Two version caveats: the CMake flag was <code>LLAMA_CUBLAS<\/code> in older releases, and the binaries were renamed (<code>main<\/code> \u2192 <code>llama-cli<\/code>, <code>server<\/code> \u2192 <code>llama-server<\/code>) in 2024. Check the repo README for the build you cloned. For AMD or Intel GPUs, substitute the ROCm\/Vulkan\/SYCL backend flag documented there rather than guessing.<\/p>\n<p>If you use Ollama instead, models live in <code>\/usr\/share\/ollama\/.ollama\/models<\/code> when it runs as a system service, or <code>~\/.ollama\/models<\/code> for a user install. See <a href=\"https:\/\/convly.ai\/ar\/how-to-install-ollama-2026\/\">\u0643\u064a\u0641\u064a\u0629 \u062a\u062b\u0628\u064a\u062a \u0645\u0646\u0635\u0629 Ollama<\/a>.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Running_GGUF_models_on_macOS\"><\/span>Running GGUF models on macOS<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Apple Silicon is unusually good at GGUF because unified memory means the GPU can address most of system RAM \u2014 a 64 GB Mac can hold a ~40 GB 4-bit 70B model that would need two 24 GB PC cards.<\/p>\n<pre><code>brew install llama.cpp\nllama-server -m ~\/models\/model-Q4_K_M.gguf -c 8192 --port 8080<\/code><\/pre>\n<p>Metal offload is enabled in the Homebrew build, so you generally do not need <code>(\u0639\u062f\u062f \u0637\u0628\u0642\u0627\u062a \u0648\u062d\u062f\u0629 \u0645\u0639\u0627\u0644\u062c\u0629 \u0627\u0644\u0631\u0633\u0648\u0645\u0627\u062a).<\/code>. macOS caps how much RAM the GPU may claim (adjustable via an <code>iogpu<\/code> sysctl, which varies by macOS release), so leave headroom for the OS. Ollama stores models in <code>~\/.ollama\/models<\/code>; LM Studio uses <code>~\/.lmstudio\/models<\/code> on current versions and <code>~\/.cache\/lm-studio\/models<\/code> on older ones \u2014 the My Models tab shows and changes the real path.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Running_GGUF_models_on_Windows\"><\/span>Running GGUF models on Windows<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Three routes, easiest first:<\/p>\n<ol>\n<li><strong>LM Studio<\/strong> \u2014 GUI installer, in-app model search, a local server toggle. Best default for non-developers.<\/li>\n<li><strong>Ollama<\/strong> \u2014 native Windows installer; models go to <code>C:\\Users\\&lt;you&gt;\\.ollama\\models<\/code>. Set the <code>OLLAMA_MODELS<\/code> environment variable to move them off the system drive. Background: <a href=\"https:\/\/convly.ai\/ar\/what-is-ollama-complete-guide-2026\/\">what is Ollama<\/a>.<\/li>\n<li><strong>Prebuilt llama.cpp binaries<\/strong> \u2014 the GitHub Releases page publishes zipped Windows builds per backend (CUDA, Vulkan, CPU). Unzip and run <code>llama-server.exe -m model.gguf -ngl 99<\/code> from PowerShell. Pick the CUDA build for NVIDIA; Vulkan is the safe cross-vendor fallback for AMD and Intel Arc.<\/li>\n<\/ol>\n<p>Windows-specific gotchas: Defender real-time scanning slows first load of a multi-gigabyte file, and running llama.cpp inside WSL2 costs you a slice of RAM to the VM. Native Windows builds are usually the simpler path.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Loading_an_arbitrary_GGUF_into_Ollama\"><\/span>Loading an arbitrary GGUF into Ollama<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Recent Ollama versions can pull straight from Hugging Face:<\/p>\n<pre><code>ollama run hf.co\/&lt;user&gt;\/&lt;repo&gt;:Q4_K_M<\/code><\/pre>\n<p>For a local file, write a Modelfile in the same directory:<\/p>\n<pre><code>FROM .\/model-Q4_K_M.gguf<\/code><\/pre>\n<p>\u062b\u0645 <code>ollama create my-model -f Modelfile<\/code> \u0648 <code>ollama run my-model<\/code>. If the GGUF lacks a usable chat template, add a <code>TEMPLATE<\/code> \u0648 <code>PARAMETER stop<\/code> line \u2014 the exact Modelfile directives have grown over releases, so check <code>ollama help create<\/code> for your version. Curated picks: <a href=\"https:\/\/convly.ai\/ar\/best-local-llms-to-run-on-ollama-2026\/\">\u0623\u0641\u0636\u0644 \u0646\u0645\u0627\u0630\u062c \u0627\u0644\u0644\u063a\u0629 \u0627\u0644\u0645\u062d\u0644\u064a\u0629 (LLMs) \u0644\u0640 Ollama<\/a>.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"When_GGUF_is_the_wrong_answer\"><\/span>When GGUF is the wrong answer<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>GGUF optimizes one user at a time. Serving many concurrent requests, or needing tensor parallelism across GPUs, points to safetensors with vLLM or SGLang. And running locally is not automatically cheaper: hosted Llama 3.3 70B is $0.10 in \/ $0.32 out per 1M tokens, while frontier hosted models like Claude Sonnet 5 sit at $2.00 in \/ $10.00 out per 1M tokens for 1M context. Run your own volume through the <a href=\"https:\/\/convly.ai\/ar\/ai-api-cost-calculator\/\">\u062d\u0627\u0633\u0628\u0629 \u062a\u0643\u0644\u0641\u0629 \u0648\u0627\u062c\u0647\u0629 \u0628\u0631\u0645\u062c\u0629 \u0627\u0644\u062a\u0637\u0628\u064a\u0642\u0627\u062a (API)<\/a> \u0648\u0627\u0644\u0640 <a href=\"https:\/\/convly.ai\/ar\/self-hosting-vs-api-calculator\/\">\u062d\u0627\u0633\u0628\u0629 \u0646\u0642\u0637\u0629 \u0627\u0644\u062a\u0639\u0627\u062f\u0644 \u0628\u064a\u0646 \u0627\u0644\u0627\u0633\u062a\u0636\u0627\u0641\u0629 \u0627\u0644\u0645\u062d\u0644\u064a\u0629 \u0648\u0648\u0627\u062c\u0647\u0629 \u0628\u0631\u0645\u062c\u0629 \u0627\u0644\u062a\u0637\u0628\u064a\u0642\u0627\u062a (API)<\/a> before buying a GPU.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Frequently_asked_questions\"><\/span>\u0627\u0644\u0623\u0633\u0626\u0644\u0629 \u0627\u0644\u0634\u0627\u0626\u0639\u0629<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<h3>What does GGUF stand for, and how is it different from GGML?<\/h3>\n<p>GGUF stands for GPT-Generated Unified Format. GGML was the earlier format from the same project; it broke compatibility whenever new metadata was needed. GGUF added an extensible key-value metadata block so new architectures and options can be added without invalidating old files. Pre-GGUF <code>\u0623\u0648 .bin<\/code> GGML files no longer load in current llama.cpp.<\/p>\n<h3>Which quantization should I download?<\/h3>\n<p>Start with Q4_K_M. If the model still leaves several GB of VRAM free, move up to Q5_K_M or Q6_K. If it does not fit, prefer a smaller model at Q4_K_M over the same model at Q2_K \u2014 a 14B at 4-bit generally beats a 32B mangled down to 2-bit. Check fit with the <a href=\"https:\/\/convly.ai\/ar\/llm-vram-calculator\/\">\u062d\u0627\u0633\u0628\u0629 \u0627\u0644\u0630\u0627\u0643\u0631\u0629 VRAM<\/a> at your intended context length.<\/p>\n<h3>Can I run GGUF models without a GPU?<\/h3>\n<p>Yes \u2014 CPU-only inference is what GGUF was built for. Speed is bound by memory bandwidth, so expect single-digit tokens per second for a 7B\u20138B model at Q4_K_M on typical dual-channel desktop RAM, and slower for anything larger. Small models like Gemma 3 4B (~3 GB at 4-bit) are the practical CPU-only choice.<\/p>\n<h3>Can I convert a Hugging Face model to GGUF myself?<\/h3>\n<p>Yes. llama.cpp ships a conversion script (<code>convert_hf_to_gguf.py<\/code> in current versions; the filename used hyphens in older ones) that produces an F16 GGUF, then the <code>llama-quantize<\/code> binary compresses it to a target quant. Check the script&#8217;s <code>--help<\/code> in the copy you cloned, and confirm your architecture is supported before starting \u2014 unsupported architectures fail at conversion.<\/p>\n<h3>Do GGUF models support vision or tool calling?<\/h3>\n<p>Tool calling works where the model&#8217;s chat template defines it and your loader honours the template. Multimodal models need a second file \u2014 an <code>mmproj<\/code> GGUF holding the vision projector \u2014 loaded alongside the text weights. The multimodal tooling in llama.cpp has been renamed and reorganised more than once, so follow the current repo docs rather than an old tutorial.<\/p>\n<h3>Does quantization change the context window?<\/h3>\n<p>No. Context is a property of the model, not the quant: Llama 3.3 70B is 128K and Llama 4 Scout is 10M whether you run F16 or Q4_K_M. What changes is whether you can afford the KV cache for that context in memory. Compare context and pricing across models on the <a href=\"https:\/\/convly.ai\/ar\/llm-leaderboard\/\">\u0644\u0648\u062d\u0629 \u062a\u0635\u0646\u064a\u0641 \u0627\u0644\u0646\u0645\u0627\u0630\u062c \u0627\u0644\u0644\u063a\u0648\u064a\u0629 \u0627\u0644\u0643\u0628\u064a\u0631\u0629 (LLM)<\/a>.<\/p>","protected":false},"excerpt":{"rendered":"<p>GGUF is a single-file container format for quantized models, created for llama.cpp. One .gguf file holds the weights, the tokenizer, [\u2026]<\/p>\n","protected":false},"author":1,"featured_media":2606,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"site-sidebar-layout":"default","site-content-layout":"","ast-site-content-layout":"default","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","ast-disable-related-posts":"","theme-transparent-header-meta":"","adv-header-id-meta":"","stick-header-meta":"","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"default","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"footnotes":""},"categories":[7],"tags":[],"class_list":["post-2605","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai-news"],"_links":{"self":[{"href":"https:\/\/convly.ai\/ar\/wp-json\/wp\/v2\/posts\/2605","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/convly.ai\/ar\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/convly.ai\/ar\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/convly.ai\/ar\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/convly.ai\/ar\/wp-json\/wp\/v2\/comments?post=2605"}],"version-history":[{"count":1,"href":"https:\/\/convly.ai\/ar\/wp-json\/wp\/v2\/posts\/2605\/revisions"}],"predecessor-version":[{"id":2607,"href":"https:\/\/convly.ai\/ar\/wp-json\/wp\/v2\/posts\/2605\/revisions\/2607"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/convly.ai\/ar\/wp-json\/wp\/v2\/media\/2606"}],"wp:attachment":[{"href":"https:\/\/convly.ai\/ar\/wp-json\/wp\/v2\/media?parent=2605"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/convly.ai\/ar\/wp-json\/wp\/v2\/categories?post=2605"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/convly.ai\/ar\/wp-json\/wp\/v2\/tags?post=2605"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}