{"id":2141,"date":"2026-08-08T06:13:04","date_gmt":"2026-08-08T06:13:04","guid":{"rendered":"https:\/\/convly.ai\/?p=2141"},"modified":"2026-08-08T06:13:04","modified_gmt":"2026-08-08T06:13:04","slug":"llama-cpp-python-guide","status":"publish","type":"post","link":"https:\/\/convly.ai\/pt\/llama-cpp-python-guide\/","title":{"rendered":"Llama Cpp Python: instala\u00e7\u00e3o, compila\u00e7\u00e3o com suporte a GPU e par\u00e2metros"},"content":{"rendered":"<div class=\"convly-tldr\">\n<ul>\n<li><strong>The plain <code>pip install llama-cpp-python<\/code> gives you a CPU-only build.<\/strong> GPU support requires either a prebuilt GPU wheel or a source build with <code>CMAKE_ARGS<\/code>.<\/li>\n<li><strong>CUDA:<\/strong> <code>CMAKE_ARGS=\"-DGGML_CUDA=on\" pip install llama-cpp-python --no-cache-dir --force-reinstall<\/code>. <strong>Apple Silicon:<\/strong> Metal is built by default in recent releases; force it with <code>-DGGML_METAL=on<\/code>.<\/li>\n<li><strong>Load a model with<\/strong> <code>Llama(model_path=\"model.gguf\", n_gpu_layers=-1, n_ctx=4096)<\/code> and confirm the verbose log says layers were offloaded to GPU.<\/li>\n<li><strong>Server mode:<\/strong> <code>python -m llama_cpp.server --model model.gguf --n_gpu_layers -1<\/code> exposes an OpenAI-compatible API on port 8000, streaming included.<\/li>\n<\/ul>\n<\/div>\n<p>llama-cpp-python is the Python binding for llama.cpp. It loads GGUF models in-process, exposes a low-level ctypes wrapper plus a high-level <code>Llama<\/code> class, and ships an OpenAI-compatible HTTP server. The default pip install compiles a CPU-only build. To use a GPU you install a GPU wheel or rebuild from source with <code>CMAKE_ARGS<\/code>, then pass <code>n_gpu_layers<\/code>.<\/p>\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_85 counter-flat ez-toc-counter ez-toc-container-direction\">\n<label for=\"ez-toc-cssicon-toggle-item-6a76fe8288e36\" class=\"ez-toc-cssicon-toggle-label\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #000000;color:#000000\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #000000;color:#000000\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/label><input type=\"checkbox\"  id=\"ez-toc-cssicon-toggle-item-6a76fe8288e36\"  aria-label=\"Toggle\" \/><nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/convly.ai\/pt\/llama-cpp-python-guide\/#Why_the_default_install_is_CPU-only\" >Why the default install is CPU-only<\/a><\/li><li class='ez-toc-page-1'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/convly.ai\/pt\/llama-cpp-python-guide\/#Installing_llama-cpp-python_with_GPU_support\" >Installing llama-cpp-python with GPU support<\/a><\/li><li class='ez-toc-page-1'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/convly.ai\/pt\/llama-cpp-python-guide\/#How_to_tell_which_build_you_have\" >How to tell which build you have<\/a><\/li><li class='ez-toc-page-1'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/convly.ai\/pt\/llama-cpp-python-guide\/#Loading_a_GGUF_and_running_a_first_completion\" >Loading a GGUF and running a first completion<\/a><\/li><li class='ez-toc-page-1'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/convly.ai\/pt\/llama-cpp-python-guide\/#The_parameters_that_matter\" >The parameters that matter<\/a><\/li><li class='ez-toc-page-1'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/convly.ai\/pt\/llama-cpp-python-guide\/#OpenAI-compatible_server_mode\" >OpenAI-compatible server mode<\/a><\/li><li class='ez-toc-page-1'><a class=\"ez-toc-link ez-toc-heading-7\" href=\"https:\/\/convly.ai\/pt\/llama-cpp-python-guide\/#Common_build_failures_and_fixes\" >Common build failures and fixes<\/a><\/li><li class='ez-toc-page-1'><a class=\"ez-toc-link ez-toc-heading-8\" href=\"https:\/\/convly.ai\/pt\/llama-cpp-python-guide\/#Frequently_asked_questions\" >Frequently asked questions<\/a><\/li><\/ul><\/nav><\/div>\n<h2><span class=\"ez-toc-section\" id=\"Why_the_default_install_is_CPU-only\"><\/span>Why the default install is CPU-only<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>The package is a thin binding around a C++ library that has to be compiled with backend support baked in at build time. There is no runtime flag that turns CUDA on after the fact. When pip builds the sdist with no <code>CMAKE_ARGS<\/code> set, CMake configures the generic CPU backend and that is what you get, permanently, until you rebuild. On macOS arm64 this is less of a problem because recent releases enable the Metal backend by default, but on Linux and Windows a bare install will run entirely on your CPU.<\/p>\n<p>Two consequences worth internalising: first, <code>n_gpu_layers=-1<\/code> on a CPU-only build silently does nothing useful, so people often conclude their GPU is &#8220;too slow&#8221; when it was never touched. Second, pip caches built wheels. Re-running the install with different <code>CMAKE_ARGS<\/code> can hand you the cached CPU wheel again, which is why every rebuild command below includes <code>--no-cache-dir --force-reinstall<\/code>.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Installing_llama-cpp-python_with_GPU_support\"><\/span>Installing llama-cpp-python with GPU support<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<h3>Option 1: prebuilt wheels (no compiler needed)<\/h3>\n<p>The project publishes wheel indexes, including a CPU index at <code>https:\/\/abetlen.github.io\/llama-cpp-python\/whl\/cpu<\/code> and CUDA variants whose path segment encodes the CUDA version, for example <code>...\/whl\/cu124<\/code>. Install with:<\/p>\n<pre><code>pip install llama-cpp-python --extra-index-url https:\/\/abetlen.github.io\/llama-cpp-python\/whl\/cu124<\/code><\/pre>\n<p>Which CUDA tags and Python versions are published changes release to release, and the indexes sometimes lag the newest PyPI version. Check the project README for the tags that currently exist rather than assuming one \u2014 a wrong tag returns a 404 and pip quietly falls back to building from source.<\/p>\n<h3>Option 2: build from source (Linux, CUDA)<\/h3>\n<p>You need a C++ toolchain, CMake, and the CUDA toolkit with <code>nvcc<\/code> on your PATH.<\/p>\n<pre><code>nvcc --version   # must print a version, not \"command not found\"\n\nCMAKE_ARGS=\"-DGGML_CUDA=on\" \n  pip install llama-cpp-python --no-cache-dir --force-reinstall --upgrade<\/code><\/pre>\n<p>The flag name has changed over the project&#8217;s life: very old guides use <code>-DLLAMA_CUBLAS=on<\/code>, mid-2024 guides use <code>-DLLAMA_CUDA=on<\/code>, and current upstream uses the <code>GGML_<\/code> prefix. If a build errors out on an unknown CMake option, that mismatch is usually why. Other backends follow the same pattern \u2014 Vulkan is <code>-DGGML_VULKAN=on<\/code>, SYCL is <code>-DGGML_SYCL=on<\/code>, and the AMD\/ROCm option has been renamed more than once, so read the README for your installed version instead of copying a flag from a forum post.<\/p>\n<p>You can cut compile time substantially by building for only your GPU&#8217;s compute capability, for example <code>-DCMAKE_CUDA_ARCHITECTURES=89<\/code> for an Ada card such as the RTX 4090, or <code>86<\/code> for a 3090. Look your card&#8217;s compute capability up in NVIDIA&#8217;s official list; if you are still choosing hardware, our <a href=\"https:\/\/convly.ai\/best-gpus-for-local-llms-2026\/\">guide to the best GPUs for running LLMs locally<\/a> covers the VRAM-per-dollar tradeoffs.<\/p>\n<h3>Option 3: macOS with Metal<\/h3>\n<pre><code>xcode-select --install\n\nCMAKE_ARGS=\"-DGGML_METAL=on\" \n  pip install llama-cpp-python --no-cache-dir --force-reinstall<\/code><\/pre>\n<p>On Apple Silicon, verify you are not running a Rosetta x86 Python: <code>python -c \"import platform; print(platform.machine())\"<\/code> must print <code>arm64<\/code>. An x86_64 interpreter produces a build with no Metal backend no matter what CMAKE_ARGS you pass. Because the GPU and CPU share memory on Apple Silicon, <code>n_gpu_layers=-1<\/code> is almost always the right setting there.<\/p>\n<h3>Option 4: Windows with CUDA<\/h3>\n<p>Install Visual Studio Build Tools 2022 with the &#8220;Desktop development with C++&#8221; workload <em>first<\/em>, then the CUDA Toolkit, so CUDA installs its MSBuild integration into an existing Visual Studio. Then, in PowerShell:<\/p>\n<pre><code>$env:CMAKE_ARGS = \"-DGGML_CUDA=on\"\npip install llama-cpp-python --no-cache-dir --force-reinstall --upgrade<\/code><\/pre>\n<p>In <code>cmd.exe<\/code> the equivalent is <code>set CMAKE_ARGS=-DGGML_CUDA=on<\/code> on its own line. Windows source builds are the most failure-prone path of the three platforms; if you only want inference and not a custom build, the prebuilt CUDA wheels or WSL2 are both less painful.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"How_to_tell_which_build_you_have\"><\/span>How to tell which build you have<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>The most reliable check is the verbose loader output, which is version-agnostic:<\/p>\n<pre><code>from llama_cpp import Llama\nllm = Llama(model_path=\".\/models\/model.gguf\", n_gpu_layers=-1, verbose=True)<\/code><\/pre>\n<p>On a CUDA build you will see backend initialisation lines mentioning CUDA and a device name, plus a tensor-loading line reporting how many of the model&#8217;s layers were offloaded to GPU. On Metal you will see Metal device lines instead. A CPU-only build prints neither and reports zero offloaded layers. Cross-check with <code>nvidia-smi<\/code> during generation: if your Python process is not holding VRAM, nothing is running on the GPU.<\/p>\n<p>Recent versions also expose a direct capability check:<\/p>\n<pre><code>from llama_cpp import llama_cpp, __version__\nprint(__version__)\nprint(llama_cpp.llama_supports_gpu_offload())<\/code><\/pre>\n<p>If that attribute raises <code>AttributeError<\/code>, your build predates it \u2014 fall back to the verbose-log method.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Loading_a_GGUF_and_running_a_first_completion\"><\/span>Loading a GGUF and running a first completion<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Point <code>model_path<\/code> at any GGUF file. If you would rather pull from Hugging Face, <code>Llama.from_pretrained(repo_id=..., filename=\"*Q4_K_M.gguf\", ...)<\/code> does the download for you when <code>huggingface-hub<\/code> is installed.<\/p>\n<pre><code>from llama_cpp import Llama\n\nllm = Llama(\n    model_path=\".\/models\/qwen2.5-7b-instruct-q4_k_m.gguf\",\n    n_gpu_layers=-1,\n    n_ctx=4096,\n    n_batch=512,\n    verbose=False,\n)\n\nout = llm.create_chat_completion(\n    messages=[{\"role\": \"user\", \"content\": \"Explain a KV cache in two sentences.\"}],\n    max_tokens=256,\n    temperature=0.7,\n)\nprint(out[\"choices\"][0][\"message\"][\"content\"])<\/code><\/pre>\n<p>For raw text continuation, call the object directly: <code>llm(\"Q: What is a GGUF file? A:\", max_tokens=128, stop=[\"Q:\"])<\/code> and read <code>out[\"choices\"][0][\"text\"]<\/code>.<\/p>\n<h3>Streaming tokens<\/h3>\n<p>Pass <code>stream=True<\/code> and iterate. The response shape mirrors the OpenAI streaming format, so the first chunk typically carries only the role and later chunks carry <code>content<\/code> deltas:<\/p>\n<pre><code>stream = llm.create_chat_completion(\n    messages=[{\"role\": \"user\", \"content\": \"Write a haiku about GGUF files.\"}],\n    stream=True,\n)\nfor chunk in stream:\n    delta = chunk[\"choices\"][0][\"delta\"]\n    if \"content\" in delta:\n        print(delta[\"content\"], end=\"\", flush=True)<\/code><\/pre>\n<p>Most GGUF files embed a chat template that llama-cpp-python applies automatically. When output looks garbled or the model never stops, the template is the first suspect \u2014 override it with the <code>chat_format<\/code> argument. Browse quantised options and their sizes in our <a href=\"https:\/\/convly.ai\/models\/\">AI models database<\/a>.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"The_parameters_that_matter\"><\/span>The parameters that matter<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<table>\n<thead>\n<tr>\n<th>Parameter<\/th>\n<th>What it does<\/th>\n<th>Practical guidance<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td><code>n_gpu_layers<\/code><\/td>\n<td>How many transformer layers to offload to GPU. Default 0, i.e. CPU only. <code>-1<\/code> means all of them.<\/td>\n<td>Start at <code>-1<\/code>. If you hit an out-of-memory error at load, step down until it fits.<\/td>\n<\/tr>\n<tr>\n<td><code>n_ctx<\/code><\/td>\n<td>Context window in tokens. Defaults to a deliberately small value (512 in current releases). Passing <code>0<\/code> tells llama.cpp to take the value from the model&#8217;s own metadata.<\/td>\n<td>Set it explicitly. <code>0<\/code> is legal but a model trained for 128K context will try to allocate a KV cache for 128K tokens, which is usually what blows up your VRAM.<\/td>\n<\/tr>\n<tr>\n<td><code>n_batch<\/code><\/td>\n<td>Logical batch size for prompt processing (prefill), not generation.<\/td>\n<td>512 is the common default. Raising it to 1024\u20132048 speeds up long prompts on GPU at the cost of more memory; lower it if you see buffer-allocation failures.<\/td>\n<\/tr>\n<tr>\n<td><code>n_ubatch<\/code><\/td>\n<td>Physical micro-batch actually submitted to the backend.<\/td>\n<td>Leave alone unless you are memory-constrained, where a smaller value reduces peak compute-buffer size.<\/td>\n<\/tr>\n<tr>\n<td><code>n_threads<\/code><\/td>\n<td>Threads for generation. <code>n_threads_batch<\/code> covers prompt processing.<\/td>\n<td>Only matters for work still on CPU. Set to physical cores, not logical (hyperthreaded) ones.<\/td>\n<\/tr>\n<tr>\n<td><code>offload_kqv<\/code><\/td>\n<td>Whether the KV cache lives on GPU.<\/td>\n<td>On by default and normally what you want; disabling it frees VRAM but costs a lot of speed.<\/td>\n<\/tr>\n<tr>\n<td><code>use_mmap<\/code> \/ <code>use_mlock<\/code><\/td>\n<td>Memory-map the file; lock it in RAM.<\/td>\n<td>Keep mmap on. Use mlock only if the OS is paging model weights out.<\/td>\n<\/tr>\n<tr>\n<td><code>chat_format<\/code><\/td>\n<td>Overrides the embedded chat template.<\/td>\n<td>Set it when the model&#8217;s built-in template is missing or wrong.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>Note that flash attention and KV-cache quantization (<code>type_k<\/code> \/ <code>type_v<\/code>) have moved around between releases \u2014 the flash-attention switch has been a boolean in some versions and a three-way auto\/on\/off setting in others. Run <code>help(Llama)<\/code> against your installed version rather than trusting a flag name from a blog post.<\/p>\n<h3>Tuning in practice<\/h3>\n<p>The two settings that interact are <code>n_gpu_layers<\/code> and <code>n_ctx<\/code>. Weights and KV cache compete for the same VRAM, and the KV cache scales roughly linearly with context length. Halving <code>n_ctx<\/code> from 8192 to 4096 often frees enough memory to offload several more layers, which is usually the better trade. Work out the budget before you start guessing with our <a href=\"https:\/\/convly.ai\/llm-vram-calculator\/\">VRAM calculator<\/a>, or check per-model figures in the <a href=\"https:\/\/convly.ai\/vram-requirements-every-major-llm-2026\/\">VRAM requirements reference<\/a>.<\/p>\n<p>Partial offload works \u2014 it is llama.cpp&#8217;s signature feature \u2014 but expect a steep drop once any layer stays on CPU, because every token must cross the PCIe bus. If you can fit all layers, do.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"OpenAI-compatible_server_mode\"><\/span>OpenAI-compatible server mode<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<pre><code>pip install \"llama-cpp-python[server]\"\n\npython -m llama_cpp.server \n  --model .\/models\/qwen2.5-7b-instruct-q4_k_m.gguf \n  --n_gpu_layers -1 \n  --n_ctx 4096 \n  --host 0.0.0.0 --port 8000<\/code><\/pre>\n<p>Server flags mirror the constructor arguments, underscores included. You get <code>\/v1\/chat\/completions<\/code>, <code>\/v1\/completions<\/code>, <code>\/v1\/models<\/code>, and interactive docs at <code>\/docs<\/code>. Any OpenAI client works, and streaming is supported over SSE:<\/p>\n<pre><code>from openai import OpenAI\n\nclient = OpenAI(base_url=\"http:\/\/localhost:8000\/v1\", api_key=\"not-needed\")\nfor event in client.chat.completions.create(\n    model=\"gpt-3.5-turbo\",  # ignored unless you set model aliases\n    messages=[{\"role\": \"user\", \"content\": \"Hello\"}],\n    stream=True,\n):\n    print(event.choices[0].delta.content or \"\", end=\"\", flush=True)<\/code><\/pre>\n<p>For more than one model, pass <code>--config_file config.json<\/code> with a <code>models<\/code> array, giving each entry a <code>model<\/code> path, a <code>model_alias<\/code> clients can request by name, and its own <code>n_gpu_layers<\/code> \/ <code>n_ctx<\/code>. Add <code>--api_key<\/code> if the port is reachable beyond localhost. Weighing this against a hosted endpoint? The <a href=\"https:\/\/convly.ai\/self-hosting-vs-api-calculator\/\">self-hosting vs API break-even calculator<\/a> puts numbers on it.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Common_build_failures_and_fixes\"><\/span>Common build failures and fixes<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<table>\n<thead>\n<tr>\n<th>Symptom<\/th>\n<th>Cause<\/th>\n<th>Fix<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>Install succeeds but no GPU lines in verbose output<\/td>\n<td>pip reused a cached CPU wheel<\/td>\n<td>Reinstall with <code>--no-cache-dir --force-reinstall<\/code><\/td>\n<\/tr>\n<tr>\n<td><code>Failed building wheel<\/code>, CMake not found<\/td>\n<td>No build toolchain<\/td>\n<td>Linux: <code>build-essential<\/code> plus CMake. macOS: <code>xcode-select --install<\/code>. Windows: VS Build Tools C++ workload<\/td>\n<\/tr>\n<tr>\n<td>nvcc not found during configure<\/td>\n<td>Driver present, toolkit missing<\/td>\n<td>Install the CUDA Toolkit. The CUDA version in <code>nvidia-smi<\/code> is the driver&#8217;s maximum, not an installed toolkit<\/td>\n<\/tr>\n<tr>\n<td>&#8220;unsupported GNU version&#8221; from nvcc<\/td>\n<td>System gcc newer than your CUDA supports<\/td>\n<td>Point CUDA at an older compiler with <code>-DCMAKE_CUDA_HOST_COMPILER=\/usr\/bin\/gcc-12<\/code><\/td>\n<\/tr>\n<tr>\n<td>Machine freezes or OOMs while compiling<\/td>\n<td>Too many parallel compile jobs<\/td>\n<td>Set <code>CMAKE_BUILD_PARALLEL_LEVEL=4<\/code> before pip install<\/td>\n<\/tr>\n<tr>\n<td>&#8220;unknown model architecture&#8221; when loading<\/td>\n<td>GGUF newer than your llama.cpp version<\/td>\n<td>Upgrade llama-cpp-python; a rebuild is required, not just a config change<\/td>\n<\/tr>\n<tr>\n<td>CUDA out of memory at load time<\/td>\n<td>Weights plus KV cache exceed VRAM<\/td>\n<td>Lower <code>n_ctx<\/code> first, then <code>n_gpu_layers<\/code><\/td>\n<\/tr>\n<tr>\n<td>Failure allocating compute buffers<\/td>\n<td><code>n_batch<\/code> too large for available memory<\/td>\n<td>Reduce <code>n_batch<\/code> (and <code>n_ubatch<\/code>)<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<h2><span class=\"ez-toc-section\" id=\"Frequently_asked_questions\"><\/span>Frequently asked questions<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<h3>Is llama-cpp-python the same as llama.cpp?<\/h3>\n<p>No. llama.cpp is the C\/C++ inference engine; llama-cpp-python vendors a specific commit of it and wraps it for Python. Because the vendored version is pinned per release, the bindings can trail upstream by days or weeks \u2014 which matters when a brand-new model architecture has just landed in llama.cpp but not yet in a published binding release.<\/p>\n<h3>How do I confirm the GPU is really being used?<\/h3>\n<p>Load with <code>verbose=True<\/code> and look for backend initialisation lines and a report of layers offloaded to GPU. Then watch <code>nvidia-smi<\/code> (or Activity Monitor&#8217;s GPU history on macOS) during a generation. If VRAM usage does not rise and tokens per second look like CPU numbers, you are on a CPU-only build.<\/p>\n<h3>Can I avoid compiling entirely?<\/h3>\n<p>Often, yes \u2014 use the project&#8217;s prebuilt wheel indexes with <code>--extra-index-url<\/code>, matching the CUDA tag to your toolkit. When no matching wheel exists for your Python version and platform, pip falls back to a source build, which typically takes several minutes with CUDA enabled.<\/p>\n<h3>Should I use llama-cpp-python, Ollama, or LM Studio?<\/h3>\n<p>Use llama-cpp-python when you want the model inside your own Python process, with direct control over sampling, logits, and grammars. Prefer <a href=\"https:\/\/convly.ai\/what-is-ollama-complete-guide-2026\/\">Ollama<\/a> for a managed daemon with model pulls and automatic memory handling, or <a href=\"https:\/\/convly.ai\/lm-studio-complete-guide-2026\/\">LM Studio<\/a> for a GUI. All three sit on llama.cpp, so quality is comparable; the difference is ergonomics.<\/p>\n<h3>Can I run a model larger than my VRAM?<\/h3>\n<p>Yes. Set <code>n_gpu_layers<\/code> to a value below the model&#8217;s layer count and the remainder runs on CPU with system RAM. It works reliably but the speed penalty is severe once a meaningful share of layers stays on CPU, so a smaller model at a higher quantization usually beats a large model half-offloaded.<\/p>\n<h3>Does GPU support work on Windows without WSL?<\/h3>\n<p>It does. Either install a prebuilt CUDA wheel, or build from source with Visual Studio Build Tools 2022 (Desktop development with C++) installed before the CUDA Toolkit, setting <code>$env:CMAKE_ARGS<\/code> in PowerShell. WSL2 remains the smoother path if you are comfortable with it, since the Linux build instructions are better trodden.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>The plain pip install llama-cpp-python gives you a CPU-only build. GPU support requires either a prebuilt GPU wheel or a [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":2142,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"site-sidebar-layout":"default","site-content-layout":"","ast-site-content-layout":"default","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","ast-disable-related-posts":"","theme-transparent-header-meta":"","adv-header-id-meta":"","stick-header-meta":"","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"default","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"footnotes":""},"categories":[7],"tags":[],"class_list":["post-2141","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai-news"],"_links":{"self":[{"href":"https:\/\/convly.ai\/pt\/wp-json\/wp\/v2\/posts\/2141","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/convly.ai\/pt\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/convly.ai\/pt\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/convly.ai\/pt\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/convly.ai\/pt\/wp-json\/wp\/v2\/comments?post=2141"}],"version-history":[{"count":1,"href":"https:\/\/convly.ai\/pt\/wp-json\/wp\/v2\/posts\/2141\/revisions"}],"predecessor-version":[{"id":2143,"href":"https:\/\/convly.ai\/pt\/wp-json\/wp\/v2\/posts\/2141\/revisions\/2143"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/convly.ai\/pt\/wp-json\/wp\/v2\/media\/2142"}],"wp:attachment":[{"href":"https:\/\/convly.ai\/pt\/wp-json\/wp\/v2\/media?parent=2141"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/convly.ai\/pt\/wp-json\/wp\/v2\/categories?post=2141"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/convly.ai\/pt\/wp-json\/wp\/v2\/tags?post=2141"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}