AI Hardware, GPUs and Local LLMs — Page 12

text-generation-webui (widely called oobabooga after its GitHub author) is a free, open-source, browser-based interface for running LLMs locally on your own hardware. Install via one-click scripts — start_windows.bat, start_linux.sh, or start_macos.sh — no manual Python environment setup required. Supports multiple backends: llama.cpp for GGUF files, ExLlamaV2 for EXL2/GPTQ on NVIDIA, and Transformers for HuggingFace models. Includes an OpenAI-compatible API extension (–extensions openai) so other apps can connect to your local model without code changes.

Pull vllm/vllm-openai:latest and run it with –runtime nvidia –gpus all –ipc=host to get a GPU-backed OpenAI-compatible server. Mount ~/.cache/huggingface into the container so model weights survive container restarts. The server exposes an OpenAI-compatible API on port 8000; test it with curl http://localhost:8000/v1/models.The three most important tuning flags are –tensor-parallel-size, –max-model-len, and –gpu-memory-utilization.

LoRA fine tuning trains a tiny set of adapter weights instead of the full model — typically 1–5% of total parameters — so you can fine-tune a 7B model on a single consumer GPU.QLoRA adds 4-bit quantisation to the frozen base model, cutting VRAM further: a 7B model fits in ~6 GB, a 13B in ~10 GB.Rank (r) and alpha are the two knobs that control how much the adapter can change the model’s behaviour.

TL;DROllama Cloud refers to running Ollama on cloud infrastructure (AWS, GCP, Azure) rather than local hardware—same CLI and API, remote execution. All models in the Ollama library work on cloud instances; you pay hourly for GPU compute instead of buying hardware. Break-even point varies by usage: the self-hosting vs API calculator shows when cloud GPUs beat local hardware purchases. Privacy trade-off: cloud hosting means your prompts and responses transit the network and touch provider infrastructure, unlike fully local inference. Ollama cloud deployments run the same Ollama server you’d install locally, but on rented GPU instances from AWS, Google Cloud, Azure, or other providers.

Jan is a free, open-source desktop app (AGPL license) that runs LLMs entirely on your own hardware — no account, no cloud, no data leaving your machine. It ships a chat interface, a model hub for downloading GGUF models, and an OpenAI-compatible local API server (default port 1337).Download from jan.ai or the GitHub Releases page — builds for Windows, macOS (Apple Silicon and Intel), and Linux. Best for privacy-focused users who want a full GUI experience; developers wanting a headless API-first workflow may prefer Ollama instead.

KoboldCpp is a single executable — download it, point it at a GGUF model file, and a browser UI plus OpenAI-compatible API start immediately on port 5001. GPU offload is controlled by –gpulayers N; start with 999 to try full offload and reduce if you hit out-of-memory errors. Use it when you want a built-in story/chat UI or need KoboldAI-compatible endpoints; use Ollama if you prefer a managed model library and CLI-first workflow. No install step, no package manager, no daemon — just a single binary and a GGUF file. KoboldCpp is a single-file local LLM runtime built on top of llama.cpp.

GGUF quantisation shrinks large diffusion models like FLUX.1 from ~24 GB to 5–12 GB, letting them run on consumer GPUs with 6–16 GB VRAM.Install the ComfyUI-GGUF custom node by city96, place .gguf files in ComfyUI/models/unet/, and use the UnetLoaderGGUF node instead of the standard UNETLoader. Q4_K_S or Q5_K_S offer the best quality-to-VRAM ratio for most cards.

The plain pip install llama-cpp-python gives you a CPU-only build.

Windows & macOS: the desktop app downloads updates itself — click the Ollama icon in the system tray or menu bar and choose Restart to update. Linux: re-run the install script: curl -fsSL https://ollama.com/install.sh | sh.