AI Hardware, GPUs and Local LLMs — Page 6

DeepSite is a free Hugging Face Space athuggingface.co/spaces/enricoros/deepsite that turns a text prompt into a complete, self-contained HTML web app. No installation, no account required to try — open the URL, describe your app, click Generate, get a live preview and a downloadable .html file. The output is a single file with inline CSS and JavaScript; host it on Hugging Face Spaces, GitHub Pages, Netlify, or any static server. Duplicate the Space and add your own Hugging Face API token to avoid queue times and pick your preferred model. Hugging Face DeepSite is an open-source, browser-based web app generator.

Run ollama ps while a model is loaded — the PROCESSOR column tells you whether Ollama is using GPU or CPU.The most common fix on NVIDIA is installing or updating the host driver so nvidia-smi sees the card, then restarting the Ollama service. If the model is larger than your VRAM, Ollama offloads layers to CPU — use the VRAM calculator to check whether your model fits before downloading it. AMD, Docker, and WSL2 each require platform-specific steps covered below.

llamafile packages a GGUF model and the llama.cpp inference engine into one executable file that runs on Linux, macOS, Windows, FreeBSD, and more — no installation needed. Run ./model.llamafile and a browser chat UI opens automatically; an OpenAI-compatible API is served at http://localhost:8080/v1.Files over 4 GB cannot run directly on Windows — use a smaller quantization or run the runtime and GGUF separately. Best for air-gapped machines, USB deployment, and one-file sharing.

The Hugging Face Inference API — now officially called Inference Providers — routes requests to Groq, Together AI, Fireworks, Cerebras, and others through a single HF token at https://router.huggingface.co/v1.Free tier: $0.10/month in credits for free accounts,$2.00/month for PRO users.

text-generation-webui (widely called oobabooga after its GitHub author) is a free, open-source, browser-based interface for running LLMs locally on your own hardware. Install via one-click scripts — start_windows.bat, start_linux.sh, or start_macos.sh — no manual Python environment setup required. Supports multiple backends: llama.cpp for GGUF files, ExLlamaV2 for EXL2/GPTQ on NVIDIA, and Transformers for HuggingFace models. Includes an OpenAI-compatible API extension (–extensions openai) so other apps can connect to your local model without code changes.

Pull vllm/vllm-openai:latest and run it with –runtime nvidia –gpus all –ipc=host to get a GPU-backed OpenAI-compatible server. Mount ~/.cache/huggingface into the container so model weights survive container restarts. The server exposes an OpenAI-compatible API on port 8000; test it with curl http://localhost:8000/v1/models.The three most important tuning flags are –tensor-parallel-size, –max-model-len, and –gpu-memory-utilization.

LoRA fine tuning trains a tiny set of adapter weights instead of the full model — typically 1–5% of total parameters — so you can fine-tune a 7B model on a single consumer GPU.QLoRA adds 4-bit quantisation to the frozen base model, cutting VRAM further: a 7B model fits in ~6 GB, a 13B in ~10 GB.Rank (r) and alpha are the two knobs that control how much the adapter can change the model’s behaviour.