- Ollama lets you run AI models like Llama, Mistral, and Gemma locally via a single command-line tool
- Install in seconds:
curl https://ollama.com/install.sh | sh(Linux/macOS) or download the Windows installer - Entry-level models need 4-8 GB VRAM; production-grade 70B models need ~40 GB VRAM at 4-bit quantization
- Break-even vs cloud APIs: ~500K tokens/month for a 70B model, less for smaller models
Ollama is a command-line tool that packages AI language models into containers you can run on your own hardware. Instead of sending prompts to OpenAI, Anthropic, or Google and paying per token, you download a model once—Llama 3.3 70B, Mistral 7B, Phi-4, or any of dozens of supported models—and inference runs entirely on your GPU or CPU. You keep your data local, pay nothing per request after the initial hardware cost, and retain full control over model behavior and uptime.
The tradeoff is hardware: you need enough VRAM to hold the model. A 7B parameter model quantized to 4-bit requires around 4-5 GB of GPU memory, which fits on a consumer RTX 4060. A 70B model needs approximately 40 GB at 4-bit, requiring a workstation card like the RTX 6000 Ada or multi-GPU setup. The VRAM calculator shows exact requirements for any model size and quantization level.
Installing Ollama
Linux
Run the official install script, which detects your distribution and sets up the ollama service:
curl -fsSL https://ollama.com/install.sh | sh
This installs the binary to /usr/local/bin/ollama and registers a systemd service. The service starts automatically and survives reboots. To verify:
ollama --version
If you’re behind a corporate proxy or need manual installation, download the binary directly from GitHub releases and place it in your PATH.
macOS
Download the .dmg installer from ollama.com/download, open it, and drag Ollama to Applications. The menu bar app starts the local server on localhost:11434. To use Ollama from Terminal:
ollama --version
macOS users on Apple Silicon (M1/M2/M3/M4) can run models using unified memory instead of discrete VRAM. A Mac Studio with 192 GB unified memory can serve a Llama 4 Maverick (240 GB at 4-bit) by swapping to disk, though inference speed drops significantly when exceeding physical RAM.
Windows
Download the OllamaSetup.exe installer from ollama.com/download and run it. The installer adds ollama.exe to your PATH and starts the background service. Open PowerShell or Command Prompt and verify:
ollama --version
Windows Subsystem for Linux (WSL2) with GPU passthrough is supported: install Ollama inside your WSL2 distribution following the Linux instructions, and ensure you have NVIDIA CUDA drivers installed on the Windows host.
Running Your First Model
Pull and run Llama 3.1 8B, a capable instruction-following model that fits in 5 GB of VRAM:
ollama run llama3.1:8b
Ollama downloads the model (approximately 4.7 GB) to ~/.ollama/models (Linux/macOS) or %USERPROFILE%.ollamamodels (Windows), loads it into memory, and drops you into an interactive prompt. Type a message, press Enter, and the model generates a response locally. Press Ctrl+D or type /bye to exit.
To run a model in the background and query it via API:
ollama serve
This starts a server on http://localhost:11434. In another terminal:
curl http://localhost:11434/api/generate -d '{
"model": "llama3.1:8b",
"prompt": "Explain TCP congestion control",
"stream": false
}'
The response comes back as JSON with the full completion in the response field.
Model Selection and VRAM Requirements
Ollama supports open-weight models from Meta, Mistral AI, Microsoft, Google, Alibaba, and others. The table below shows popular choices and their memory footprint at 4-bit quantization, taken from the VRAM requirements database:
| Model | Parameters | Context Length | VRAM at 4-bit | Use Case |
|---|---|---|---|---|
| Llama 3.1 8B | 8B | 128K | ~5 GB | General instruction-following, fits consumer GPUs |
| Mistral 7B | 7B | 32K | ~4.5 GB | Fast inference, good code generation |
| Phi-4 | 14B | 16K | ~9 GB | Reasoning and math, efficient for size |
| Mistral NeMo 12B | 12B | 128K | ~7.5 GB | Long-context tasks, multilingual |
| Qwen3 8B | 8B | 128K | ~5 GB | Strong multilingual, competitive with larger models |
| Gemma 3 4B | 4B | 128K | ~3 GB | Smallest viable model, runs on integrated GPUs |
| Llama 3.3 70B | 70B | 128K | ~40 GB | Production-grade reasoning, matches GPT-4 class |
| DeepSeek R1 Distill Llama 70B | 70B | 128K | ~40 GB | Distilled reasoning model, strong STEM performance |
To list all available models on your system:
ollama list
To delete a model and free disk space:
ollama rm llama3.1:8b
Model tags follow the format name:size or name:version. Omitting the tag defaults to :latest. See the full catalog at best local LLMs for Ollama.
Common Commands and Configuration
Run a model with custom parameters:
ollama run llama3.1:8b --temperature 0.7 --top-p 0.9
Pass a prompt directly without entering interactive mode:
ollama run llama3.1:8b "Write a Python function to parse ISO 8601 dates"
Load a model into memory without prompting (useful for warming up before serving requests):
ollama pull llama3.1:8b
Check which models are currently loaded in VRAM:
ollama ps
Set the number of GPU layers to offload (useful for partial GPU offloading when VRAM is limited):
OLLAMA_NUM_GPU=35 ollama run llama3.3:70b
By default, Ollama offloads all layers to GPU. Reducing OLLAMA_NUM_GPU keeps some layers on CPU, trading speed for lower VRAM usage. A 70B model with 20 layers on GPU might need only 20 GB VRAM but run 3-5× slower.
To change the model storage directory, set OLLAMA_MODELS before running:
export OLLAMA_MODELS=/mnt/nvme/ollama_models ollama pull llama3.1:8b
Ollama uses memory-mapped files, so models load faster from NVMe than SATA SSDs. Expect 10-15 seconds to load an 8B model on a modern system, 60-90 seconds for a 70B model.
Hardware Requirements
The limiting factor is VRAM, not compute. A 4-bit quantized 70B model needs 40 GB of GPU memory but runs adequately on previous-generation architectures. An RTX 3090 (24 GB) can serve two 8B models or one 30B model. An RTX 4090 (24 GB) handles the same with 30-40% higher throughput due to Ada Lovelace’s improved tensor cores.
Recommended GPUs by budget:
- Entry ($300-500): RTX 4060 Ti 16 GB handles 8B and 12B models
- Prosumer ($1000-1500): RTX 4090 or A4000 Ada runs 30B models comfortably
- Workstation ($4000-7000): RTX 6000 Ada (48 GB) or dual RTX 4090s for 70B models
- Multi-model serving ($10K+): A100 80GB or H100 80GB for running multiple 70B instances
See the GPU buying guide for detailed comparisons. Apple Silicon users benefit from unified memory: an M2 Ultra with 192 GB can run models that would require a $25K NVIDIA setup, though token throughput is 2-3× lower than an A100.
Cost Comparison: Local vs API
Cloud APIs charge per token. Claude Sonnet 5 costs $2.00 per million input tokens and $10.00 per million output tokens. A typical coding assistant session generates 500K tokens per month (250K in, 250K out), costing $3,000 annually.
A Llama 3.3 70B model self-hosted on a $6,000 workstation (RTX 6000 Ada) has zero marginal cost after hardware. The break-even point is approximately 500K tokens per month. Below that, APIs are cheaper; above it, local inference wins. Use the self-hosting calculator to model your specific workload.
Smaller models break even faster. An 8B model on a $500 RTX 4060 Ti pays for itself in 3-4 months compared to cloud APIs at moderate usage (100K tokens/month). The upside is privacy: your code, documents, and internal data never leave your network.
Frequently Asked Questions
Can Ollama run models on CPU only?
Yes, but inference is 10-50× slower depending on model size. An 8B model generates 1-3 tokens per second on a modern Ryzen or Intel CPU, compared to 40-80 tokens/sec on an RTX 4090. Set OLLAMA_NUM_GPU=0 to force CPU-only mode. Practical for batch processing or low-traffic use, unusable for interactive chat.
How does quantization affect quality?
4-bit quantization reduces model size by 75% with minimal quality loss—benchmarks show 1-3% accuracy degradation on MMLU and HumanEval compared to FP16. 3-bit (Q3) quantization cuts size further but degrades reasoning and instruction-following noticeably. Ollama defaults to Q4_0, which balances quality and VRAM. You can pull 8-bit or FP16 versions by specifying tags like llama3.1:8b-q8_0, doubling VRAM requirements.
Can I fine-tune models with Ollama?
No. Ollama is an inference runtime, not a training framework. To fine-tune a model, use tools like Hugging Face Transformers, Axolotl, or LLaMA Factory, export the result in GGUF format, and import it into Ollama via a Modelfile. The process is documented in Ollama’s GitHub repository under docs/import.md.
What is the Ollama API and how do I use it?
Ollama exposes an OpenAI-compatible REST API on localhost:11434. The /api/generate endpoint handles completions, and /api/chat supports multi-turn conversations with message history. You can integrate it into existing codebases by swapping the base URL: instead of https://api.openai.com/v1, point your client to http://localhost:11434. Many frameworks like LangChain and LlamaIndex have native Ollama support.
How many requests per second can Ollama handle?
A single loaded model serves one request at a time. Throughput depends on model size and hardware: an RTX 4090 generates 60-80 tokens/sec for an 8B model, 15-25 tokens/sec for a 70B model. To handle concurrent users, either scale horizontally (multiple machines) or use batching at the application layer. Ollama does not have built-in request queuing; you need a reverse proxy like NGINX or a load balancer.
Can I run multiple models simultaneously?
Yes, if you have sufficient VRAM. Load a second model with ollama run in a new terminal while the first is running. Ollama shares the GPU across models. Two 8B models (10 GB total) run comfortably on a 24 GB RTX 4090. Switching between models is near-instant if both are already loaded; otherwise, expect load times of 10-90 seconds depending on model size.
