Ollama GPT OSS: Running Open-Source LLMs Locally (Guide)

If you searched “ollama gpt oss”, you likely want to run open-source large language models locally using Ollama—but OpenAI has not released an open-weight model family called “GPT-OSS”. OpenAI released GPT-2 in 2019 as open-weight, but their recent models (GPT-4, GPT-4o, GPT-4.1) remain proprietary API-only products. What does exist: hundreds of open-weight models from Meta (Llama), Mistral AI, Alibaba (Qwen), and others that you can pull and run through Ollama on your own hardware. This guide covers which models work, the VRAM each size requires, how quantization affects quality, and how they compare.

What Ollama Actually Runs

Ollama is a local inference engine that downloads, quantizes, and runs open-weight LLMs on macOS, Linux, and Windows. It does not host OpenAI models. The Ollama models list includes Llama 3.1, Mistral 7B, Qwen 2.5, Gemma 2, Phi-3, and dozens more. Each model is available in multiple quantization levels (Q4_K_M, Q5_K_M, FP16) to trade VRAM for quality.

Full installation steps: how to install Ollama.

Open-Source Model Families Available in Ollama

The AI models database lists specs, VRAM requirements, and pricing for 37 models including all of the above.

Pulling and Running a Model

Once Ollama is installed, pull a model with ollama pull <model>:<tag>. The tag specifies parameter count and quantization. Examples:

ollama pull llama3.1:8b
ollama pull mistral:7b-instruct-q4_K_M
ollama pull qwen2.5:7b
ollama pull gemma2:9b

Run the model:

ollama run llama3.1:8b

This opens an interactive chat session. Type your prompt, press Enter, and the model generates a response locally. To exit, type /bye.

To use the model programmatically via the API:

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.1:8b",
  "prompt": "Explain recursion in one sentence."
}'

Ollama exposes an OpenAI-compatible /v1/chat/completions endpoint, so you can point existing OpenAI SDK code at http://localhost:11434 and swap in a local model.

Parameter Sizes and VRAM Requirements

Model size (7B, 13B, 70B) determines quality and VRAM footprint. Larger models reason better but need more GPU memory. Quantization (Q4, Q5, FP16) compresses weights to reduce VRAM at a small quality cost.

Use the VRAM calculator to estimate memory needs for any model and quantization level. The VRAM requirements guide lists precise values for every major model.

For GPU buying decisions, see best GPUs for running LLMs locally.

Quantization: Quality vs VRAM Trade-Off

Quantization reduces model weight precision from 16-bit floats (FP16) to 4-bit or 5-bit integers (Q4, Q5). This cuts VRAM by 60-75% with minimal quality loss for most tasks.

• Q4_K_M: 4-bit quantization. Smallest VRAM, slight quality drop on complex reasoning. Good for chatbots, summarization, code completion.
• Q5_K_M: 5-bit quantization. ~20% more VRAM than Q4, closer to FP16 quality. Best balance for most users.
• Q8_0: 8-bit quantization. Nearly FP16 quality, 50% VRAM reduction. Use if you have the VRAM headroom.
• FP16: Full precision. Best quality, double the VRAM of Q4. Only needed for research or if quality regressions are unacceptable.

Example: Llama 3.1 8B at Q4_K_M uses 4.5 GB and scores 67.2 on MMLU. At FP16 it uses 14 GB and scores 68.1 on MMLU. For most tasks the 0.9-point difference is imperceptible.

To pull a specific quantization:

ollama pull llama3.1:8b-instruct-q4_K_M
ollama pull llama3.1:8b-instruct-fp16

If you omit the quantization tag, Ollama defaults to Q4_K_M.

Performance Comparison: 7B Class

The 7B-8B size class is the most popular for local use. It fits consumer GPUs and delivers strong results on coding, reasoning, and chat.

Qwen 2.5 7B and Gemma 2 9B lead on MMLU (general knowledge). Llama 3.1 8B leads on HumanEval (coding). Mistral 7B is the fastest and most permissive license (Apache 2.0). The LLM leaderboard ranks all models by intelligence, price, and context length.

Which Model to Pick

• Best overall quality (7B class): ollama pull qwen2.5:7b or ollama pull llama3.1:8b
• Best for coding: ollama pull llama3.1:8b or ollama pull qwen2.5-coder:7b
• Fastest inference: ollama pull mistral:7b or ollama pull gemma2:2b (for CPU/low VRAM)
• Multilingual: ollama pull qwen2.5:7b (supports 29 languages)
• Long documents (128k+ context): ollama pull llama3.1:8b or ollama pull qwen2.5:7b
• Strongest reasoning (if you have 40+ GB VRAM): ollama pull llama3.1:70b or ollama pull qwen2.5:72b

The best local LLMs for Ollama guide benchmarks all models and recommends specific tags per use case.

When to Self-Host vs Use an API

Running Ollama locally makes sense if:

• You already own a GPU with 12+ GB VRAM (RTX 3060, 4060 Ti, or better)
• You process sensitive data that cannot leave your network
• You generate more than 5 million tokens per month (API costs exceed self-hosting TCO)
• You need guaranteed uptime and no rate limits

Use a hosted API (OpenAI, Anthropic, Groq) if:

• You generate fewer than 1 million tokens/month (self-hosting GPU amortization takes years)
• You need the absolute best quality (Claude 3.5 Sonnet, GPT-4o outperform all open models)
• You don’t want to manage inference infrastructure

The self-hosting vs API calculator estimates break-even based on your token volume, GPU cost, and electricity rate. The API cost calculator projects monthly spend per model.

Platform-Specific Notes

macOS

Ollama uses Metal for GPU acceleration on M1/M2/M3 Macs. Unified memory means VRAM = system RAM. An M2 Max with 64 GB can run Llama 3.1 70B at Q4 (40 GB) with room for the OS. Install via Homebrew:

brew install ollama

Start the service:

ollama serve

Linux

Ollama requires NVIDIA GPUs with CUDA 11.8+ or AMD GPUs with ROCm 5.7+. Install with:

curl -fsSL https://ollama.com/install.sh | sh

This installs the binary to /usr/local/bin/ollama and creates a systemd service. Start it:

sudo systemctl start ollama

Models download to ~/.ollama/models. To change the location, set OLLAMA_MODELS=/path/to/models in /etc/systemd/system/ollama.service.

Windows

Ollama for Windows requires NVIDIA GPUs with CUDA 11.8+ and driver 520+. Download the installer from ollama.com and run it. The service starts automatically. Models download to C:Users<YourName>.ollamamodels.

To run from PowerShell:

ollama run llama3.1:8b

Frequently Asked Questions

Does OpenAI offer an open-source model I can run in Ollama?

No. OpenAI released GPT-2 (2019) as open-weight, but all recent models (GPT-3.5, GPT-4, GPT-4o, o1) are proprietary and API-only. If you want OpenAI-quality reasoning locally, try Llama 3.1 70B or Qwen 2.5 72B—both outperform GPT-3.5 on most benchmarks and run in Ollama with 40 GB VRAM at Q4 quantization.

Can I run Ollama on a CPU without a GPU?

Yes, but inference is 10-50× slower. Small models (Gemma 2 2B, Qwen 2.5 0.5B, Phi-3.5 mini 3.8B) are usable on modern CPUs (16+ threads). Larger models (7B+) will generate 1-3 tokens per second on CPU, which is too slow for interactive use. If you don’t have a GPU, consider using a hosted API instead—see the self-hosting vs API calculator.

How much does it cost to run Ollama compared to OpenAI API?

OpenAI charges $0.15 per million input tokens and $0.60 per million output tokens for GPT-4o mini. A 12 GB GPU (RTX 4060 Ti, $400) running Llama 3.1 8B costs $0.05/hour in electricity (at $0.12/kWh, 400W system draw). Break-even is around 3-5 million tokens/month. The API cost calculator and self-hosting calculator show exact TCO for your usage.

What is the difference between Q4_K_M, Q5_K_M, and FP16?

Q4_K_M uses 4-bit quantization (smallest VRAM, slight quality loss). Q5_K_M uses 5-bit quantization (20% more VRAM, closer to full quality). FP16 is full 16-bit precision (best quality, double the VRAM of Q4). For most tasks, Q5_K_M is the best balance. Use FP16 only if you have spare VRAM and need the last 1-2% of quality for research or production.

Can I fine-tune models in Ollama?

No. Ollama is an inference engine, not a training framework. To fine-tune an open model, use Hugging Face Transformers with PEFT/LoRA, Axolotl, or LLaMA Factory. Export the fine-tuned weights to GGUF format and import them into Ollama with ollama create. The Ollama complete guide covers this workflow.

Which GPU should I buy to run 7B models locally?

For Q4 quantization (4.5 GB VRAM): RTX 3060 12 GB ($250 used) or RTX 4060 Ti 16 GB ($450 new). For FP16 (14 GB VRAM): RTX 4060 Ti 16 GB or RTX 4070 ($550-600). For 70B models at Q4 (40 GB): dual RTX 3090 24 GB ($1800 used) or A100 40 GB ($6000+ used). The best GPUs guide benchmarks price/performance for every size class.