VRAM Requirements for Every Major LLM in 2026 (Quantization Cheat Sheet)

The most common question we get from local-LLM newcomers in 2026 isn’t “which model should I use” — it’s “will this model run on my GPU?”

This guide is the answer. We’ve tested every major open LLM at every common quantization on hardware ranging from a 12 GB RTX 3060 to an 80 GB H100, and what follows is the cheat sheet we wish existed when we started.

A reminder for the impatient: VRAM is the binding constraint. If your model + KV cache + context doesn’t fit in VRAM, inference falls off a cliff. Everything below assumes you want pure GPU inference; if you’re willing to do CPU offload, divide the throughput by 5–10×.

The quick-reference table

Every major 2026 open LLM and its VRAM needs at common quantization levels. Numbers are for the model weights only, at 8 K context. Add 1–2 GB for KV cache headroom per 8 K of context you actually use.

A practical note: for daily use, Q4_K_M is the recommended balance of size and quality. The quality drop versus FP16 is small (typical perplexity increase < 2%) and the memory savings are enormous (~3.3× smaller). Q5_K_M is marginally better quality at ~17% more memory. Q3 and IQ2 are emergency-only — quality degrades noticeably.

KV cache memory — the part everyone forgets

The numbers above are model weights only. The KV cache — the running memory of all tokens in your conversation — also lives in VRAM and grows linearly with context length.

Rough KV cache size, per 1 K tokens of context, at FP16:

Quantizing the KV cache (an option in llama.cpp and vLLM in 2026) cuts this by ~2–4× with a small quality cost. Most production setups now use Q8 KV cache — it’s nearly free quality-wise and saves substantial VRAM at long context.

If you plan to use 32 K or longer context, add KV cache to your VRAM math before picking a GPU.

GPU compatibility matrix

Which models comfortably fit on each common GPU, at recommended quants, with 8 K context? “Comfortably” means model + KV cache + 1 GB system headroom.

The patterns to internalize:

1. 12 GB is the entry floor. Below it, you’re constrained to tiny models that don’t justify a dedicated GPU.
2. 24 GB is the inflection point. It’s the cheapest tier where Llama 3 70B becomes possible (at compromised quants).
3. 32 GB unlocks 70B properly. This is the entire reason to choose the RTX 5090 over the 4090.
4. 48 GB is comfortable territory. Most things you want to do fit cleanly.
5. 128 GB unified is the consumer ceiling. Above this, you’re buying server hardware.

Choosing the right quant for your hardware

The right quantization isn’t always “the biggest one that fits.” Quality matters, and sometimes a smaller model at a better quant beats a bigger model at a worse one.

Rough quality ranking (perplexity-based, lower is better):

• FP16 / BF16 — original. Quality reference baseline.
• Q8_0 — ~0.3% perplexity increase. Essentially indistinguishable.
• Q6_K — ~0.5% increase. Indistinguishable in practice.
• Q5_K_M — ~1.0% increase. Slight quality drop, still very high quality.
• Q4_K_M — ~1.5–2.5% increase. Recommended for most users.
• Q4_K_S — ~3% increase. Noticeably worse than Q4_K_M for similar size.
• Q3_K_M — ~5–8% increase. Visibly affected output.
• Q3_K_S — ~10% increase. Use only if Q4 won’t fit.
• IQ2_XXS — ~15–25% increase. Last resort.

The general rule: prefer a smaller-parameter model at Q5_K_M over a bigger model at Q3_K_S for everyday tasks. A Qwen 32B at Q5 generally beats a Llama 3 70B at IQ2_XXS on real-world benchmarks despite the latter sounding more impressive on paper.

Exception: coding and reasoning tasks where the bigger model’s raw knowledge advantage often survives heavy quantization. For code generation specifically, even Q3_K_S of a 70B model can outperform a Q5_K_M 30B.

MoE models — the asterisk

Mixture-of-experts (MoE) models like Mixtral 8x22B et DeepSeek V3 have an asymmetry that confuses newcomers:

• VRAM needed = total parameters (because you must hold all experts)
• Compute used = active parameters per token (much less)

Mixtral 8x22B is 141 B total / 39 B active. It needs 80+ GB of VRAM to run, but the per-token speed is closer to running a 40 B dense model.

DeepSeek V3 is 236 B total / 21 B active. It needs 150 GB+ of VRAM, but token speed approaches a 20 B dense model. This is why DeepSeek V3 is “fast for its size” — you pay the VRAM tax but get the compute discount.

If your hardware can hold an MoE model, it’s often the best choice. If it can’t, the dense model in the same parameter class is what you want.

Quick-start setups by budget

For people who want a concrete answer, here are tested setups at five budget points in 2026:

The “best value tier” in 2026 remains the used RTX 3090 / 4090 — these are the only consumer GPUs where the price-per-VRAM math is favorable, and both will remain capable through at least 2028.

For the deep dive on which GPU to pick, see best GPUs for local LLMs in 2026.

FAQ

How much VRAM do I need to run Llama 3 70B locally in 2026?

Minimum 24 GB for Llama 3 70B at Q3_K_S (which is rough quality). 32 GB lets you run Q4_K_M comfortably (the recommended quant). 40+ GB is needed for Q5_K_M. With 24 GB and 8 K context, you have basically zero headroom; pushing context to 32 K requires CPU offload or a more aggressive quant.

What’s the difference between Q4_K_M and Q4_K_S?

Both are 4-bit quantizations of the same model. Q4_K_M (“medium”) uses 5 bits for some critical weight groups, making it slightly larger but noticeably better quality than Q4_K_S (“small”). For nearly identical VRAM, Q4_K_M is preferred. Q4_K_S only makes sense when you’re trying to squeeze a model into a tight VRAM budget.

Can I run an LLM that’s bigger than my VRAM?

Yes — using CPU offload, where some model layers run on the CPU using system RAM instead of GPU VRAM. The performance penalty is severe (5–10× slower), but it lets you run models that wouldn’t otherwise fit. Practical for occasional use, painful as a daily driver. Both llama.cpp and Ollama support this out of the box via the n_gpu_layers setting.

Does the KV cache really matter for VRAM planning?

Yes, especially at long context. For Llama 3 70B at 32 K context, the KV cache alone is ~5 GB. If you’re already at the edge of your VRAM, you’ll OOM the moment a conversation gets long. Plan for KV cache and consider Q8 KV-cache quantization (option in modern inference engines) to roughly halve it.

Is there a way to run Llama 3 405B at home?

Yes, but you need 200+ GB of memory at usable quants. The realistic 2026 paths: Mac Studio M4 Ultra 512 GB ($12K, slow per-token but works), 8× RTX 4090 ($13K, complex setup), Nvidia DIGITS ($3K, purpose-built), or CPU + 256 GB DDR5 RAM with mid-range GPU for partial offload ($8K, slow). See our how-to guide on running Llama 3 405B at home.

Are there any 2026 quantization formats I should know besides GGUF?

Yes — AWQ (Activation-aware Weight Quantization) and GPTQ are both still widely used, especially for vLLM and TensorRT-LLM deployments. They’re slightly better quality at the same bit count than GGUF in some cases. For consumer local-LLM use with llama.cpp/Ollama/LM Studio, GGUF remains dominant in 2026 because of its simplicity and broad tooling support.

Will Q4 quantization affect coding ability?

Less than you’d think, but yes. For straightforward code completion, Q4_K_M is essentially identical to FP16. For complex multi-step reasoning across a codebase, Q4 occasionally produces worse logic than Q5+. If you do serious coding with local models, prefer Q5_K_M and choose your hardware to support it.

Bottom line

VRAM planning for local LLMs in 2026 isn’t complicated, but it does reward precision. Pick the parameter class first (the model size that has the capability you need), then pick the smallest quant that gives acceptable quality (Q4_K_M is usually right), then add KV cache for your real context length, then size your GPU accordingly.

If you only remember three numbers, remember these:

• 12 GB runs 8 B models cleanly.
• 24 GB runs 30 B at quality quants, 70 B uncomfortably.
• 32 GB runs 70 B at quality quants.

Everything past 32 GB enters server territory, and everything below 12 GB enters phone/embedded territory. The bulk of 2026 local-LLM activity happens in the 12–32 GB range, which is exactly the consumer GPU range — by design, not coincidence.