ComfyUI GGUF: Run Large Diffusion Models on Low-VRAM GPUs

GGUF is a binary format developed by the llama.cpp project for storing quantised neural network weights. Originally built for LLMs, it was adapted for diffusion model UNets, and the ComfyUI-GGUF custom node by city96 brings that support into ComfyUI. The practical result: FLUX.1-dev, which requires roughly 24 GB of VRAM in full BF16 precision, becomes runnable on a 6–8 GB GPU at Q4 quantisation — with image quality that is often indistinguishable from the full-precision model to the naked eye.

Why GGUF Matters for Image Generation

Standard diffusion checkpoints ship as .safetensors files in FP16 or BF16. For older-generation models (SD1.5 at ~2 GB, SDXL at ~7 GB), that is manageable on mid-range hardware. For FLUX.1 and SD3, full-precision files are 24 GB and 16 GB respectively — beyond the VRAM ceiling of any single consumer GPU.

GGUF quantisation maps each weight to a smaller integer representation. A Q8_0 weight uses 8 bits instead of 16, roughly halving model size with negligible quality loss. Q4 variants use 4 bits per weight, cutting size to roughly one-quarter of fp16. The savings stack up dramatically on models with billions of parameters. Use the VRAM calculator to estimate how much GPU memory a specific quantisation level will need before you download anything.

FLUX.1 also has a large T5-XXL text encoder (~9.3 GB in fp16). Combining a GGUF UNet with a GGUF T5-XXL encoder makes the full FLUX.1 pipeline fit on cards that would otherwise be completely unable to run it. Both components can be loaded in GGUF format independently.

Installing the ComfyUI-GGUF Custom Node

Via ComfyUI Manager (Recommended)

1. Open ComfyUI and click Manager in the sidebar.
2. Go to Install Custom Nodes.
3. Search for ComfyUI-GGUF.
4. Click Install next to the entry by city96, then restart ComfyUI.

Manual Installation

Clone the repository into your custom_nodes directory, then install its Python dependency:

cd ComfyUI/custom_nodes
git clone https://github.com/city96/ComfyUI-GGUF

Linux / macOS (venv):

source ComfyUI/venv/bin/activate
pip install -r ComfyUI/custom_nodes/ComfyUI-GGUF/requirements.txt

Windows (portable ComfyUI):

ComfyUIpython_embedspython.exe -m pip install -r ComfyUIcustom_nodesComfyUI-GGUFrequirements.txt

Windows (venv):

ComfyUIvenvScriptsactivate.bat
pip install -r ComfyUIcustom_nodesComfyUI-GGUFrequirements.txt

The main dependency is the gguf Python package. Restart ComfyUI after installation.

Getting GGUF Model Files

city96 uploads GGUF versions of FLUX.1-dev and FLUX.1-schnell to Hugging Face under the username city96. Search for city96 FLUX.1-dev-gguf on Hugging Face to find the repository. Each repository contains multiple .gguf files, one per quantisation level. Download the single file matching the quantisation level you want — you do not need all of them.

GGUF versions of the T5-XXL text encoder are also available on Hugging Face (search t5-v1_1-xxl-encoder-gguf). The CLIP-L encoder is small enough (~240 MB) that quantising it is rarely worth the effort.

Community members also upload GGUF variants of fine-tuned FLUX models on CivitAI. The same node and directory structure applies regardless of where you download the file from.

Where to Place Model Files

The VAE for FLUX is not quantised and stays in its normal location. Only the UNet and text encoder benefit from GGUF loading.

Using the GGUF Loader Nodes

After installing ComfyUI-GGUF and restarting, new nodes appear in the node browser. These replace their standard equivalents in your workflow — you cannot load a .gguf file with the standard UNETLoader.

For a standard FLUX workflow: swap UNETLoader for UnetLoaderGGUF, and swap DualCLIPLoader for DualCLIPLoaderGGUF if you are also using a GGUF T5-XXL. Connect the outputs to the same downstream nodes as before — the tensor shapes are compatible.

Choosing a Quantisation Level

The table below shows approximate figures for FLUX.1 UNet weights only. Your total VRAM budget must also cover the VAE (~350 MB), text encoders (240 MB for CLIP-L, plus whatever you allocate for T5-XXL), and working tensors during generation. Use the VRAM requirements reference alongside this table, and consult the GPU guide if you are deciding which card to buy.

Practical starting points: Q5_K_S or Q4_K_S for 8–12 GB cards (best quality-to-VRAM ratio). Q3_K_S or Q2_K for 6 GB cards if you are also using a quantised T5-XXL. Q8_0 for 16 GB cards where you want maximum fidelity while still fitting the model entirely in VRAM.

Quality and Speed Trade-offs

At Q8_0, differences from BF16 are virtually undetectable in side-by-side comparisons. At Q4_K_S, fine textures and text rendering may show very subtle degradation on close inspection, but the outputs are production-quality for most use cases. Q2_K introduces visible softness and occasional artefacts, particularly in faces and fine detail — it is a last resort for extremely memory-constrained hardware.

Generation speed with GGUF can be comparable to or faster than loading a BF16 model with CPU offloading, because quantised weights reduce memory bandwidth pressure. However, if your GPU can hold the full BF16 model entirely in VRAM without offloading, that will generally be faster than GGUF inference. The speed advantage of GGUF is most pronounced when the alternative involves swapping layers between VRAM and system RAM.

On Apple Silicon (macOS), the MPS backend supports GGUF inference via ComfyUI-GGUF, but performance characteristics differ from NVIDIA CUDA. Results are functional but generation speed may be slower than on a comparable NVIDIA GPU. Apple Silicon’s unified memory architecture means the practical VRAM ceiling is higher than discrete GPUs of the same nominal capacity, so you may be able to run higher quantisation levels than the table above suggests.

On AMD GPUs (Linux, ROCm), GGUF inference works via PyTorch’s ROCm backend. Install the ROCm-compatible PyTorch build for ComfyUI before installing ComfyUI-GGUF. Performance and compatibility vary by GPU generation; RDNA 3 cards (RX 7000 series) have the most reliable support.

Frequently Asked Questions

Can I load a .gguf file with the standard UNETLoader node?

No. The standard UNETLoader only accepts .safetensors files. You must install the ComfyUI-GGUF custom node and use UnetLoaderGGUF to load .gguf diffusion models.

Do I need to quantise my own models, or can I download pre-quantised files?

For FLUX.1-dev and FLUX.1-schnell, city96 provides pre-quantised GGUF files on Hugging Face covering every major quantisation level. Download the specific .gguf file for the level you want — you do not need to run any quantisation tools yourself.

Does GGUF work for SD1.5 and SDXL models?

Technically yes, but the benefit is small. SD1.5 is ~2 GB and SDXL is ~7 GB in fp16 — both already fit comfortably on mid-range consumer GPUs. GGUF quantisation is most valuable for very large models (FLUX.1, SD3) where full-precision weights exceed most cards’ VRAM capacity.

Is a GGUF T5-XXL noticeably worse than the full fp16 version?

At Q8_0 or Q5_K_S, prompt following and text encoding quality is effectively identical to fp16. Lower quantisation levels (Q2_K, Q3_K_S) can occasionally produce slightly degraded prompt adherence on complex prompts, but this is subtle compared to the UNet quality impact at the same quantisation level.

My workflow was built for a full-precision FLUX model. Do I need to rebuild it?

No. Replace UNETLoader with UnetLoaderGGUF and (optionally) DualCLIPLoader with DualCLIPLoaderGGUF. All downstream connections — KSampler, VAE decode, conditioning — remain unchanged. The node outputs are tensor-compatible with the rest of a standard FLUX workflow.

Why does ComfyUI still run out of VRAM even with a quantised model?

The GGUF UNet is only part of total VRAM usage. A full FLUX pipeline also loads CLIP-L (~240 MB), T5-XXL (up to ~9.3 GB in fp16), and the VAE (~350 MB), plus working buffers during generation. If you are still running out, load the T5-XXL as GGUF too via DualCLIPLoaderGGUF, and consider enabling ComfyUI’s built-in VAE tiling for the decode step.