GGUF Format · llama.cpp #24423 (Draft)

DiffusionGemma GGUF: Download & Convert

Q: Which GGUF quantization should I choose?

Q4_K_M is the recommended starting point — it fits on most 24 GB GPUs with the best quality-to-size ratio. Q4_0 for 16 GB cards. Q5_K_M for better quality if you have 20+ GB VRAM. Q8_0 if VRAM isn't a constraint (28+ GB).

PR #24423 is still in draft. GGUF conversion and inference work from the PR branch — not in mainline yet. Ollama and LM Studio are blocked until the PR lands. Here’s what works today and what’s coming.

Download GGUF (~10min) Convert Yourself (~20min)

GGUF Support Status — llama.cpp PR Branch, Ollama & LM Studio Blocked

llama.cpp PR #24423 is still in draft. The diffusion-gemma architecture is only in the PR branch — you must fetch it to convert and run GGUF files. Ollama and LM Studio build on llama.cpp mainline and cannot bundle DiffusionGemma until the PR lands and is merged.

Engine	GGUF Support	How to Use
llama.cpp	Works (PR branch)	`git fetch origin pull/24423/head` + `make`. Full guide →
Ollama	Waiting for engine update	Custom Modelfile + mainline llama.cpp. See step-by-step
LM Studio	Waiting for engine update	Needs LM Studio to bundle updated llama.cpp. Alternatives available
vLLM	Works (native)	Doesn't use GGUF — uses HF format directly

"model type diffusion_gemma not supported" means you're on an old llama.cpp version. Build from the PR branch: git fetch origin pull/24423/head:diffusion-support && git checkout diffusion-support && make -j$(nproc). llama.cpp guide has the full commands →

Pre-Converted GGUF Downloads

Community-converted GGUF files are available on HuggingFace. These work with the llama.cpp PR branch (PR #24423 in draft). The most common quantization levels:

Quant	Size	VRAM Needed	Quality
Q4_0	~14GB	16GB	Good — small quality loss
Q4_K_M	~15GB	16–18GB	Best balance — recommended
Q5_K_M	~17GB	20GB	Better quality, needs more VRAM
Q8_0	~26GB	28GB+	Near-perfect, for 3090/4090

Recommended: Start with Q4_K_M. It fits on most 24GB GPUs and offers the best quality-to-size ratio. If you have a 16GB GPU, use Q4_0.

Convert from PyTorch to GGUF

Step 1: Download the Original Model

# From HuggingFace
huggingface-cli download google/diffusiongemma-26b-A4B-it \
  --local-dir ./diffusiongemma-26b

# Or use git-lfs
git lfs install
git clone https://huggingface.co/google/diffusiongemma-26b-A4B-it

Step 2: Convert to FP16 GGUF

cd llama.cpp
python convert_hf_to_gguf.py \
  ../diffusiongemma-26b \
  --outtype f16 \
  --outfile diffusiongemma-26b-f16.gguf

Step 3: Quantize to 4-bit

./build/bin/llama-quantize \
  diffusiongemma-26b-f16.gguf \
  diffusiongemma-26b-q4_k_m.gguf \
  Q4_K_M

Step 4: Verify the GGUF

./build/bin/llama-cli \
  -m diffusiongemma-26b-q4_k_m.gguf \
  -p "Hello world" \
  -n 32

GGUF FAQ

Why does Ollama say "model type diffusion_gemma not supported"?

Ollama bundles its own copy of llama.cpp, which hasn't been updated to include PR #24423 yet (still in draft). Once Ollama releases a new version with the updated engine, it will recognize DiffusionGemma GGUF files. Until then, our Ollama guide has a workaround using a custom Modelfile + mainline llama.cpp.

Why does LM Studio fail with "model type not supported"?

LM Studio bundles its own build of llama.cpp. PR #24423 is still in draft — not in llama.cpp mainline yet. LM Studio can’t bundle until the PR lands. There’s no workaround for LM Studio specifically (you can’t swap its internal engine), but 3 alternatives are available.

Can I use the pre-converted GGUF files today?

Yes. With PR #24423 in draft, pre-converted DiffusionGemma GGUF files work with the llama.cpp PR branch — just git fetch origin pull/24423/head:diffusion-support && git checkout diffusion-support and make. These same GGUF files will also work with Ollama and LM Studio once the PR lands and they update their bundled llama.cpp engines. The unsloth GGUF on HuggingFace has 107k+ downloads.

Which GGUF quantization should I choose?

Q4_K_M is the recommended starting point — it fits on most 24 GB GPUs with the best quality-to-size ratio. Q4_0 for 16 GB cards. Q5_K_M for better quality if you have 20+ GB VRAM. Q8_0 if VRAM isn't a constraint (28+ GB).

Does vLLM use GGUF format?

No. vLLM loads models directly from HuggingFace format (PyTorch/safetensors), not GGUF. If you're using vLLM, you don't need GGUF at all. vLLM has supported DiffusionGemma since day 0 with Google's collaboration.

PR #24423 In Draft — What’s Next

llama.cpp PR #24423 is still in draft. GGUF conversion and inference work from the PR branch. The GGUF files you download today will work with the PR branch, and will work with Ollama and LM Studio once the PR lands in mainline and they update their bundled engines.

Next milestone: Ollama and LM Studio release updated builds. Track ecosystem status →

View PR #24423 →

Trying to pick the right quant? Stop guessing.

The Production Guide has a complete quant comparison table — Q4_0 through FP16, benchmarked on 5 GPUs, with recommended quants per workload. Plus Docker Compose, GPU tuning, 20+ error fixes, and multi-user serving.

See the Production Guide

Last updated: July 1, 2026