llama.cpp · PR #24423 (Draft)

Run DiffusionGemma with llama.cpp

PR #24423 is still in draft. Must build from the PR branch — not in mainline yet. Working commands for CUDA, Metal, and CPU. Maintainer feedback pending on a more general diffusion-server design.

Build Guide (~20min) Run Inference

Build llama.cpp with Diffusion Support

Prerequisites

# Ubuntu / Debian
sudo apt install build-essential cmake git

# macOS
xcode-select --install
brew install cmake

# Verify cmake
cmake --version  # needs 3.18+

Clone & Build from PR Branch

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
# PR #24423 is still in draft — must fetch PR branch
git fetch origin pull/24423/head:diffusion-support
git checkout diffusion-support
# Now build from PR branch

Build with CUDA (NVIDIA GPU)

mkdir build && cd build
cmake .. -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
make -j$(nproc)

# Verify
./bin/llama-cli --version

Build with Metal (Apple Silicon)

mkdir build && cd build
cmake .. -DGGML_METAL=ON -DCMAKE_BUILD_TYPE=Release
make -j$(sysctl -n hw.logicalcpu)

Build CPU-only (fallback)

mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release
make -j$(nproc)

Run Inference

Basic text generation

./build/bin/llama-cli \
  -m diffusiongemma-26b-q4_k_m.gguf \
  -p "Explain how diffusion models work for text generation:" \
  -n 256 \
  --diffusion-steps 8 \
  --temperature 0.7

Interactive chat mode

./build/bin/llama-cli \
  -m diffusiongemma-26b-q4_k_m.gguf \
  --interactive \
  --diffusion-steps 8 \
  --temp 0.7

Performance Expectations

GPU	Quantization	Tokens/s (est.)
RTX 4090	Q4_K_M	500–700
RTX 3090	Q4_K_M	300–500
M3 Max (Metal)	Q4_0	100–200
CPU (8+ cores)	Q4_0	5–15

Note: These are rough estimates from community testing. The diffusion speed advantage is most visible on high-end GPUs. CPU-only loses the 4x benefit.

Troubleshooting

"Unsupported model architecture" error

You're on an older llama.cpp version or a different branch. Make sure you're on the PR branch: git checkout diffusion-support && git pull origin pull/24423/head. PR #24423 is still in draft — diffusion_gemma is not in master yet.

Build fails with "cublas not found"

# Verify CUDA toolkit is installed
nvcc --version
# Set CUDA path if needed
export CUDA_PATH=/usr/local/cuda
cmake .. -DGGML_CUDA=ON -DCUDAToolkit_ROOT=$CUDA_PATH

Segfault on first token

The GGUF file might be corrupted or from an incompatible converter version. Re-download from the official source or re-convert with the latest convert.py.

PR #24423 — In Draft (Not Merged)

PR #24423 is still in draft. Must build from the PR branch. Maintainer wants a more general diffusion-server design before merging — has merge conflicts. Ollama and LM Studio cannot bundle until this lands. Ecosystem status →

What	Status
PR #24423	Draft ⚠️
Author	danielhanchen
Merged	Late June 2026
Next milestone	Ollama / LM Studio engine updates

DiffusionGemma support is in PR #24423, still in draft. You must build from the PR branch — git fetch origin pull/24423/head:diffusion-support && git checkout diffusion-support && make -j$(nproc). Once the PR lands in mainline, a simple git clone && make will work.

Want to go further?

This tutorial covers local llama.cpp setup. The Production Guide covers Docker Compose, GPU tuning, 20+ error fixes, and multi-user serving — everything after compile succeeds.

See the Production Guide

Last updated: July 1, 2026