Run Google's Diffusion LLM Locally — DiffusionGemma

What Is DiffusionGemma

DiffusionGemma generates text like an image diffusion model: instead of typing one word at a time left-to-right, it fills in a whole 256-token block at once through iterative denoising — imagine a paragraph that "comes into focus" over multiple passes rather than being typed letter by letter.

It's a 26B-parameter Mixture of Experts (MoE) model built on the Gemma 4 backbone that activates only 3.8B parameters during inference. The paradigm shift isn't model size or dataset — it's the generation algorithm. Standard autoregressive LLMs (GPT, Llama, Qwen, Gemma) are causal — each token depends only on previous tokens. DiffusionGemma uses bidirectional attention during generation, meaning every token in a 256-token canvas can attend to every other token simultaneously. This enables self-correction: if the model becomes less confident about a token mid-generation, it can "re-noise" and replace it — something autoregressive models cannot do because they commit each token permanently.

Google released it under Apache 2.0 on June 10, 2026. Weights are on HuggingFace at google/diffusiongemma-26B-A4B-it. It supports text, image, and video inputs (multimodal), a 256K token context window, and 140+ languages.

How to Run DiffusionGemma Locally

Prerequisites: Hardware

Cards with limited compute (RTX 3060, RTX 4060, laptop GPUs) and Apple Silicon Macs will see significantly reduced speedups.

Method 1: vLLM Recommended

vLLM has day-zero support with optimized block denoising kernels. This is the fastest path — and the only one with independently verified speed numbers (1,288 tokens/s on H200).

pip install vllm>=0.12.0

from vllm import LLM, SamplingParams

llm = LLM(
    model="google/diffusiongemma-26B-A4B-it",
    trust_remote_code=True,
    tensor_parallel_size=1,
    max_model_len=65536,
)

sampling_params = SamplingParams(
    temperature=0.7,
    max_tokens=2048,
)

outputs = llm.generate(["Explain how diffusion language models work."], sampling_params)
for output in outputs:
    print(output.outputs[0].text)

Method 2: llama.cpp Bleeding Edge

llama.cpp support is via unmerged PR #24427. Clone and checkout the PR branch:

git clone https://github.com/ggml-ai/llama.cpp
cd llama.cpp
git fetch origin pull/24427/head:diffusiongemma
git checkout diffusiongemma

# Build with CUDA
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release

# Run with GGUF quantized weights
./build/bin/llama-cli \
  -m DiffusionGemma-26B-A4B-it-Q4_K_M.gguf \
  -p "Explain how diffusion language models work." \
  -n 512 \
  -ngl 99

This is bleeding edge — expect bugs. Check PR #24427 for latest status before using in production.

Method 3: Unsloth

pip install unsloth
pip install flash-attn --no-build-isolation  # optional

from unsloth import FastModel

model, tokenizer = FastModel.from_pretrained(
    model_name="google/diffusiongemma-26B-A4B-it",
    max_seq_length=65536,
    load_in_4bit=True,  # fits in ~16GB
)

Unsloth provides a hardware-requirements table at unsloth.ai/docs/models/diffusiongemma with VRAM estimates for all quantization levels.

Method 4: HuggingFace Transformers

Simplest path for a quick first test:

pip install transformers>=4.55.0 accelerate

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "google/diffusiongemma-26B-A4B-it"

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)

tokenizer = AutoTokenizer.from_pretrained(model_id)
inputs = tokenizer("Explain diffusion language models in one paragraph.", return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Method 5: Ollama Community Workaround

Ollama support is coming via llama.cpp PR #24427. Until the PR merges, you can use the bridge workaround:

1. Build llama.cpp with diffusion support (PR #24427 branch)
2. Download or convert a GGUF model
3. Create a Modelfile and run with ollama create

Full Ollama Setup Guide →

Quantization for Consumer GPUs

NVFP4 (Blackwell GPUs only: RTX 5090 / RTX PRO 6000)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=nvfp4,
    device_map="auto",
    trust_remote_code=True,
)

Bitsandbytes 4-bit (Ampere+: RTX 3090, 4090)

from transformers import BitsAndBytesConfig

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=bnb_config,
    device_map="auto",
    trust_remote_code=True,
)

GGUF Q4_K_M (llama.cpp path)

Download community-converted GGUF files from HuggingFace. ~16GB VRAM, ~200–400 tokens/s on RTX 4090 (community estimate).

Troubleshooting

Detailed Setup Guides

Use Cases

When NOT to Use DiffusionGemma