DiffusionGemma Benchmarks
How fast is diffusion generation on real hardware? Tokens/s, VRAM usage, and apples-to-apples comparisons with autoregressive models of similar size.
Generation Speed by GPU
All measurements use the DiffusionGemma 26B-A4B-it model at Q4_K_M quantization, 256-token generation, single request (batch=1). Tested with llama.cpp PR branch (PR #24423 — in draft).
Single Request Throughput (tokens/s)
| GPU | VRAM | Tokens/s | Latency (256 tok) | Speedup* |
|---|---|---|---|---|
| RTX 4090 | 24GB | ~180 | ~1.4s | 3.6x |
| RTX 3090 | 24GB | ~155 | ~1.7s | 3.1x |
| RTX 4070 Ti | 12GB | ~120 | ~2.1s | 2.4x |
| A100 80GB | 80GB | ~320 | ~0.8s | 4.0x |
| Mac M2 Ultra | 76GB (U) | ~45 | ~5.7s | 1.8x |
*Speedup vs autoregressive Gemma 2 27B at same precision. Benchmarks from community testing. Your results will vary with quantization, batch size, and system load.
Multi-GPU Throughput
Tensor-parallel scaling with vLLM 0.8.0.dev (TP fix shipped June 26). Effective tokens/s across GPU pairs.
| Config | Total VRAM | Tokens/s | vs 1-GPU | Best For |
|---|---|---|---|---|
| 2× RTX 4090 (TP=2) | 48GB | ~340 | 1.9x | Batched serving |
| 2× RTX 3090 (TP=2) | 48GB | ~290 | 1.9x | Budget dual-GPU |
| 2× RTX 5060 Ti (TP=2) | 32GB | ~210 | — | Min entry dual-GPU |
| 2× A100 80GB (TP=2) | 160GB | ~600 | 1.9x | High-throughput API |
TP=2 scaling is ~1.9x (not 2x) due to all-gather communication overhead on the self-conditioning soft-embed path. TP=4 delivers ~3.2x on data center GPUs with NVLink. All measurements Q4_K_M, 256-token output, vLLM serve, batch=1.
Diffusion vs Autoregressive: Apples to Apples
Why Diffusion Is Faster
Autoregressive models generate tokens one-by-one — 256 output tokens = 256 sequential forward passes. DiffusionGemma generates all 256 tokens simultaneously in ~8 refinement steps. That's 256 passes vs 8 passes for the same output.
Throughput Comparison (RTX 4090, 256-token output, Q4_K_M)
| Model | Architecture | Tokens/s | Time to 256 tok |
|---|---|---|---|
| DiffusionGemma 26B | Diffusion | ~180 | ~1.4s |
| Gemma 2 27B | Autoregressive | ~50 | ~5.1s |
| Llama 3.1 8B | Autoregressive | ~95 | ~2.7s |
| Qwen 2.5 32B | Autoregressive | ~35 | ~7.3s |
A 26B diffusion model generating faster than an 8B autoregressive model — that's the key advantage of block-level parallel decoding.
VRAM Usage
Memory at Rest vs Under Load
| Quant | Model Size | VRAM Idle | VRAM Gen | Min GPU |
|---|---|---|---|---|
| Q4_K_M | ~16GB | ~16.5GB | ~17.8GB | 24GB |
| Q4_0 | ~15GB | ~15.5GB | ~16.8GB | 24GB |
| Q2_K | ~10GB | ~10.5GB | ~11.5GB | 12GB |
| FP16 | ~52GB | ~53GB | ~56GB | 80GB (A100) |
VRAM Gen includes KV cache for a 32K context window. Longer contexts eat more memory — budget ~0.5GB per 8K tokens of context length for Q4 quant.
Tuning for Max Throughput
Batch vs Latency Tradeoff
DiffusionGemma scales well with batch size since all sequences in a batch share the same diffusion timestep:
| Batch Size | Total Tokens/s | Per-Request Latency | VRAM Delta |
|---|---|---|---|
| 1 | ~180 | ~1.4s | baseline |
| 4 | ~520 | ~2.0s | +2.5GB |
| 8 | ~880 | ~2.3s | +5.5GB |
RTX 4090, Q4_K_M, 256-token generation. Batch = concurrent generation requests.
Liked the Numbers. Want the Full Picture? →
These benchmarks show single-GPU, single-request numbers. The Production Manual covers multi-GPU scaling, continuous batching tuning, queue management, and real-world throughput under concurrent load. 8 chapters, copy-paste ready. $39 $29, lifetime updates.
Last updated: July 1, 2026