Skip to main content
Back

Peking University's SAID Makes Open-Source Diffusion LLMs 9x Faster Without Retraining

SAID cuts diffusion LLM inference by up to 9.1x by spending compute where it matters most, not uniformly across all tokens

Paper, Full Access
  • 9.1x speedup: SAID accelerates diffusion LLM inference on LLaDA-8B and LLaDA 1.5 with no retraining required.
  • Core idea: Splits tokens into "scaffold" (semantic anchors, full denoising) and "detail" (filler, fewer steps) to reallocate compute.
  • CHLG module: Assigns extra denoising steps only to low-confidence tokens, further sharpening the speed-accuracy tradeoff.
  • Best gains on knowledge/reasoning: Up to 5.77x latency reduction on GPQA and ARC-C, with slight accuracy improvements.
  • Weaker on chain-of-thought math: GSM8K sees accuracy drops because step-by-step reasoning has no true "filler" tokens.
  • Open source: Code available at github.com/TH-AI-Lab-PKU/SAID for LLaDA-8B and LLaDA 1.5.

Diffusion language models have a compelling pitch: instead of generating one token at a time like GPT-style models, they refine an entire sequence in parallel, potentially unlocking much faster generation. By generating tokens in parallel through an iterative denoising process, diffusion LMs possess inherent advantages in reducing inference latency and capturing bidirectional context. The catch? High-quality generation typically requires many denoising steps, often comparable to the generation length, and naively reducing the number of steps tends to degrade quality sharply.

A new paper from Peking University proposes SAID (Scaffold-Aware Iterative Decoding), a training-free framework that attacks this bottleneck head-on. The core insight is simple but powerful: not every token in a sequence needs the same amount of denoising work. Some tokens anchor the meaning of a sentence; others just fill in the details. SAID exploits this asymmetry to cut inference cost dramatically, achieving up to a 9.1x speedup on LLaDA-8B with competitive accuracy across math, coding, and knowledge benchmarks.

The real cost of parallel generation

To understand why this matters, it helps to know how diffusion LLMs actually work. Unlike image diffusion models, a diffusion LLM's forward process involves gradually masking tokens in a text sequence until it's fully masked, and the reverse process then learns to iteratively predict and reconstruct these masked tokens to recover the original sentence. The problem is that MDLM sampling remains expensive: generation requires many full-sequence denoising passes with a large transformer, and unlike autoregressive decoding, this process cannot benefit from KV caching.

This is the paradox at the heart of diffusion LLMs. They can update many positions at once, but each of those updates requires a full forward pass through the entire model. Decoding efficiency remains a central bottleneck for diffusion LLMs. Generation proceeds via a reverse-diffusion chain with many refinement steps, and practitioners must choose step budgets and transfer schedules conservatively to avoid quality loss. The result: a model that is theoretically parallel but practically slow.