Stanford's NCM Generates Text 31x Faster Than Diffusion Models
A Stanford and Hazy Research team unveils a two-stage text generator that beats diffusion language models by 10 to 31 times on wall-clock speed.
Takeaways
- Stanford's Hazy Research introduces Next-Concept Modeling, a coarse-to-fine parallel text generator.
- A residual VQ-VAE encodes text into hierarchical discrete codes, predicted level by level in parallel.
- 21x faster than top diffusion baseline on LM1B with better generative perplexity at matched parameters.
- 31x faster on XSum summarization at matching ROUGE-L; 10x faster on OpenWebText.
- Beats autoregressive and diffusion baselines on Sudoku across all difficulty levels with fewer steps.
- Code, checkpoints and datasets open-sourced at HazyResearch/ncm.
Stanford’s Hazy Research group has proposed Next-Concept Modeling (NCM), a text-generation method that represents each sequence as a hierarchy of discrete codes. A transformer predicts every code within a level in parallel, starting with a short conceptual sketch and adding detail at progressively finer levels.
At matched parameter counts, the paper reports better generative perplexity than the strongest diffusion language model baseline on LM1B with 21 times faster sampling. NCM remains competitive on OpenWebText at 10 times the speed and matches the strongest diffusion model’s ROUGE-L score on XSum while running 31 times faster. These results come from the paper’s hardware and benchmark configurations, so production gains will depend on sequence length, batch size, implementation, and accelerator.
Where diffusion spends its compute
Autoregressive language models generate tokens sequentially, making output length a direct source of latency. Diffusion language models (DLMs) can update many tokens simultaneously, but generation usually requires repeated denoising passes over the full sequence. Reducing the number of passes lowers compute while often degrading output quality.
NCM reduces both the number and cost of those passes. Early predictions operate on short, coarse code sequences. Later predictions add detail, and the longest representation is processed once. This structure gives the model a fixed generation path instead of the many token-unmasking orders used to train diffusion models.
A hierarchy built in two stages
- Train the tokenizer.