SemiCat Boosts Diffusion Text AI Accuracy from 4% to 50% Without Extra Training
A new study dissects why continuous diffusion text models work, replacing self-conditioning with cheaper tricks that push TinyGSM accuracy from 4% to 50%.
Takeaways
- New paper dissects continuous diffusion language models, focusing on self-conditioning and embedding geometry as key design levers.
- DoubleCat replaces self-conditioning with a two-level interpolant, no extra forward pass, tripling TinyGSM accuracy.
- SemiCat adds discrete token reveals on top of DoubleCat, hitting 49.94% on TinyGSM without any self-conditioning.
- At 1.7B parameters, SemiCat beats discrete diffusion on 4 of 6 likelihood benchmarks and improves unconditional generation.
- Cross-entropy training pushes embedded flows toward a geometry whose decoding error rate matches one-hot flows.
- Builds on CANDI's token identifiability framework; appendix covers distillation and asynchronous schedules.
Continuous diffusion language models, or CDLMs, generate text by progressively denoising continuous token representations. In the new study On the Design Space of Continuous Diffusion Language Models, researchers examine self-conditioning and embedding geometry, then propose cheaper training designs. Their strongest SemiCat configuration raises TinyGSM accuracy from 4.42% to 49.94% without self-conditioning.
For developers evaluating diffusion-based text generation, the changes reduce training cost and clarify which components produce the gains. Self-conditioning usually requires an additional model pass during training. SemiCat removes that pass while retaining the continuous states used for guidance and controllable generation.
Why Gaussian noise misfits tokens
A CDLM commonly represents each token with a one-hot vector, containing a single 1 and zeros elsewhere, or with a learned dense embedding. Training adds Gaussian noise to these representations and teaches a denoiser to recover cleaner token states. Gaussian diffusion fits continuous signals such as pixels directly, whereas categorical tokens must first be mapped into a continuous space.
A CANDI analysis separates the resulting damage into discrete corruption, which determines whether the token remains identifiable, and continuous corruption, which measures how far its representation has moved. Those processes advance at different rates as vocabulary size changes, so one noise schedule can poorly serve both. This coupling helps explain why continuous diffusion has often trailed masked discrete diffusion on text.
Self-conditioning feeds an earlier model prediction into a later denoising step. The added prediction improves token recovery and exposes the network to model-generated states, but producing it during training typically requires another forward pass.