Skip to main content
Back

RWTH Aachen Wires Diffusion Language Models Into Speech Recognition, Cutting Word Errors

RWTH Aachen researchers show masked and uniform-state diffusion models can rescore ASR hypotheses and even join CTC decoding, cutting word error rates.

Paper, Full Access
  • RWTH Aachen paper integrates masked and uniform-state diffusion language models into ASR.
  • New sample-level and global mask normalization estimators cut MDLM rescoring WER from 4.73% to 4.52% on LibriSpeech dev-other.
  • Novel CTC plus USDM joint decoder combines framewise acoustic scores with labelwise diffusion scores at every denoising step.
  • Joint decoding hits 4.71% WER versus 5.08% CTC greedy baseline and 4.80% USDM rescoring alone.
  • Autoregressive LMs still lead at 3.86% WER, but diffusion LMs may scale better with more training.
  • All code and recipes released, using a 24-layer Diffusion Transformer trained on LibriSpeech LM data.

A team from RWTH Aachen University and AppTek has wired diffusion language models, the parallel-generation cousins of GPT-style autoregressive models, into speech recognition pipelines. Their new paper lays out a full recipe for using two flavors of discrete diffusion models to rescore transcripts from a CTC acoustic model, and introduces a joint decoding scheme where the diffusion model actively participates in building the hypothesis rather than just re-ranking finished candidates.

The result is a measurable drop in word error rate on LibriSpeech, along with what may be the first systematic comparison of masked versus uniform-state diffusion for automatic speech recognition (ASR). The authors publish all their code and recipes.

Why bolt a diffusion model onto a speech recognizer

Speech recognizers typically combine an acoustic model with a language model that reranks or guides decoding. Autoregressive LMs are the standard choice thanks to their strong linguistic capabilities and ability to absorb external textual knowledge, but their strictly left-to-right structure caps decoding speed. That serialization is the bottleneck the authors want to break.

Diffusion language models generate text by starting from a corrupted sequence and iteratively denoising it, which lets them attend bidirectionally and predict many tokens in parallel. Two variants show up in the paper:

  • Masked diffusion language models (MDLM): corrupts text by randomly masking tokens and learns to reconstruct the sequence during the reverse generative pass. A special [MASK] token marks the positions to fill.
  • Uniform-state diffusion models (USDM): during the forward process, tokens are replaced with random samples from the vocabulary rather than a mask token. Since corrupted tokens are indistinguishable from clean ones, the model must re-evaluate every position at each denoising step, enabling a self-correcting property where previously predicted tokens can be updated or fixed.