Shanghai Jiao Tong Teaches Speech AI to Say Nothing When Unsure
A new metric and training framework teaches ASR models to say 'I don't know' instead of confidently hallucinating wrong words
Takeaways
- Researchers from Shanghai Jiao Tong University introduce RAS, a new ASR metric that rewards models for abstaining on uncertain words rather than hallucinating.
- A special
<PH>placeholder token is added to the model vocabulary; the model outputs it instead of guessing when audio is ambiguous. - RAS balances informativeness and error aversion via a parameter alpha=0.5064, calibrated from 980 human preference annotations across 42 listeners.
- Training uses two stages: supervised fine-tuning on placeholder-corrected transcripts, then RL via GRPO with RAS as the reward signal.
- On noisy audio (0 dB SNR), the method improves RAS by 0.2657 over baseline Whisper-Tiny; on code-switching TALCS, it flips a -0.11 RAS to +0.48.
- Code is open-sourced; RAS can be used as a drop-in complement to WER in existing evaluation pipelines.
Speech recognition systems have a dangerous habit: when audio gets noisy or ambiguous, they keep transcribing anyway, producing fluent-sounding text that is simply wrong. A model that confidently mishears a medical term or a legal name can cause real harm precisely because the output looks trustworthy. A new paper from Shanghai Jiao Tong University's X-LANCE Lab tackles this head-on, introducing both a new evaluation metric and a training pipeline that teaches ASR models to abstain when they are not sure.
The problem with always having an answer
ASR systems often produce confident yet incorrect transcriptions under noisy or ambiguous conditions, which can be misleading for both users and downstream applications. The standard fix has been to look at Word Error Rate (WER), which counts how many words were wrong. But standard evaluation based on WER focuses solely on accuracy and fails to capture transcription reliability. WER cannot tell you whether the system should have said anything at all.
Existing confidence estimation approaches do attach uncertainty scores to transcriptions, but they do so after the fact. These metrics cannot assess whether an ASR system makes an appropriate trade-off between informativeness and reliability under uncertainty. In this work, the authors argue that reliable ASR should be able to explicitly abstain from localized uncertain predictions. The key word is localized: prior work on abstention in ML typically rejects an entire input. In speech, that is too blunt. A sentence might be 90% clear and 10% inaudible, and throwing away the whole thing wastes good information.

Introducing the placeholder
The authors introduce a fine-grained abstention paradigm that augments the output space with a dedicated placeholder. Unlike full-sentence rejection, this approach allows the model to selectively abstain on ambiguous segments while transcribing the remaining content with high commitment. In practice, the model outputs a special <PH> token wherever it is uncertain, rather than guessing. A downstream system receiving chronic disease of <PH> and <PH> gland knows exactly which slots are missing, rather than receiving plausible-sounding hallucinations it has no way to distrust.