Skip to main content
Back

Academia Sinica's GatherMOS Uses GPT-5 to Judge Speech Quality Without Training Data

GatherMOS uses GPT-5 as a meta-evaluator, combining acoustic descriptors with DNSMOS and VQScore pseudo-labels to predict speech quality without heavy training.

  • GatherMOS uses GPT-5 as a meta-evaluator to predict speech quality MOS scores without training a dedicated model.
  • It fuses acoustic descriptors like RMS, ZCR, and MFCCs with pseudo-labels from DNSMOS and VQScore.
  • Zero-shot GatherMOS beats DNSMOS, VQScore, naive averaging, and CNN-BLSTM/MOS-SSL on VoiceBank-DEMAND under limited data.
  • Few-shot in-context examples push SRCC to 0.8473 when support samples match test conditions.
  • Mismatched few-shot samples hurt generalization, revealing a domain-bias tradeoff versus zero-shot robustness.
  • Read the full paper on arXiv for the prompt format and full results tables.

Predicting how humans will rate the quality of a piece of speech audio sounds simple but eats through labeled data like nothing else. A team from Academia Sinica, NVIDIA, and the University of Palermo just proposed a workaround: skip training a dedicated model, and instead have a large language model reason over the outputs of several weaker predictors. Their framework, GatherMOS, uses GPT-5 as a meta-evaluator that fuses lightweight acoustic features with pseudo-labels from existing quality estimators.

The MOS prediction bottleneck

Mean Opinion Score (MOS) is the standard yardstick for speech quality, a number usually between 1 and 5 that averages ratings from human listeners. Human evaluation remains the gold standard, but it requires enough listeners to produce reliable, generalizable scores. That is expensive and slow, so the field has turned to non-intrusive predictors that estimate MOS directly from audio without needing a clean reference signal.

Deep learning approaches dominate this space, with recent efforts leaning on large pre-trained backbones like wav2vec 2.0 and Whisper. The catch is the usual one: reliable performance still hinges on a sufficient number of training samples. When labeled MOS data is scarce, these models fall apart.

Letting an LLM do the aggregation

Earlier attempts at using LLMs for this task fed audio-derived text directly into ChatGPT and asked it to judge naturalness. The authors point out that such studies rely on direct prompting and lack explicit mechanisms to incorporate acoustic features or auxiliary predictors, which limits their robustness. Their preliminary experiments indicate that ChatGPT struggles with raw audio, requiring intermediate features as proxies. Handcrafted features alone, like RMS or MFCCs, are too coarse to correlate well with perceptual quality.