HumDial-EIBench Catches Top Audio AI Models Faking Emotional Understanding
A new benchmark tests audio language models on real human conversations and reveals they still treat speech as glorified text transcription.
Takeaways
- HumDial-EIBench evaluates audio language models on real human multi-turn dialogues, not synthesized speech.
- Four tasks: emotional trajectory, causal reasoning, empathetic response, and acoustic-semantic conflict.
- Cognitive tasks reformulated as MCQs with adversarial distractors to bypass unreliable LLM-as-judge scoring.
- Gemini-2.5-flash leads with 88% trajectory accuracy; Freeze-Omni scores just 11%.
- All models collapse on conflict samples: Qwen2.5-Omni drops from 88% to 22% under sarcasm.
- Code and dataset released at github.com/ASLP-lab/HumDial-EIBench.
A new benchmark called HumDial-EIBench puts eight leading audio language models (ALMs) through a battery of emotional intelligence tests built from real, multi-turn human dialogues, and the results are unflattering. Even the top performers collapse when a speaker's tone contradicts their words, exposing a persistent bias where models effectively transcribe audio first and reason about it as text.
Why existing audio benchmarks miss the point
Most multi-turn dialogue benchmarks rely entirely on TTS-synthesized speech, which strips out the acoustic nuance that makes emotion recognition hard in the first place. Some benchmarks incorporate authentic human recordings, but confine evaluation to static, single-turn emotion recognition, failing to test emotional trajectory tracking and implicit causal reasoning across multiple conversational turns. Current frameworks also lean heavily on unstable LLM-as-a-judge scoring, which introduces subjective instability and blurs the line between genuine reasoning errors and superficial generation fluency.
The authors, from Northwestern Polytechnical University's ASLP group, argue this masks a fundamental question: do end-to-end ALMs actually hear emotion, or are they still using audio as a proxy for text?
Four tasks, one uncomfortable answer
HumDial-EIBench is built on the test set of the ICASSP 2026 HumDial Challenge and comprises 1,077 evaluation samples split across Chinese and English. Rather than lean on open-ended generation throughout, the authors convert cognitive tasks into multiple-choice questions with adversarial distractors. The four tasks are: