Skip to main content
Back

HumDial-EIBench Catches Top Audio AI Models Faking Emotional Understanding

A new benchmark tests audio language models on real human conversations and reveals they still treat speech as glorified text transcription.

  • HumDial-EIBench evaluates audio language models on real human multi-turn dialogues, not synthesized speech.
  • Four tasks: emotional trajectory, causal reasoning, empathetic response, and acoustic-semantic conflict.
  • Cognitive tasks reformulated as MCQs with adversarial distractors to bypass unreliable LLM-as-judge scoring.
  • Gemini-2.5-flash leads with 88% trajectory accuracy; Freeze-Omni scores just 11%.
  • All models collapse on conflict samples: Qwen2.5-Omni drops from 88% to 22% under sarcasm.
  • Code and dataset released at github.com/ASLP-lab/HumDial-EIBench.

A new benchmark called HumDial-EIBench puts eight leading audio language models (ALMs) through a battery of emotional intelligence tests built from real, multi-turn human dialogues, and the results are unflattering. Even the top performers collapse when a speaker's tone contradicts their words, exposing a persistent bias where models effectively transcribe audio first and reason about it as text.

Why existing audio benchmarks miss the point

Most multi-turn dialogue benchmarks rely entirely on TTS-synthesized speech, which strips out the acoustic nuance that makes emotion recognition hard in the first place. Some benchmarks incorporate authentic human recordings, but confine evaluation to static, single-turn emotion recognition, failing to test emotional trajectory tracking and implicit causal reasoning across multiple conversational turns. Current frameworks also lean heavily on unstable LLM-as-a-judge scoring, which introduces subjective instability and blurs the line between genuine reasoning errors and superficial generation fluency.

The authors, from Northwestern Polytechnical University's ASLP group, argue this masks a fundamental question: do end-to-end ALMs actually hear emotion, or are they still using audio as a proxy for text?

Four tasks, one uncomfortable answer

HumDial-EIBench is built on the test set of the ICASSP 2026 HumDial Challenge and comprises 1,077 evaluation samples split across Chinese and English. Rather than lean on open-ended generation throughout, the authors convert cognitive tasks into multiple-choice questions with adversarial distractors. The four tasks are: