Skip to main content
Back

Xiaomi's CLSR Makes Audio AI Answer 30-Minute Lectures 10x Faster

A new retriever from Wuhan University and Xiaomi turns speech into text-like tokens before matching, cutting long-audio QA time tenfold.

  • CLSR is an end-to-end speech retriever that extracts question-relevant clips from long audio for downstream spoken QA.
  • It converts acoustic features into text-like tokens via CIF and a VQ adaptor before contrastive alignment with real text.
  • Beats CLAP, SpeechDPR and Whisper+BGE pipelines on Spoken-SQuAD, LibriSQA, SLUE-SQA-5, and DRCD.
  • On 30-minute Spoken Wikipedia audio, boosts Qwen-Audio EM from 18.00 to 27.60 with a 10x speedup.
  • Retrieval quality collapses once WER exceeds roughly 16.75%, so noisy domains remain a limitation.
  • Code available at github.com/193746/CLSR, built on FunASR; accepted at AAAI.

Large audio language models handle a 30 second clip well, but drop them into a 30 minute lecture and they fall apart. A team from Wuhan University and Xiaomi has released CLSR, a contrastive language-speech retriever that pulls only the question-relevant chunks out of long recordings before handing them to a downstream audio model. Rather than forcing raw acoustic vectors to align with text embeddings directly, CLSR first converts the audio into text-like tokens, then does the matching in text space.

Why long audio breaks spoken QA

Most existing spoken QA models are limited to audio segments under one minute, yet meetings, lectures, and podcasts routinely run past ten. The natural fix is retrieval augmented generation: chunk the audio, score each chunk against the question, and feed only the top matches into the audio LLM. That works beautifully for text, but speech retrievers have been the weak link.

Diagram showing how a speech retriever narrows long audio to short context before the SQA model answers

Two existing approaches have problems. End-to-end models like CLAP were built for pairing sounds with captions and, according to the authors, struggle with question-to-speech-content alignment. Pipeline approaches run the whole recording through an ASR system like Whisper and then apply a text retriever such as BGE, which works but runs slowly and compounds transcription errors across the entire document.

A text-like bridge across the modality gap

CLSR's central idea is that the gap between acoustic features and text embeddings is too wide to close with a single contrastive loss. The model inserts an intermediate step: it first maps speech representations into text-like representations, then aligns those with actual text representations inside the text modality.