Skip to main content
Back

KIT Builds a 1M-Example Dataset to Fix Long-Form Speech AI

KIT's IWSLT 2026 system turns short speech datasets into 1M long-form training examples and exposes a critical flaw in how re-ranking is applied across mixed speech tasks

  • 1M+ training examples built from scratch by concatenating short audio clips, using LLMs to generate labels, and translating to German, Italian, and Chinese — no new recordings needed.
  • Likelihood re-ranking breaks semantic tasks: it cuts ASR WER by 24.9 points but tanks QA by 11 and summarization by 8.6 by over-selecting segmented audio candidates.
  • Lik.+MBR hybrid fixes the tradeoff: combining likelihood with Minimum Bayes Risk decoding retains most ASR gains while limiting damage to QA and summarization tasks.
  • Chain-of-Thought task tokens collapsed: the model almost always predicted the wrong task token, defaulting to summarization even for transcription inputs.
  • Cascaded system aced the surprise task (Quality Estimation) with 72.2% accuracy; the end-to-end audio model scored 0.0%, showing text-based pipelines generalize better to unseen tasks.
  • Dataset and checkpoints are public on HuggingFace under the YapayNet organization.

Teaching a speech model to transcribe, translate, summarize, and answer questions from the same long audio clip , all from a single natural-language instruction , is harder than it sounds. The dominant audio encoders cap out at 30 seconds per pass, most training data lives in short utterances, and if you try to re-rank outputs to improve quality, you can accidentally make things worse. Researchers at the Karlsruhe Institute of Technology (KIT) tackled all three of these problems head-on in their submission to IWSLT 2026's Instruction Following Track, and the lessons they documented are worth paying attention to.

The long-audio gap nobody talks about

Most modern speech-language models are built on top of the Whisper encoder. Most models rely on the Whisper encoder, which natively supports only 30 seconds per inference pass. Newer multimodal models like Phi-4 and Qwen2.5-Omni have removed that architectural limit, but they lack exposure to long-form audio during training, a gap that manifests as significant performance degradation even on basic ASR. The IWSLT benchmark surfaces this directly: the long-form track requires handling audio up to 15 minutes, which is 30x longer than what most models were trained on.

IWSLT proposed the second edition of the Instruction Following task, aimed at testing general models for the speech modality, which better reflects current trends in the research community. This year's track introduced new tasks including an unknown surprise task, posing a genuine challenge against overfitting to known tasks. The six tasks covered were: Automatic Speech Recognition (ASR), Speech Translation (ST), Spoken Question Answering (SQA), Speech Summarization (SSUM), Audio Chaptering (ACHAP), and the surprise task.

Building 1M training examples from scratch

KIT's core insight was that you don't need new recordings to build long-form training data. You need a smart augmentation pipeline. The team built a three-stage framework applied to existing short-form corpora:

  • Segment concatenation: Short audio clips are grouped by speaker and session, then greedily stitched together into chunks up to 15 minutes. This converts datasets like LibriSpeech and EuroParl-ST into long-form audio-transcript pairs without any new recording.