NTU and NUS Show Text Shape Beats Accent in Fixing Bilingual Speech AI
A systematic study of code-switching speech recognition finds that text patterns matter more than accent, and introduces SECT to synthesize training data with LLMs.
Takeaways
- New systematic study benchmarks code-switching ASR methods across ASRU and SEAME datasets from model and data perspectives.
- Proposes SECT prompting: three simple rules that guide LLMs to generate linguistically valid code-switching text.
- SECT-generated text reaches CMI 17.3 versus real 17.0, cutting perplexity from 1794 to 540.
- Fine-tuning Whisper-small with SECT-augmented data drops ASRU error rate from 8.8% to 8.3%.
- Textual switching patterns matter far more than speaker accent for downstream ASR performance.
- Language-specific encoder tricks help small models but barely help large multilingual models like Whisper.
Mixing two languages inside a single sentence is normal in bilingual conversations, but it wrecks most speech recognition systems. Even a large multilingual model like Whisper sees its error rate jump when Mandarin and English collide inside one utterance. A new study from a team at NTU and NUS pulls apart why this happens, tests the field's most-cited fixes side by side, and proposes a lightweight prompting recipe called SECT that lets an LLM synthesize the training text you cannot collect from real speakers.
What makes the paper unusual is that it does not propose one more architecture. It runs a controlled bake-off across model-centric and data-centric methods on two benchmarks with very different personalities, then isolates which failure modes each fix actually addresses.
Two failure modes that get tangled together
The authors define the core challenge as two distinct problems that most prior work lumps together. Accent, particularly common among L2 speakers, can blur the phonetic boundaries between the two languages, making the acoustic mapping harder. Code-switching introduces language confusion within the matrix language, meaning the dominant or structural language in a mixed utterance. That confusion is primarily textual and arises from the insertion of elements from an embedded language, which disrupts the expected syntactic and phonological patterns during language modeling.
On top of that, annotated code-switching corpora are tiny, because labeling them requires bilingual annotators. So even when Mandarin and English are individually high-resource, the mixed version is stubbornly low-resource.
The model-side bake-off
The paper compares two dominant families of algorithmic fixes. The first adds an auxiliary language identification or language diarization task alongside ASR. The second splits the encoder into language-specific modules (a matrix-language encoder and an embedded-language encoder) whose outputs are mixed by a built-in language posterior.