Telefónica Adapts Voxtral-Mini for Dysarthric Speech by Touching Only 1.6% of Parameters
Researchers freeze a 4.7B-parameter SpeechLLM and inject speaker identity into every transformer layer to transcribe dysarthric speech without breaking the base model
Takeaways
- Core idea: FiLM conditioning injects speaker identity into every frozen transformer layer of a 4.7B SpeechLLM, adapting it to dysarthric speech without modifying base weights.
- Only 1.6% of parameters trained: The FiLM generator bank (~48.3M params) and speaker embedding extractor (~25.2M params) are the only trainable components.
- Competitive WER: FiLM achieves 16.36% WER on TORGO with post-processing, vs. 10.90% for full fine-tuning and 22.09% for the base model.
- Preserves model capabilities: Unlike F-LoRA (which collapsed MCQA accuracy to 8.4%), FiLM conditioning retained 60.7% accuracy on paralinguistic questions.
- Key limitation: Requires prior knowledge of whether input speech is pathological to zero out healthy-speaker embeddings at inference time.
- Open source: Code is available at github.com/ferugit/film-spk-asr; evaluated on TORGO (English) and NeuroVoz (Spanish) datasets.
Speech recognition has become remarkably good at understanding typical voices. But for the estimated millions of people living with neurological conditions like ALS, Parkinson's disease, or cerebral palsy, most modern ASR systems still fail badly. The acoustic patterns these conditions produce are so different from the training data that even state-of-the-art models struggle to keep up. A new paper from researchers at Telefónica Innovación Digital, Universidad Autónoma de Madrid, and Brno University of Technology proposes a clever fix: instead of retraining the whole model, inject speaker-specific information directly into each layer of a frozen encoder using a technique called Feature-wise Linear Modulation (FiLM).
Why pathological speech is so hard
ASR systems continue to struggle with speech produced by individuals with neuromotor disorders such as ALS or Parkinson's disease. These conditions frequently cause dysarthria, a motor speech disorder that impairs articulatory clarity, alters prosody, and produces high inter-speaker variability, creating an acoustic mismatch with the normative speech on which modern ASR systems are trained. In plain terms: the model has never heard speech that slurs, slows, or breaks in these particular ways, and it has no way to adapt on the fly.
A major challenge in improving ASR for dysarthric speech is the scarcity of annotated dysarthric speech datasets. This data scarcity makes the usual solution, fine-tuning on in-domain examples, fragile. When you only have a handful of speakers to train on, the model tends to memorize them rather than generalize. And fine-tuning a large model on a small dataset risks something worse: catastrophic forgetting, where the model loses its ability to transcribe healthy speech in the process of learning pathological speech.
The standard playbook and its tradeoffs
The dominant approaches to this problem are full fine-tuning (FFT) and parameter-efficient fine-tuning methods like LoRA. Both work by modifying the model's weights. Voxtral closes a gap between classic ASR systems, which deliver cost-efficient transcription but lack semantic understanding, and more advanced LLM-based models, which provide transcription and language understanding. But as ASR systems evolve into full SpeechLLMs capable of answering questions about audio, preserving those broader capabilities becomes critical. Adapting the weights risks degrading everything the model learned to do beyond simple transcription.