Alibaba's TLA-SA Fixes Voice Cloning Drift Without Extra Data or Fine-Tuning
A new training strategy called TLA-SA fixes the speaker mismatch problem in flow-matching TTS by weighting alignment loss across time and depth.
Takeaways
- TLA-SA adds a time and layer weighted speaker alignment loss to flow-matching TTS training.
- Analysis shows speaker identity concentrates in early denoising steps and shallow layers, not uniformly.
- Gains of 3.0% Sim-WavLM and 2.9x faster convergence on a 100k-hour corpus versus baseline.
- Matches FireRedTTS-2 speaker similarity while using 4x less data and no SFT stage.
- Works on both LM-based (CosyVoice 2) and LM-free (F5-TTS) architectures.
- Demo available at tlasa-tts.github.io; weights and code not yet released.
Zero-shot text-to-speech has gotten remarkably good at cloning voices from a few seconds of audio, but there is a persistent complaint about the current wave of flow-matching models: they sound natural, yet the cloned voice often drifts from the reference speaker. A new paper from Shanghai Jiao Tong University's X-LANCE Lab, Alibaba's Taobao & Tmall Group, and Nanjing University proposes Time-Layer Adaptive Speaker Alignment (TLA-SA), a training tweak that closes that gap without a separate fine-tuning stage or a bigger dataset.
Why flow-matching TTS forgets the speaker
Flow-matching has become the default backbone for state-of-the-art TTS systems like Seed-TTS, CosyVoice 2, and F5-TTS. The model learns to transform Gaussian noise into a Mel spectrogram by following a smooth vector field, conditioned on text and a speaker prompt. The standard training objective implicitly models the speech distribution without explicitly enforcing perceptual attributes like speaker identity, so fine-grained timbre cues are learned only as a side effect of matching the overall distribution.
Prior fixes bolt on an auxiliary alignment loss that pushes intermediate features toward embeddings from a pre-trained speaker encoder. These methods apply the loss uniformly across layers and timesteps, and the authors show that assumption is wrong.
Where speaker identity actually lives
Using a training-free similarity metric called Centered Kernel Nearest-Neighbor Alignment (CKNNA), which quantifies the similarity between two sets of embeddings, the team probed where speaker information sits inside a converged flow-matching model. Two patterns jump out: