Meta's SeamlessM4T v2 Replaces Three AI Models With One for 96 Languages
Meta's SeamlessM4T v2 delivers speech and text translation across nearly 100 languages in one model, with faster inference and a novel UnitY2 architecture.
Takeaways
- Meta's SeamlessM4T v2 Large is a 2.3B-parameter all-in-one speech and text translation model.
- Supports 101 speech input languages, 96 text languages, and 35 speech output languages.
- New UnitY2 architecture adds non-autoregressive unit decoding for faster speech generation.
- Beats prior SOTA on Fleurs by 20% BLEU in direct speech-to-text translation.
- Ships with toxicity mitigation, gender bias evaluation, and inaudible audio watermarking.
- Available in Transformers under CC-BY-NC 4.0 (non-commercial only).
Meta's FAIR team released SeamlessM4T v2 Large, a single 2.3B-parameter model that handles speech-to-speech, speech-to-text, text-to-speech, text-to-text translation, and automatic speech recognition across a large slice of the world's languages. It has quietly become one of the most downloaded multilingual speech models on Hugging Face, and it is the backbone that Meta's SeamlessExpressive and SeamlessStreaming systems build on.
One model, five tasks, ninety-six languages
The pitch: you no longer need to chain an ASR model, an MT model, and a TTS model together to translate speech between two languages. SeamlessM4T is an all-in-one multilingual and multimodal translation model delivering high-quality output for speech and text in nearly 100 languages.
Task coverage spans:
- Speech-to-speech translation (S2ST)
- Speech-to-text translation (S2TT)
- Text-to-speech translation (T2ST)
- Text-to-text translation (T2TT)
- Automatic speech recognition (ASR)
Language coverage is asymmetric, which is worth understanding before you plan a project around it. There are 101 languages for speech input, 96 languages for text input and output, and 35 languages for speech output. You can transcribe or translate spoken input from almost any major world language, but if you want the model to actually speak the result out loud, your target list narrows considerably.
What UnitY2 actually changes
The headline architecture change from v1 is UnitY2, a multitask framework with hierarchical character-to-unit upsampling and non-autoregressive text-to-unit decoding. Meta reports it considerably improves over SeamlessM4T v1 in both quality and inference speed.
Two ideas do the heavy lifting: