ByteDance's MegaTTS3 Clones Voices in 3 Seconds With 0.45B Parameters
ByteDance open-sources MegaTTS3, a 0.45B-parameter voice cloning model that generates one minute of speech in just 8 diffusion steps
Takeaways
- ByteDance open-sources MegaTTS3, a zero-shot TTS model with 0.45B parameters that rivals much larger systems.
- Core innovation is sparse alignment: loose phoneme boundary hints guide a latent diffusion transformer, balancing robustness and naturalness.
- 8-step generation via PeRFlow distillation enables high-quality one-minute speech synthesis at low latency.
- WaveVAE compresses 24 kHz audio to 25 Hz latents, making diffusion faster and reconstruction near-lossless.
- Supports Chinese/English code-switching and accent intensity control via tunable
p_wandt_wparameters. - WaveVAE encoder withheld from public release for security; a community fork on Hugging Face provides the full encoder/decoder.
MegaTTS3 is ByteDance's latest open-source text-to-speech system, built in collaboration with Zhejiang University. At its core, it is a lightweight diffusion transformer-based TTS model designed to clone voices from short audio samples and generate natural-sounding speech in multiple languages. The repo has already crossed 6,000 stars, and for good reason: it hits a quality bar that most models twice its size struggle to match.
The alignment problem nobody talks about
Most TTS systems fall into one of two camps, and both have a catch. Models without explicit speech-text alignment modeling exhibit less robustness, especially for hard sentences in practical applications, while predefined alignment-based models suffer from naturalness constraints of forced alignments. Forced alignment means you rigidly map each phoneme to a fixed time window , the output sounds correct but stiff, like a GPS voice reading directions.
MegaTTS3 introduces a sparse alignment algorithm that guides the latent diffusion transformer. It provides sparse alignment boundaries to reduce the difficulty of alignment without limiting the search space, thereby achieving high naturalness. Think of it as giving the model rough road signs rather than a turn-by-turn script , it knows roughly where each word should land, but has the freedom to phrase it naturally.
How it's built
The authors use a two-stage architecture, beginning with a WaveVAE-based speech compression model followed by a latent diffusion transformer for speech generation. The WaveVAE is the secret weapon here: it compresses 24 kHz audio into a compact 25 Hz latent representation, making the diffusion process operate on a much smaller, cleaner signal rather than raw waveforms or mel-spectrograms.