k2-fsa's OmniVoice Clones Any Voice Across 600 Languages at 40x Real-Time Speed
k2-fsa open-sources OmniVoice, a zero-shot TTS model covering 600+ languages at 40x real-time speed with Apache 2.0 licensing
Takeaways
- k2-fsa (the Kaldi/k2 team, with Daniel Povey) open-sourced OmniVoice, a zero-shot TTS model covering 600+ languages under Apache 2.0.
- Uses a diffusion language model architecture with a Qwen LLM backbone — non-autoregressive, 40x faster than real-time (RTF 0.025).
- Trained on 581,000 hours of open-source multilingual audio — the broadest language coverage of any zero-shot TTS model.
- Three modes: voice cloning (3–10s reference clip), voice design (text attributes like "female, british accent"), and auto voice.
- Supports inline non-verbal tags like
[laughter], pinyin/CMU phoneme pronunciation overrides, and multi-GPU batch inference. - Limitations: voice design unstable on low-resource languages; ~4GB VRAM required; cross-lingual cloning introduces accent bleed.
The k2-fsa team , the research group behind Kaldi and k2, with Daniel Povey as a core contributor , has open-sourced OmniVoice, a zero-shot text-to-speech model that covers more than 600 languages. That is not a typo. Most production TTS systems top out at a few dozen languages. OmniVoice blows past that ceiling and does it without needing any fine-tuning on a target voice , just a short audio clip.
The problem with existing multilingual TTS
Most competitive TTS models today , including XTTS v2 , support somewhere between 17 and 30 languages. Scaling further has historically meant either training separate models per language or accepting severe quality degradation on low-resource languages. The standard architecture compounds this: a two-stage pipeline that first converts text to semantic tokens, then converts those to audio, introduces error propagation at every step.
At its core, OmniVoice uses a novel diffusion language model-style discrete non-autoregressive (NAR) architecture. Unlike conventional discrete NAR models that suffer from performance bottlenecks in complex two-stage pipelines, OmniVoice directly maps text to multi-codebook acoustic tokens. Cutting out the intermediate semantic stage removes a major source of quality loss at scale.
How it actually works
OmniVoice uses a diffusion language model architecture. Unlike traditional autoregressive TTS, this approach uses a Qwen LLM backbone to perform iterative unmasking/diffusion over multi-codebook discrete audio tokens. Think of it like BERT-style masked language modeling, but applied to audio tokens instead of text , the model learns to predict masked audio tokens from context, then iteratively refines the full sequence at inference time.
Two key technical innovations make this work: a full-codebook random masking strategy for efficient training, and initialization from a pre-trained LLM to ensure superior intelligibility. The LLM initialization is a significant detail , it means the model starts with strong language understanding baked in, which is what allows it to generalize across hundreds of languages without dedicated per-language training data.