OpenBMB's VoxCPM2 Clones Voices Across 30 Languages Without Speech Tokenizers
VoxCPM2 is a fully open-source 2B-parameter TTS model covering 30 languages, voice cloning, and natural-language voice design — no discrete tokenizer required
Takeaways
- VoxCPM2 is a fully open-source 2B-parameter TTS model from OpenBMB/Tsinghua that covers 30 languages and 9 Chinese dialects without any discrete speech tokenizer.
- It unifies zero-shot voice cloning, natural-language voice design, style-controllable cloning, and continuation cloning in a single backbone with shared weights.
- The asymmetric AudioVAE V2 encodes at 16 kHz and decodes at 48 kHz, achieving implicit super-resolution while keeping the autoregressive sequence compact at 6.25 Hz.
- On a 30-language internal benchmark it achieves 1.68% average WER/CER; on MiniMax-MLS-Test it leads speaker similarity on 22 of 24 languages.
- Inference runs at 0.13 RTF (7x real-time) on a single RTX 4090 with ~8 GB VRAM using the Nano-vLLM backend.
- Weights, fine-tuning code, and inference tools are released under Apache 2.0 at GitHub and Hugging Face.
VoxCPM2 is a bold bet against the dominant approach in modern speech synthesis. While nearly every major TTS system today , from CosyVoice to Qwen3-TTS , converts audio into discrete tokens before generating speech, VoxCPM2 skips that step entirely. The result is a single 2B-parameter model that handles 30 languages, zero-shot voice cloning, natural-language voice design, and style-controllable synthesis, all without ever touching a discrete speech tokenizer.
Why tokenizers are a bottleneck
The mainstream TTS pipeline works like this: encode audio into discrete tokens using a neural codec (like EnCodec or RVQ), train a language model to predict those tokens, then decode back to a waveform. It works well, and it inherits LLM-style scaling properties. But there's a catch: quantization throws away fine-grained acoustic detail. To compensate, most high-quality systems add a second-stage diffusion or flow-matching decoder to restore what was lost , creating a fragmented pipeline where semantic planning and acoustic rendering are optimized separately.
Although effective in achieving strong perceptual quality, this decoupled design fragments high-level semantic planning from low-level acoustic rendering, preventing end-to-end joint optimization. Moreover, overall performance remains heavily dependent on the modeling capacity of the intermediate discrete speech tokenizer. VoxCPM2's predecessor, VoxCPM, proposed a different path: model continuous speech representations directly, but use a hierarchical internal structure to keep semantic and acoustic concerns cleanly separated.
The architecture: a hierarchy without hard splits
VoxCPM introduced a hierarchical backbone consisting of a Text-Semantic Language Model (TSLM), a differentiable semi-discrete bottleneck based on Finite Scalar Quantization (FSQ), and a Residual Acoustic Language Model (RALM). The TSLM primarily captures high-level semantic and prosodic structures, the FSQ bottleneck compresses them into a stable skeleton, and the RALM recovers fine-grained acoustic details. A Local Diffusion Transformer (LocDiT) then generates the actual continuous latent patches from all of this context.

Think of it like a two-level plan: the TSLM decides what to say and roughly how (prosody, rhythm), the FSQ bottleneck creates a compressed but differentiable "sketch," and the RALM fills in the acoustic texture. The whole thing is trained end-to-end , no external tokenizer, no frozen codec, no separate acoustic model.
VoxCPM2 scales this foundation in three directions:
- Capability: VoxCPM2 unifies four user-facing capabilities within a single backbone: basic TTS supporting multilingual and cross-lingual synthesis, natural-language voice design that generates entirely new voices from free-form text descriptions without any reference audio, controllable cloning that clones a speaker from a short reference audio while following style instructions, and continuation-based cloning for high-fidelity audio continuation from a paired reference audio and its transcript.