Skip to main content
Back

OpenBMB's VoxCPM2 Clones Voices Across 30 Languages Without Speech Tokenizers

VoxCPM2 is a fully open-source 2B-parameter TTS model covering 30 languages, voice cloning, and natural-language voice design — no discrete tokenizer required

  • VoxCPM2 is a fully open-source 2B-parameter TTS model from OpenBMB/Tsinghua that covers 30 languages and 9 Chinese dialects without any discrete speech tokenizer.
  • It unifies zero-shot voice cloning, natural-language voice design, style-controllable cloning, and continuation cloning in a single backbone with shared weights.
  • The asymmetric AudioVAE V2 encodes at 16 kHz and decodes at 48 kHz, achieving implicit super-resolution while keeping the autoregressive sequence compact at 6.25 Hz.
  • On a 30-language internal benchmark it achieves 1.68% average WER/CER; on MiniMax-MLS-Test it leads speaker similarity on 22 of 24 languages.
  • Inference runs at 0.13 RTF (7x real-time) on a single RTX 4090 with ~8 GB VRAM using the Nano-vLLM backend.
  • Weights, fine-tuning code, and inference tools are released under Apache 2.0 at GitHub and Hugging Face.

VoxCPM2 is a bold bet against the dominant approach in modern speech synthesis. While nearly every major TTS system today , from CosyVoice to Qwen3-TTS , converts audio into discrete tokens before generating speech, VoxCPM2 skips that step entirely. The result is a single 2B-parameter model that handles 30 languages, zero-shot voice cloning, natural-language voice design, and style-controllable synthesis, all without ever touching a discrete speech tokenizer.

Why tokenizers are a bottleneck

The mainstream TTS pipeline works like this: encode audio into discrete tokens using a neural codec (like EnCodec or RVQ), train a language model to predict those tokens, then decode back to a waveform. It works well, and it inherits LLM-style scaling properties. But there's a catch: quantization throws away fine-grained acoustic detail. To compensate, most high-quality systems add a second-stage diffusion or flow-matching decoder to restore what was lost , creating a fragmented pipeline where semantic planning and acoustic rendering are optimized separately.

Although effective in achieving strong perceptual quality, this decoupled design fragments high-level semantic planning from low-level acoustic rendering, preventing end-to-end joint optimization. Moreover, overall performance remains heavily dependent on the modeling capacity of the intermediate discrete speech tokenizer. VoxCPM2's predecessor, VoxCPM, proposed a different path: model continuous speech representations directly, but use a hierarchical internal structure to keep semantic and acoustic concerns cleanly separated.

The architecture: a hierarchy without hard splits

VoxCPM introduced a hierarchical backbone consisting of a Text-Semantic Language Model (TSLM), a differentiable semi-discrete bottleneck based on Finite Scalar Quantization (FSQ), and a Residual Acoustic Language Model (RALM). The TSLM primarily captures high-level semantic and prosodic structures, the FSQ bottleneck compresses them into a stable skeleton, and the RALM recovers fine-grained acoustic details. A Local Diffusion Transformer (LocDiT) then generates the actual continuous latent patches from all of this context.

Architecture diagram of VoxCPM2 showing the unified sequence organization with TSLM and RALM components

Think of it like a two-level plan: the TSLM decides what to say and roughly how (prosody, rhythm), the FSQ bottleneck creates a compressed but differentiable "sketch," and the RALM fills in the acoustic texture. The whole thing is trained end-to-end , no external tokenizer, no frozen codec, no separate acoustic model.

VoxCPM2 scales this foundation in three directions:

  • Capability: VoxCPM2 unifies four user-facing capabilities within a single backbone: basic TTS supporting multilingual and cross-lingual synthesis, natural-language voice design that generates entirely new voices from free-form text descriptions without any reference audio, controllable cloning that clones a speaker from a short reference audio while following style instructions, and continuation-based cloning for high-fidelity audio continuation from a paired reference audio and its transcript.