Zyphra's ZONOS3-Mini Clones Voices in 43 Languages Directly on Your Phone
Zyphra's new open-source TTS model runs real-time voice cloning on phones and laptops with 100M active parameters and Apache 2.0 weights.
Takeaways
- Zyphra released ZONOS3-Mini, an Apache 2.0 TTS with 100M active and 800M total MoE parameters.
- Runs in real time on-device: under 800 ms time-to-first-audio on a Pixel 9, under 450 ms in M3 MacBook browser.
- Supports 43 languages with zero-shot voice cloning from a 5-10 second reference clip.
- Word-level control over emotion, pace, projection, whispering, and delivery for games and narration.
- New ZAC-1 codec drops speech tokens from 774 to 150 per second; trained on 112 H100s in 2.5 days.
- Open runtimes for WebGPU, CUDA, MLX, and ggml; SGLang-Omni serves an OpenAI-compatible endpoint.
Zyphra has released ZONOS3-Mini, a text-to-speech model that clones voices from short recordings, supports 43 languages, and applies emotion and delivery controls to individual text spans. Its weights use the Apache 2.0 license, its runtimes are open source, and local inference works on phones, laptops, and browser-based applications.
Local execution removes the network latency, usage fees, and audio transfer associated with a hosted speech API. Zyphra reports time to first audio below 800 milliseconds on a Pixel 9 and below 450 milliseconds in a browser on an M3 MacBook Air, with both devices generating audio at least as fast as playback.
How 800 million parameters fit local inference
ZONOS3-Mini is an 11-layer language model built from Gated DeltaNet sequence mixers and two interleaved global-attention layers. Gated DeltaNet handles long sequences with lower computational cost than full attention, while the global layers can draw information from the entire sequence. Full AttnRes connections let each layer combine outputs from earlier layers instead of relying only on the immediately preceding state.
The latent mixture-of-experts backbone contains 64 experts and uses top-2 routing from the first layer onward. Each token activates about 100 million of the model’s 800 million parameters, reducing the arithmetic required for generation. All expert weights still need to remain available through memory, storage, quantization, or runtime offloading, so active parameter count does not represent the complete deployment footprint.
The model tokenizes text with SuperBPE and predicts ten streams of 4,096-way audio codes at 15 frames per second. A causal, multi-resolution inverse short-time Fourier transform decoder reconstructs those codes incrementally as 44.1 kHz audio.
Zyphra also replaced the Descript Audio Codec with Zyphra Audio Codec 1, or ZAC-1. The new codec reduces the representation from 774 to 150 speech tokens per second, an approximately fivefold reduction in the sequence the language model must process. Combined with sparse expert routing, the shorter sequence makes interactive local generation practical.
Latency from a Pixel to an H100
| Hardware | Runtime context | Reported result |
|---|---|---|
| Pixel 9 | Local device | Under 800 ms to first audio; real-time generation |
| M3 MacBook Air | Browser | Under 450 ms to first audio; real-time generation |
| NVIDIA H100 | Peak-throughput serving | 457 seconds of audio per second of wall time |
Time to first audio measures how long a user waits before playback can begin, while real-time generation means synthesis keeps pace with playback. The H100 number measures peak aggregate throughput and should not be read as single-request latency; production results will depend on batch size, sequence length, precision, and concurrency.
Training in 2.5 days
Zyphra reports training ZONOS3-Mini on 112 NVIDIA H100 GPUs for 2.5 days. Post-training used Direct Preference Optimization and Group Relative Policy Optimization, two methods commonly associated with language-model alignment. DPO learns from preferred and rejected outputs, while GRPO compares groups of candidates against reward signals. Zyphra applied them to improve speaker similarity and intelligibility.
One benchmark, two quality signals
ZTTS2-Eval, Zyphra’s accompanying benchmark, combines FLEURS-R read speech in 35 languages with VoxBlink2 recordings collected in less controlled conditions across 20 languages. Its composite quality score assigns equal weight to standardized speaker similarity and word error rate, subtracting the latter because lower transcription error indicates clearer speech.
| Measurement | ZONOS3-Mini result | Interpretation |
|---|---|---|
| Composite score | 66.2 | Speaker-embedding mode |
| Word error rate | 7.79 | Clean subset; lower is better |
| Speaker similarity | 83.4 | Clean subset; higher is better |
Zyphra’s evaluation places ZONOS3-Mini ahead of the tested open models, including Fish S2 Pro, and several proprietary systems on the benchmark’s composite score. Because Zyphra designed the benchmark and reported the results, independent testing remains necessary. The published audio samples provide material for direct listening comparisons across systems and languages.
Control reaches individual text spans
The conditioning system exposes controls for identity, acoustic properties, and delivery:
- Clone a voice from a 5-to-10-second reference recording or a single 192-dimensional speaker embedding.
- Create a character voice by specifying age, accent, and vocal texture without reference audio.
- Select among 10 loudness buckets measured in LUFS.
- Choose among 10 bandwidth buckets measured in hertz.
- Set speaking rate across 30 buckets measured in UTF-8 bytes per second.
- Apply emotion, pace, projection, liveliness, and whispering controls to individual word spans.
- Generate as much as 10 minutes of audio in one pass.
A minimal Python path
After installing the project package and loading a reference recording, the core Python workflow creates a speaker embedding, builds the conditioning input, generates codec tokens, and decodes them into a waveform tensor:
import torchaudio
from zonos.model import Zonos
from zonos.conditioning import make_cond_dict
wav, sr = torchaudio.load("reference.wav")
model = Zonos.from_pretrained("Zyphra/ZONOS3-Mini", device="cuda")
speaker = model.make_speaker_embedding(wav, sr)
conditioning = make_cond_dict(
text="Hello, world!",
speaker=speaker,
language="en-us",
)
codes = model.generate(model.prepare_conditioning(conditioning))
audio = model.autoencoder.decode(codes)[0].cpu()
Five paths to deployment
| Runtime | Target | Use |
|---|---|---|
| Browser runtime | Supported browsers and local hardware | Client-side applications and interactive demos |
| CUDA | NVIDIA GPUs | Local development and GPU serving |
| MLX | Apple Silicon | Native inference on recent Macs |
| C++/ggml | CPU, Metal, or Vulkan | Portable native deployment |
| SGLang-Omni | GPU servers | Streaming API serving |
SGLang-Omni exposes an OpenAI-compatible /v1/audio/speech endpoint with streaming support, which can reduce adapter work when replacing a commercial TTS service. Teams still need to validate authentication, request fields, speaker conditioning, error handling, and streaming behavior against their existing clients. Zyphra Cloud provides a managed option for deployments that do not need self-hosting.
Products that benefit from local speech
Sub-second startup and offline execution expand the range of products that can synthesize speech without sending text, reference recordings, or generated audio through an external service:
- Games can generate improvised NPC dialogue in a consistent character voice without an API call for every line.
- Screen readers and accessibility tools can operate offline, including deployments that require languages such as Odia, Kannada, or Georgian.
- Voice assistants can keep conversations on the device, reducing network latency and exposure of recorded speech.
- Audiobook and narration systems can apply emotion and emphasis to selected spans instead of relying only on document-level settings.
- Prototypes can run locally before a team commits to hosted capacity or per-character pricing.
Voice cloning moves safeguards in-house
A 5-to-10-second cloning requirement also lowers the effort needed to imitate a person’s voice. Apache 2.0 grants broad rights to use and modify the software, but it does not establish consent to reproduce a voice or satisfy laws governing likeness, biometric data, fraud, and disclosure.
Production deployments need controls suited to their risk and jurisdiction:
- Record authorization for each cloned voice and define its permitted uses.
- Protect voice embeddings and reference recordings as sensitive data.
- Require authentication, rate limits, and audit logs for synthesis endpoints.
- Label or disclose synthetic speech where policy or law requires it.
- Monitor for impersonation, fraud, harassment, and attempts to bypass restrictions.
- Test pronunciation, speaker identity, and intelligibility separately for every supported language used in production.
A local baseline to test on real workloads
ZONOS3-Mini gives developers a permissively licensed option for multilingual voice cloning with span-level controls and several local runtimes. Deployment decisions should rest on measurements from target speakers, languages, devices, concurrency levels, and long-form inputs. Zonos Voice Studio provides listening samples, while the released runtimes allow teams to measure latency, memory use, and output quality on representative hardware.