Skip to main content
Back

Zyphra's ZONOS3-Mini Clones Voices in 43 Languages Directly on Your Phone

Zyphra's new open-source TTS model runs real-time voice cloning on phones and laptops with 100M active parameters and Apache 2.0 weights.

  • Zyphra released ZONOS3-Mini, an Apache 2.0 TTS with 100M active and 800M total MoE parameters.
  • Runs in real time on-device: under 800 ms time-to-first-audio on a Pixel 9, under 450 ms in M3 MacBook browser.
  • Supports 43 languages with zero-shot voice cloning from a 5-10 second reference clip.
  • Word-level control over emotion, pace, projection, whispering, and delivery for games and narration.
  • New ZAC-1 codec drops speech tokens from 774 to 150 per second; trained on 112 H100s in 2.5 days.
  • Open runtimes for WebGPU, CUDA, MLX, and ggml; SGLang-Omni serves an OpenAI-compatible endpoint.

Zyphra has released ZONOS3-Mini, a text-to-speech model that clones voices from short recordings, supports 43 languages, and applies emotion and delivery controls to individual text spans. Its weights use the Apache 2.0 license, its runtimes are open source, and local inference works on phones, laptops, and browser-based applications.

Local execution removes the network latency, usage fees, and audio transfer associated with a hosted speech API. Zyphra reports time to first audio below 800 milliseconds on a Pixel 9 and below 450 milliseconds in a browser on an M3 MacBook Air, with both devices generating audio at least as fast as playback.

How 800 million parameters fit local inference

ZONOS3-Mini is an 11-layer language model built from Gated DeltaNet sequence mixers and two interleaved global-attention layers. Gated DeltaNet handles long sequences with lower computational cost than full attention, while the global layers can draw information from the entire sequence. Full AttnRes connections let each layer combine outputs from earlier layers instead of relying only on the immediately preceding state.

The latent mixture-of-experts backbone contains 64 experts and uses top-2 routing from the first layer onward. Each token activates about 100 million of the model’s 800 million parameters, reducing the arithmetic required for generation. All expert weights still need to remain available through memory, storage, quantization, or runtime offloading, so active parameter count does not represent the complete deployment footprint.

Diagram of the ZONOS3-Mini text-to-speech inference pipeline
ZONOS3-Mini converts text and conditioning inputs into codec tokens, then decodes them into a 44.1 kHz waveform.

The model tokenizes text with SuperBPE and predicts ten streams of 4,096-way audio codes at 15 frames per second. A causal, multi-resolution inverse short-time Fourier transform decoder reconstructs those codes incrementally as 44.1 kHz audio.

Zyphra also replaced the Descript Audio Codec with Zyphra Audio Codec 1, or ZAC-1. The new codec reduces the representation from 774 to 150 speech tokens per second, an approximately fivefold reduction in the sequence the language model must process. Combined with sparse expert routing, the shorter sequence makes interactive local generation practical.

Latency from a Pixel to an H100

Hardware Runtime context Reported result
Pixel 9 Local device Under 800 ms to first audio; real-time generation
M3 MacBook Air Browser Under 450 ms to first audio; real-time generation
NVIDIA H100 Peak-throughput serving 457 seconds of audio per second of wall time

Time to first audio measures how long a user waits before playback can begin, while real-time generation means synthesis keeps pace with playback. The H100 number measures peak aggregate throughput and should not be read as single-request latency; production results will depend on batch size, sequence length, precision, and concurrency.

Training in 2.5 days

Zyphra reports training ZONOS3-Mini on 112 NVIDIA H100 GPUs for 2.5 days. Post-training used Direct Preference Optimization and Group Relative Policy Optimization, two methods commonly associated with language-model alignment. DPO learns from preferred and rejected outputs, while GRPO compares groups of candidates against reward signals. Zyphra applied them to improve speaker similarity and intelligibility.

One benchmark, two quality signals

ZTTS2-Eval, Zyphra’s accompanying benchmark, combines FLEURS-R read speech in 35 languages with VoxBlink2 recordings collected in less controlled conditions across 20 languages. Its composite quality score assigns equal weight to standardized speaker similarity and word error rate, subtracting the latter because lower transcription error indicates clearer speech.

Measurement ZONOS3-Mini result Interpretation
Composite score 66.2 Speaker-embedding mode
Word error rate 7.79 Clean subset; lower is better
Speaker similarity 83.4 Clean subset; higher is better

Zyphra’s evaluation places ZONOS3-Mini ahead of the tested open models, including Fish S2 Pro, and several proprietary systems on the benchmark’s composite score. Because Zyphra designed the benchmark and reported the results, independent testing remains necessary. The published audio samples provide material for direct listening comparisons across systems and languages.

Control reaches individual text spans

The conditioning system exposes controls for identity, acoustic properties, and delivery:

  • Clone a voice from a 5-to-10-second reference recording or a single 192-dimensional speaker embedding.
  • Create a character voice by specifying age, accent, and vocal texture without reference audio.
  • Select among 10 loudness buckets measured in LUFS.
  • Choose among 10 bandwidth buckets measured in hertz.
  • Set speaking rate across 30 buckets measured in UTF-8 bytes per second.
  • Apply emotion, pace, projection, liveliness, and whispering controls to individual word spans.
  • Generate as much as 10 minutes of audio in one pass.

A minimal Python path

After installing the project package and loading a reference recording, the core Python workflow creates a speaker embedding, builds the conditioning input, generates codec tokens, and decodes them into a waveform tensor:

import torchaudio
from zonos.model import Zonos
from zonos.conditioning import make_cond_dict

wav, sr = torchaudio.load("reference.wav")
model = Zonos.from_pretrained("Zyphra/ZONOS3-Mini", device="cuda")

speaker = model.make_speaker_embedding(wav, sr)
conditioning = make_cond_dict(
    text="Hello, world!",
    speaker=speaker,
    language="en-us",
)

codes = model.generate(model.prepare_conditioning(conditioning))
audio = model.autoencoder.decode(codes)[0].cpu()

Five paths to deployment

Runtime Target Use
Browser runtime Supported browsers and local hardware Client-side applications and interactive demos
CUDA NVIDIA GPUs Local development and GPU serving
MLX Apple Silicon Native inference on recent Macs
C++/ggml CPU, Metal, or Vulkan Portable native deployment
SGLang-Omni GPU servers Streaming API serving

SGLang-Omni exposes an OpenAI-compatible /v1/audio/speech endpoint with streaming support, which can reduce adapter work when replacing a commercial TTS service. Teams still need to validate authentication, request fields, speaker conditioning, error handling, and streaming behavior against their existing clients. Zyphra Cloud provides a managed option for deployments that do not need self-hosting.

Products that benefit from local speech

Sub-second startup and offline execution expand the range of products that can synthesize speech without sending text, reference recordings, or generated audio through an external service:

  • Games can generate improvised NPC dialogue in a consistent character voice without an API call for every line.
  • Screen readers and accessibility tools can operate offline, including deployments that require languages such as Odia, Kannada, or Georgian.
  • Voice assistants can keep conversations on the device, reducing network latency and exposure of recorded speech.
  • Audiobook and narration systems can apply emotion and emphasis to selected spans instead of relying only on document-level settings.
  • Prototypes can run locally before a team commits to hosted capacity or per-character pricing.

Voice cloning moves safeguards in-house

A 5-to-10-second cloning requirement also lowers the effort needed to imitate a person’s voice. Apache 2.0 grants broad rights to use and modify the software, but it does not establish consent to reproduce a voice or satisfy laws governing likeness, biometric data, fraud, and disclosure.

Production deployments need controls suited to their risk and jurisdiction:

  • Record authorization for each cloned voice and define its permitted uses.
  • Protect voice embeddings and reference recordings as sensitive data.
  • Require authentication, rate limits, and audit logs for synthesis endpoints.
  • Label or disclose synthetic speech where policy or law requires it.
  • Monitor for impersonation, fraud, harassment, and attempts to bypass restrictions.
  • Test pronunciation, speaker identity, and intelligibility separately for every supported language used in production.

A local baseline to test on real workloads

ZONOS3-Mini gives developers a permissively licensed option for multilingual voice cloning with span-level controls and several local runtimes. Deployment decisions should rest on measurements from target speakers, languages, devices, concurrency levels, and long-form inputs. Zonos Voice Studio provides listening samples, while the released runtimes allow teams to measure latency, memory use, and output quality on representative hardware.