Skip to main content
Back

VieNeu-TTS v3 Turbo Runs Vietnamese Voice Cloning on a Consumer GPU

A from-scratch Vietnamese TTS hits 48 kHz audio, instant voice cloning, and real-time CPU inference with an OpenAI-compatible streaming API.

  • VieNeu-TTS v3 Turbo ships 48 kHz Vietnamese and English TTS with instant voice cloning, open-sourced under Apache 2.0 on GitHub
  • Trained from scratch on 10,000+ hours of bilingual audio with 25 built-in preset voices across Northern, Central, Southern dialects
  • Real-time CPU inference via torch-free ONNX install; RTF 0.5 on a 12th-gen Core i5, 0.35 with int8 quantization
  • GPU streaming on RTX 3060 hits 115 ms first-audio and 16 concurrent streams via a single-replay CUDA graph
  • OpenAI-compatible /v1/audio/speech endpoint drops into Pipecat, LiveKit, and the OpenAI SDK with Docker profiles for CPU and GPU
  • Proprietary v4 with near-original cloning fidelity is held back from open source over misuse concerns; v3 Turbo remains the latest open release

Vietnamese text-to-speech has fewer capable open models than English, and many available systems pair synthetic-sounding output with demanding hardware. The VieNeu-TTS project, now above 3,000 GitHub stars, has released v3 Turbo with 48 kHz output, voice cloning from a clip lasting 3 to 8 seconds, and an OpenAI-compatible streaming API. The current open release uses the Apache 2.0 license.

According to the project documentation, its bilingual architecture was trained from scratch on more than 10,000 hours of Vietnamese and English audio. Reported tests put time to first audio at about 115 milliseconds, batched throughput at 50 to 90 times real-time on an RTX 3060, and CPU generation at roughly 1.6 to 1.8 times real-time. Those characteristics support local voice agents, offline applications, and deployments that keep speech data away from cloud services.

What v3 Turbo ships

The model combines the MOSS audio codec, which represents speech as model-friendly audio tokens, with a custom phonemizer called sea-g2p that maps written text to pronunciation units.

  • 48 kHz audio output, increased from 24 kHz in earlier versions
  • 25 preset voices, including 10 labeled “Editor’s Picks” for stable generation without a reference clip
  • Experimental emotion and non-verbal cues embedded in text, including [cười] for a chuckle, [thở dài] for a sigh, and [hắng giọng] for clearing the throat
  • Batched generation with up to 32 items per batch
  • A multi-speaker Conversation mode that batches an entire script across speakers
  • Voice cloning from a 3- to 8-second recording, with automatic denoising of the reference audio
  • Low-rank adaptation, or LoRA, for memory-efficient fine-tuning of a voice or reading style on a consumer GPU with as little as 6 GB of VRAM

CPU inference without PyTorch

VieNeu-TTS keeps its minimal CPU installation free of PyTorch by running through ONNX Runtime and avoiding any Torch import. On a CUDA host, the same API automatically switches to PyTorch and enables batched inference. Installing vieneu with pip install vieneu therefore provides a smaller dependency chain for local and edge deployments.

Project-reported benchmarks on an RTX 3060 and 12th-generation Core i5
Mode RTF Notes
GPU, batched 0.011 to 0.02 30 sentences totaling 130 seconds of audio generated in 1.5 to 2.3 seconds
GPU, single sentence 0.10 Short requests are limited by GPU launch overhead
CPU, v3 Turbo FP32 0.55 to 0.62 48 kHz default output
CPU, v3 Turbo INT8 0.35 to 0.37 Requires AVX-512 VNNI or AVX-VNNI instructions
CPU, v3 Nano at 8 steps 0.11 24 kHz output with lower speech quality

Real-time factor, or RTF, divides generation time by the duration of the resulting audio. An RTF of 0.02 means the model takes 0.02 seconds to generate one second of speech, equivalent to 50 times real-time. Values below 1 can generate audio faster than it plays.

The INT8 CPU path requires VNNI instructions, and the project warns that older processors may produce garbled audio. Text length, concurrency, processor support, audio playback, and surrounding application code will also affect end-to-end latency.

One CUDA graph per frame

Continuous batching groups active requests so the GPU can advance several streams together, while VieNeu-TTS replays one fused CUDA graph for each generated audio frame. A CUDA graph records a fixed sequence of GPU operations to reduce repeated launch overhead. Here, it combines the acoustic decoder, sampling, repetition penalty, and model backbone step without requiring torch.compile or a C++ toolchain.

Each batch size incurs about 500 milliseconds of capture time on its first call, after which the graph can be reused. Servers can invoke warm_fused() during startup to pay that cost before accepting traffic.

In the project’s RTX 3060 tests, 16 concurrent streams produced a median time to first audio of 185 milliseconds and peak VRAM use of 1.1 GB. A newly arriving request started in 134 milliseconds while 15 other streams were playing. At 32 streams, per-stream RTF reached 0.93, leaving little capacity for scheduling delays or workload spikes.

Deployment capacity exceeds the simultaneous stream count because a stream consumes a slot only while the system generates a spoken reply. The project estimates that 16 concurrent streams can serve roughly 45 to 80 active voice-chat users, depending on response length and turn timing.

OpenAI clients can switch URLs

The bundled server exposes POST /v1/audio/speech with the request shape used by OpenAI’s text-to-speech endpoint. OpenAI SDK clients, Pipecat, and LiveKit Agents can connect by changing base_url. A minimal Python client looks like this:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="x",
)

with client.audio.speech.with_streaming_response.create(
    model="vieneu-v3-turbo",
    voice="Mai Anh",
    input="Xin chào!",
    response_format="pcm",
) as response:
    for chunk in response.iter_bytes(4096):
        play(chunk)

The server also provides endpoints for listing models and voices, while POST /v1/voices creates a cloned voice from an uploaded recording. Docker Compose profiles named api-gpu and api-cpu cover both deployment paths.

Nano targets slower CPUs

VieNeu v3 Nano is a preview build for devices that cannot sustain the Turbo CPU path. Its 48-million-parameter ONNX model uses flow matching, a generative method that iteratively transforms noise into speech. Nano produces 24 kHz audio, includes 11 preset voices and voice cloning, and runs about three times faster than Turbo FP32 on CPU. The project lists a 282 MB download and a load time of about three seconds.

Nano trades speech quality for that speed, with the largest reported degradation on English and code-switched text. Turbo remains the stronger choice when pronunciation and speaker consistency take priority over CPU throughput.

Why v4 stays API-only

VieNeu v4 remains proprietary and available only through the VieNeu API because, according to the project, its near-original speaker similarity from a short reference clip creates greater impersonation and fraud risks. The open repository continues to identify v3 Turbo as its latest public release, with possible v3.x updates still planned.

Self-hosted deployments therefore retain v3 Turbo’s lower voice-cloning ceiling. Services that accept uploaded reference recordings should still require speaker consent and add access controls, rate limits, audit logs, and a process for reviewing abuse reports.

From voice agents to air-gapped apps

VieNeu-TTS v3 Turbo fits projects that need Vietnamese or bilingual speech generation with local control over inference, voices, and deployment infrastructure.

  • Voice agents and conversational applications that need low first-audio latency and an OpenAI-compatible endpoint
  • Audiobook, podcast, and dubbing workflows that can batch multi-speaker scripts through Conversation mode
  • Offline, on-device, and air-gapped systems that cannot send text or audio to a hosted TTS provider
  • Custom voices and reading styles fine-tuned through LoRA on a consumer GPU

The Hugging Face weights are available for download, and the repository README links to a Colab notebook for testing the model before installation. Production evaluations should measure Vietnamese pronunciation, English and code-switched output, voice similarity, cue handling, latency, and memory use with the intended hardware and workload.