Tencent's AuK Merges 16 Speech Tasks Into One Open-Source Model
Tencent open-sourced AuK and AuK-Flash, a 1.5B unified speech model that handles TTS, editing, enhancement, and separation from plain-language instructions.
Takeaways
- Tencent open-sourced AuK, a 1.5B unified speech generation and editing model under MIT license.
- AuK-Flash is a distilled 4-step variant offering 4.5x wall-clock speedup with no CFG.
- One natural-language interface handles TTS, voice cloning, content editing, accent removal, denoising, and source separation.
- Built on Qwen2.5-Omni-3B encoder, joint speech-audio-music VAE, and hybrid MMDiT plus DiT rectified-flow transformer.
- Trained on ~3.03B instruction-audio pairs across 1.95M hours spanning five task families.
- Ships with GitHub code, ComfyUI nodes, vLLM-Omni serving, MLX support, and a browser demo Space.
Tencent has released AuK and the distilled AuK-Flash on Hugging Face under an MIT license. The 1.5-billion-parameter generator handles text-to-speech, voice cloning, audio editing, enhancement, and source separation through a shared instruction interface. Developers can describe a task in English or Chinese and attach reference audio when the operation requires it.
| Component | Details |
|---|---|
| Generator | 1.5 billion parameters |
| Conditioning encoder | Qwen2.5-Omni-3B |
| Training data | About 3.03 billion instruction-audio pairs covering 1.95 million hours |
| Task coverage | 16 capabilities across five task families |
| Flash inference | Four sampling steps without classifier-free guidance |
| License | MIT |
Many speech jobs, one interface
The project groups generation and editing capabilities that commonly require separate models, preprocessing pipelines, and audio formats. Zero-shot cloning uses a short reference recording to synthesize new speech in the same voice, while instruction-based editing modifies an existing recording according to a text prompt.
Supported operations include:
- Generating speech from text with or without a reference voice
- Replacing spoken words while preserving surrounding audio
- Changing timbre, tone, emotion, or speaking style
- Reducing a regional accent while retaining speaker identity and content
- Adding coughs, laughter, breaths, and other nonverbal sounds
- Removing noise and improving recording quality
- Extracting a speaker from a mixed recording
- Separating music stems
- Converting whispered speech into normal voicing
A shared interface can reduce the glue code required for workflows that combine several operations. A pipeline for editing dialogue, removing background noise, and inserting a timed sound can send those requests to one model instead of coordinating multiple task-specific repositories.