UniVoice Beats Singing Specialists While Matching Speech-Only AI at 329M Params
UniVoice uses factorized conditioning and a learned null melody token to let a single 0.3B model match dedicated TTS quality while crushing prior unified singing baselines
Takeaways
- One model, two modalities: UniVoice is a 0.3B model that generates both natural speech and controllable singing from a single DiT backbone.
- The core fix: Factorized conditioning separates content, melody, and timbre signals so speech and singing gradients never compete over shared melody parameters.
- Null melody token: For speech, a learned null token replaces MIDI melody input, proven to approximate the mathematically optimal melody-marginalized generation.
- Speech quality matches specialists: 5.26% phoneme error rate, on par with F5-TTS (5.21%) and CosyVoice3 (5.30%) despite using far less speech data.
- Singing quality crushes prior unified models: 16.22% singing PER vs. Vevo1.5's 45.07%, while also beating dedicated singing model Soul-X-Singer (26.22%).
- New benchmark released: UniSinging-Eval covers 900 samples across 12 musical styles; inference code and model weights to be open-sourced.
Building a single model that can both speak and sing sounds straightforward until you look at what each task actually needs. Speech synthesis wants freedom: the model should infer natural rhythm and intonation from language context alone. Singing synthesis wants control: the model must follow an explicit melody, hitting specific pitches at specific times. Naively training one model on both tasks forces these two demands to fight over the same parameters, and quality on both sides suffers. UniVoice, a new paper from Giant Network and the Shanghai Conservatory of Music, proposes a clean architectural fix for this conflict.
The gradient war nobody talks about
The root problem is what the authors call a representation conflict. When a single melody encoder is shared across speech and singing training, the two tasks push its parameters in opposite directions. Speech examples send gradients saying "melody is irrelevant, output a constant"; singing examples send gradients saying "encode these MIDI notes faithfully." These gradient directions are negatively correlated, meaning each task actively degrades the other's melody representation. The authors prove this formally and show the magnitude of the eliminated conflict is proportional to the negative inner product of those opposing gradients.

Factorize, don't fight
UniVoice is a unified speech and singing voice generation framework based on conditional flow matching that factorizes the condition into content, melody, and timbre, encoded by modality-appropriate encoders and consumed by a shared Diffusion Transformer (DiT) backbone. The key insight is to separate the conditioning signals so they never compete:
- Content: Phoneme sequences processed by a ConvNeXt encoder, shared across both modalities using a unified IPA phoneme system.
- Melody: For singing, a Conformer encoder converts MIDI note sequences into frame-level melodic embeddings. For speech, the melody slot is filled by a learned null token that signals "infer prosody from context."
- Timbre: A 3-10 second reference audio clip is encoded by the same VAE used for target audio, enabling zero-shot voice cloning without speaker embeddings or fine-tuning.