CoSync-DiT Beats Every AI Dubbing System With 16% Better Lip Sync
A new flow-matching dubbing model mimics how professional actors listen, watch, then speak, hitting state-of-the-art lip-sync on in-the-wild movie footage.
Takeaways
- CoSync-DiT proposes a three-stage flow-matching dubbing model inspired by how professional voice actors work.
- The architecture separates acoustic style, visual calibration, and text alignment across different transformer layers.
- JSAR regularization uses frame-level contrastive learning plus CTC loss to lock in timing and pronunciation.
- Beats AlignDiT by 16.89 points AVSync on CelebV-Dub and cuts WER from 20.98% to 5.59% on zero-shot movies.
- Remains stable at low sampling steps, enabling few-step inference without quality collapse.
- Code, weights, and configs promised as open-source; no repository released yet with the preprint.
Getting an AI to dub a movie is harder than it sounds. The synthesized speech has to match a target speaker's timbre, articulate the script correctly, and land every syllable on the right lip movement, all at once. A new architecture called CoSync-DiT takes a swing at all three by breaking generation into a sequence that mirrors how a human voice actor prepares a take: listen to the reference, watch the lips, then articulate.
Movie dubbing aims to synthesize speech that preserves the vocal identity of a reference clip while syncing to the lip movements in a target video. Prior methods either fail at precise lip-sync because they force alignment at the phoneme-duration level, or they achieve tighter timing but degrade timbre and pronunciation when the reference audio is noisy. Both failure modes have blocked deployment in real film post-production.
Where previous dubbing systems break
Prior work splits into two camps. The explicit-alignment group (StyleDubber, ProDubber, InstructDubber) predicts an integer duration for each phoneme and expands text to match. That guarantees clean pronunciation but forces speech into rigid integer intervals that never quite land on real lip motion. A newer contrastive variant like FlowDubber still leans on an external forced aligner (Montreal Forced Aligner) to extract ground-truth phoneme boundaries, which locks pronunciation into predefined intervals and damages naturalness.
The implicit-alignment camp, led by AlignDiT, dropped the aligner and instead fused text with masked audio and visual features via cross-attention at every transformer layer. Forcing text to align to everything simultaneously makes the model buckle under interference from the early frames of the reference audio, especially on in-the-wild footage, and applying text cross-attention uniformly across all layers muddies the alignment dynamics.