Skip to main content
Back

UNSW and Dolby's TAPF Beats Audio-Visual Fusion Without Losing Sound Quality

A new fusion strategy adds visual context to audio tokenizers without wrecking reconstruction quality, resolving a long-standing trade-off in omni-modal systems.

Paper, Full Access
  • TAPF is the first video-enhanced audio tokenizer that preserves reconstruction fidelity while improving understanding.
  • Fusing before quantization avoids gradient conflict at the discrete bottleneck caused by straight-through estimators.
  • Knowledge distillation beats contrastive learning in discrete tokenizers; contrastive collapses at high fusion weights.
  • Dynamic temporal windows sized by visual salience concentrate tokens on high-information moments under compression.
  • At 50 tokens/sec, TAPF matches audio-only models running at 400 tokens/sec, an 8x token efficiency gain.
  • Read the full paper on arXiv.

Audio tokenizers are the quiet workhorse inside every end-to-end speech model. They chop continuous waveforms into discrete tokens a language model can chew on, then reconstruct audio from those tokens on the way out. Existing tokenizers hit a wall on complex understanding tasks, and every attempt to fix that by adding video context has ended up making the reconstructed audio sound worse.

A new paper from researchers at UNSW Sydney and Dolby Laboratories, Timing-Aware Pre-Quantization Fusion (TAPF), claims to be the first method that fuses visual information into an audio tokenizer without paying a reconstruction tax. The core finding: where and how you fuse matters far more than which fancy loss you pick.

The trade-off nobody could dodge

If you want a model that both understands audio and generates it, you need discrete tokens. Audio alone is often ambiguous, so researchers have tried piping in visual context. Semantic understanding improves, but reconstruction quality invariably degrades. That kills the approach for omni-modal systems like Moshi, Step-Audio, or Qwen2.5-Omni that need natural-sounding speech synthesis.

The authors argue this degradation comes from three under-examined design choices: where the fusion happens in the pipeline, what alignment objective you use, and how you handle time. Get those wrong and the discrete quantization bottleneck turns competing gradients against each other.

Three findings that reshape the design space

The paper runs a systematic sweep across fusion strategies before proposing anything new. The results are unusually clean:

  • Fusion location: Fuse before the vector quantizer, not inside or after it. Pre-quantization fusion lets reconstruction and alignment gradients find compromise solutions in continuous space, whereas quantization-level fusion forces competing gradients through a discrete bottleneck where they interfere.