Skip to main content
Back

Cambridge Researchers Beat Fixed Audio-Visual Fusion With 94.2% Precision on BBC Archives

A new query-adaptive framework detects whether audio or video is actually useful per query before fusing them, beating fixed multimodal fusion by 4.2 points on a 12,000-video broadcast archive.

Paper, Full Access
  • Core finding: Always fusing audio and video for person retrieval hurts precision when one modality is absent -- adaptive selection beats fixed fusion by 4.2 points P@1.
  • Method: Cross-modal rank consistency (how much audio and face top-lists agree) is used as a feature to classify which modalities are active per query, achieving 89% detection accuracy.
  • Results: Adaptive system hits 94.2% P@1 on BBC Rewind, vs. 90.0% for fixed fusion and 93.4% for face-only baseline.
  • Dataset: BBC Rewind -- 12,594 broadcast videos (409 hours) from 1948-1979 Northern Ireland, a genuinely in-the-wild archive with variable audio quality.
  • Zero-shot: All embedding models run zero-shot; no labeled training data from the target archive is required for the modality detector.
  • Limitation: Evaluated on a single archive subset; binary modality decisions and ~11% detector error rate leave room for improvement toward the 96.6% oracle ceiling.

Most multimodal retrieval systems operate on a simple assumption: more signals equal better results. Feed in a face, feed in a voice, fuse the scores, and rank. It sounds reasonable -- until you try it on real-world footage where a speaker is off-camera, or a face appears with no accompanying speech. A new paper from researchers at Cambridge, Queen's University Belfast, and the University of Surrey challenges this assumption head-on, and the results are striking.

The paper, "To Be Multimodal or Not to Be: Query-Adaptive Audio-Visual Person Retrieval via Active Modality Detection", proposes a system that first asks which modalities are actually present in a query before deciding how to combine them. The answer, it turns out, matters enormously.

The problem with always fusing

When retrieving a person from a video archive by voice and face, the real question is whether the system should be multimodal at all -- because in real-world broadcast archives, a target may be heard but unseen, seen but unheard, or both. This is fundamentally different from the controlled benchmarks most retrieval systems are trained and evaluated on, where both modalities are reliably present.

Fusing scores from an absent modality injects noise, degrading precision below the best unimodal system. In other words, a system that always fuses audio and video can actually perform worse than one that just uses the face -- or just uses the voice -- when one of those signals contains no useful information about the target person.

Prior work on the MVSE project demonstrated that multimodal fusion consistently outperforms single-modal retrieval, but that framework treats all modalities equally during fusion, assuming each is informative for every query. When a person is only heard or only seen, the uninformative modality injects noise, degrading retrieval.

The key insight: agreement as a signal

The core idea is elegant. When both modalities are active, files retrieved by one also score highly on the other -- but this agreement breaks down when a modality is absent. Think of it this way: if you search for a person by voice and the top-ranked results by voice are also the top-ranked results by face, that's strong evidence that both the voice and face are genuinely present and informative. If the two ranked lists look completely different, one of the modalities is probably just producing noise.