Video AI Quietly Fails the Middle: a Benchmark Exposes Hidden Position Bias
A new benchmark reveals that video-understanding models give better summaries to some videos just because of where they appear in the input queue
Takeaways
- MLLMs give systematically different quality summaries to the same video depending on where it appears in a multi-video input, even with identical content.
- A new benchmark built from ActivityNet and News videos tests 9 open-source and proprietary MLLMs across Cooking, Domestic, Leisure, and News domains.
- Three new metrics -- Coverage, Directional Positional Bias (DPB), and Middle-Edge Gap (MEG) -- reveal that middle-position videos are often underserved even when overall directional bias looks small.
- Increasing visual token budget or output length does not reliably fix positional imbalance across models or domains.
- Prompt-level mitigations offer partial relief but are inconsistent; the bias is structural to transformer attention.
- The paper motivates building order-invariant multimodal architectures rather than relying on prompt workarounds.
Multimodal Large Language Models are increasingly used for video understanding, yet their reliability under multi-video inputs remains poorly understood. A new paper exposes a concrete failure mode: the quality of a per-video summary can change with the video's input slot, even when the underlying content is unchanged. In other words, swap the order of two videos and you may get a meaningfully worse summary for one of them -- not because the model understood it less, but simply because of where it sat in the queue.
The "Lost in the Middle" problem, now in video
Research has shown that large language models tend to overemphasize information at the beginning and end of a document or conversation, while neglecting the middle. This "position bias" means that if a user is relying on an LLM-powered system to process a long input, the model is more likely to attend to content at the start or end. This well-documented phenomenon in text has a name -- "lost in the middle" -- and it has now been systematically studied in the multimodal, multi-video domain for the first time.
Current MLLMs are still primarily developed for single-video inputs. When applied to multi-video settings, they typically concatenate frames from different sources into a single sequence, which can lead to training-inference mismatch, information loss under limited context budgets, and the absence of explicit mechanisms for cross-video coordination. This structural gap is exactly what this paper probes.
A benchmark built for the problem
The researchers construct a benchmark over Cooking, Domestic, Leisure, and News domains, and introduce an evaluation protocol that measures Coverage, Directional Positional Bias (DPB), and Middle-Edge Gap (MEG), separating earlier-versus-later preference from middle-position weakness. These three metrics are the paper's key methodological contribution -- each captures a different flavor of positional unfairness.