Peking University's M³Eval Reveals Most Video AI Models Barely Remember What They Watch
M³Eval is the first benchmark to test video AI memory using cognitive psychology experiments, exposing deep flaws in today's best models
Takeaways
- M³Eval is the first benchmark to systematically test memory in multimodal video models, using four tasks adapted from cognitive psychology experiments.
- All tested models, including GPT-5.4 and Gemini-3.1-Pro-Preview, perform near or below random chance on most memory tasks, far behind human baselines.
- Models fail to separate parallel video streams: attention maps become diffused in split-screen settings, causing source confusion between simultaneous videos.
- Unlike humans, models show no asymmetry between proactive and retroactive interference, revealing a fundamentally different memory architecture from biological cognition.
- Models are far worse at temporal source grounding (knowing which storyline an event belongs to) than spatial grounding (knowing which screen side it appeared on).
- The benchmark, code, and dataset are publicly available, covering 2,403 questions across 451 videos (~403 hours total).
Multimodal AI models can now watch hour-long videos and answer questions about them. But can they actually remember what they watched? A new benchmark from Peking University and the University of Wisconsin-Madison says the answer, for most models, is a resounding no. M³Eval is the first evaluation framework built specifically to probe memory in multimodal models, and it reveals a set of systematic, deeply-rooted failures that go far beyond what existing video benchmarks can detect.
The gap nobody was measuring
Existing video benchmarks primarily focus on visual perception and reasoning. While some tasks implicitly involve memory, they are not designed to isolate memory mechanisms, providing only a partial and indirect assessment. The distinction matters. A model can answer a question about a single scene by pattern-matching on visual features. Remembering that scene accurately after watching two similar videos back-to-back, or reconstructing the correct order of events from an interleaved stream, requires something fundamentally different.
Multimodal evaluation has moved past the pure image-QA era. By 2026, leading frontier models all clear 80% on MMMU-Pro, with a spread of under 3 points. The differentiating axis has moved. Video understanding is now the frontier, and within video understanding, memory is the next unsolved layer.
Psychology meets video AI
M³Eval is grounded in cognitive psychology, where memory is studied through carefully constructed stimuli that isolate specific mechanisms. The team adapted four classic experimental paradigms from human memory research into video-based QA tasks:
- Divided Attention: Two semantically similar videos are shown side by side simultaneously. The model must keep track of which event happened in which video, a task humans find taxing but manageable.
- Memory Interference: Two similar videos are concatenated sequentially. The model is asked about one of them. This tests whether earlier or later content "bleeds into" recall of the target, a phenomenon called proactive and retroactive interference in psychology.