Skip to main content
Back

Northeastern's MemoryCard Beats Frame Retrieval With 21.8% Accuracy Gain on Long Videos

MemoryCard organizes long videos into topic-aware memory units, boosting VLM question-answering accuracy by up to 21.8% without increasing token costs

  • Core idea: MemoryCard pre-processes long videos into topic-segmented "Memory Cards" combining keyframes, event summaries, and transcripts for retrieval.
  • Performance: Achieves up to 21.8% relative accuracy improvement on long-video QA benchmarks (Video-MME, MLVU, LongVideoBench) at the same token budget.
  • Key shift: Moves from frame-level to event-level evidence units, enabling coherent semantic retrieval instead of pixel-level frame matching.
  • Reusable index: Memory Card banks are built once per video and reused across all queries, amortizing preprocessing cost.
  • Plug-and-play: Works as an augmentation wrapper around any existing VLM with no fine-tuning required.
  • Open source: Full code available at github.com/NEUIR/MemoryCard; paper at arxiv.org/abs/2606.05917.

Getting a vision-language model (VLM) to answer questions about a long video is deceptively hard. The relevant evidence might be a three-second clip buried 40 minutes in, surrounded by hours of unrelated footage. A new paper from Northeastern University and Tsinghua University introduces MemoryCard, a framework that rethinks how video evidence is organized before it ever reaches the answering model, and the gains are substantial.

The frame-centric dead end

Long-video question answering remains challenging for VLMs, as answer-relevant evidence is often sparse, transient, and temporally dispersed across lengthy video contexts. The dominant approach has been to work harder on individual frames: sample them more cleverly, compress their tokens, or pick only the ones most relevant to the query.

Unlike images, video inherently possesses additional temporal redundancy. While capturing complete temporal information typically requires at least 24 FPS, processing a 10-minute HD video at even 1 FPS still yields token sequences orders of magnitude larger than those from high-resolution images, rendering conventional transformer-based models impractical for real-world deployment. The field has responded with a wave of token compression methods, but they all share the same blind spot.

Existing frame-centric approaches improve efficiency through uniform sampling, query-aware frame selection, visual-token compression, and adaptive resolution strategies. However, they still rely on isolated and fragmented frames as the fundamental evidence units, limiting VLMs' ability to effectively capture coherent event-level semantics. A single frame cannot tell you that a character spent the last five minutes building toward a decision. That requires understanding an event as a whole.

Comparison of frame retrieval methods vs MemoryCard on a video QA example

What a Memory Card actually is

MemoryCard's core idea is to shift the fundamental unit of evidence from a frame to an