Skip to main content
Back

WeChat Researchers Built F2RVLM to Find Buried Chat Fragments Across Thousands of Turns

A new retrieval framework tackles the messy reality of long group chats by finding coherent multi-message, multi-image fragments instead of isolated utterances

Paper, Full Access
  • New task defined: Fine-grained Fragment Retrieval (FFR) retrieves coherent multi-message, multi-image conversation fragments, not just isolated utterances.
  • Two systems built: F2RVLM handles single-dialogue retrieval; FFRS (Fragment Embedding Model + F2RVLM reranker) scales to large corpora.
  • RL-trained model: F2RVLM uses GRPO reinforcement learning with multi-objective rewards and difficulty-aware curriculum sampling for coherent fragment outputs.
  • New benchmark: MLDR dataset averages 25.45 turns per dialogue -- the longest multimodal dialogue retrieval dataset to date -- plus a real WeChat test set averaging 75.38 turns.
  • Strong results: F2RVLM and FFRS consistently outperform leading VLMs including GPT-4o on both in-domain and real-world benchmarks.
  • Open release: Code and dataset available at f2rvlm.github.io.

Anyone who has tried to find a specific conversation thread in a long group chat knows the pain: you remember the gist of what was said, but the relevant messages are scattered across dozens of turns, mixed with images, and buried under unrelated tangents. Standard search tools return individual messages in isolation, stripping away the context that made them meaningful in the first place. A new paper from researchers at WeChat's parent organization tackles this head-on with a framework called Fine-grained Fragment Retrieval (FFR) -- a system designed to find coherent, multi-message, multi-image conversation fragments from long-form dialogues.

The gap that existing retrieval misses

Traditional dialogue retrieval aims to select the most appropriate utterance or image from recent dialogue history, but these approaches often fail to meet users' actual needs for revisiting semantically coherent content scattered across long-form conversations. Think of it like the difference between a search engine returning a single sentence versus returning the whole relevant paragraph. Even leading Vision-Language Models (VLMs) struggle here -- they often retrieve incoherent utterance-image fragments because, while optimized for generating responses from visual-textual inputs, they lack explicit supervision to ensure semantic coherence within retrieved fragments.

The problem is compounded by scale. Real-world WeChat conversations reflect authentic user interactions with noisy inputs, informal expressions, and frequent topic shifts, posing greater challenges for robust retrieval than any clean benchmark can simulate. Retrieving a fragment that spans, say, a planning discussion with shared photos requires understanding topical boundaries, not just keyword overlap.

Comparison of traditional utterance retrieval vs fine-grained fragment retrieval approaches

Two problems, two systems

The paper defines FFR and then splits it into two distinct settings, each with its own solution:

  • FFR within a Single Dialogue: Given one long conversation, find the fragment most relevant to a query. The model here is F2RVLM, a generation-based retrieval model that reasons over the full dialogue and outputs the relevant span.
  • FFR within a Dialogue Corpus: Given a large collection of dialogues (think: an entire organization's chat history), find the right fragment across all of them. This is handled by