Skip to main content
Back

SUTD's QRC Framework Teaches AI to Search the Web Before Judging Memes

A new zero-shot framework uses live web retrieval to understand memes that emerged hours ago, beating static VLMs on hate and harm detection tasks.

Paper, Full Access
  • Core problem: VLMs have static knowledge cutoffs and fail on memes referencing recent events, slang, or viral moments.
  • New framework: Query-Retrieve-Conclude (QRC) is a zero-shot pipeline that identifies knowledge gaps, retrieves live web evidence, and synthesizes it for meme understanding.
  • Three stages: Stage I generates targeted questions via reverse image search; Stage II retrieves live web answers; Stage III synthesizes clean knowledge statements for downstream models.
  • Ablation finding: Removing live web retrieval drops Hatefulness F1 to 0.62, even with a 32B model -- static parametric knowledge alone is insufficient.
  • New benchmark: Authors release a curated dataset of memes from 2024-2026 with background knowledge annotations, tested across 3 understanding datasets and 5 detection tasks.
  • No public code yet: The paper is on arXiv but no GitHub repository has been released alongside it.

Memes are one of the fastest-moving forms of communication on the internet. They remix current events, inside jokes, and cultural references into images that can go viral within hours. The problem? Multimodal memes are dynamic and often require up-to-date background knowledge for interpretation, yet existing methods often overlook such knowledge or rely on fixed parametric knowledge of pretrained models that may be incomplete, outdated, or unavailable for emerging memes. A new paper from Singapore University of Technology and Design tackles this head-on with a framework that treats the open web as a live knowledge base.

The knowledge gap that breaks modern VLMs

Vision-language models (VLMs) like GPT-4o or LLaVA are trained on massive datasets, but their knowledge has a hard cutoff. A meme referencing a geopolitical event from last week, a viral tweet from yesterday, or a new slang term coined this month is effectively invisible to them. The point of a meme pun can only be understood in relation to time-evolving real-world events -- the model either knows the context or it doesn't, and there's no mechanism to go find out.

VLM performance heavily depends on the implicit knowledge acquired during pre-training, and the reliance on implicit knowledge complicates tracing the reasoning behind classifications. This is especially painful for content moderation, where the urgency for automated systems stems from the exponential growth in multimodal content on social media platforms. The field has been stuck in a loop: train on labeled datasets that go stale, or prompt VLMs zero-shot and accept mediocre accuracy.

Flowchart showing zero-shot vs proposed QRC prompting methodology for meme understanding and detection

Query, Retrieve, Conclude

The authors introduce Query-Retrieve-Conclude (QRC), a zero-shot framework that identifies missing knowledge, retrieves open-web evidence, and synthesizes evidence-grounded background knowledge for meme understanding and detection. The key insight is to treat knowledge acquisition as a structured three-stage pipeline rather than a single model call.