Skip to main content
Back

Training-Free MoA Framework Beats Supervised Models by 19.6% on Multi-Document Summarization

A training-free multi-agent framework combines knowledge graphs and LLMs to summarize multiple documents, beating supervised models without any fine-tuning

Pro

Paper, Full Access
  • Training-free MDS: A new Mixture-of-Agents framework summarizes multiple documents without any fine-tuning, beating supervised baselines on 4 datasets.
  • Three specialized agents run in parallel: an Extractor (sentence scoring), a KGSum (knowledge graph with contrastive edges), and an Abstractor (iterative self-refinement).
  • Adaptive Multi-Perspective Fusion (AMF) dynamically weights each agent's output based on quality metadata, resolving conflicts before final synthesis.
  • State-of-the-art results: +19.6% ROUGE-1 over best supervised model on Multi-News; works on English and Vietnamese without language-specific tuning.
  • KGSum is the key driver: Ablation shows removing it causes the worst performance drop; contrastive relation extraction alone doubles ROUGE-2 on news data.
  • No public code yet; GPT-4.1-mini is the best backbone (~$0.017/sample on Multi-News); Gemini-1.5-flash-8b offers 10x cost savings with competitive results.

Summarizing a pile of documents is deceptively hard. Not because any single document is difficult, but because the real challenge is figuring out what multiple sources agree on, where they contradict each other, and which facts matter most across the whole collection. Most existing approaches either require massive labeled training datasets, or just dump everything into a single LLM prompt and hope for the best. A new paper accepted at Neural Computing and Applications takes a different approach: a training-free mixture-of-agents framework that orchestrates three specialized agents, each attacking the summarization problem from a different angle, and then intelligently fuses their outputs.

Why single-model summarization breaks down

Multi-Document Summarization (MDS) is the task of reading a cluster of related documents and producing a single coherent summary. Think: summarizing five news articles about the same event, or condensing a set of scientific papers into a related-work section. The difficulty is that documents often repeat information, contradict each other, and contain complex webs of entities and relationships that span across sources.

Supervised models like PEGASUS and PRIMERA can do this well, but they require large labeled datasets of (document cluster, reference summary) pairs. That makes them brittle: they struggle in low-resource languages, specialized domains, or any setting where you don't have thousands of labeled examples. Meanwhile, just prompting a single LLM with all the documents runs into context-length limits and tends to produce summaries that miss inter-document relationships entirely.

Three agents, one problem

The proposed MoA (Mixture of Agents) framework decomposes summarization into three parallel specialized pipelines, each producing its own summary and a rich metadata package describing how confident it is:

  • Extractor Agent: Scores every sentence using a hybrid of semantic similarity (via Longformer embeddings) and positional bias, then selects the top sentences while filtering near-duplicates. This gives you a factually grounded, extractive baseline.