Sakana AI's MLR Catches 73% of Planted Errors That Stump Other AI Reviewers
A new TMLR paper from Sakana AI researchers proposes a three-agent reviewer that catches 73% of planted errors in main claims.
Takeaways
- New TMLR paper from Sakana AI researchers introduces a benchmark and framework for LLM-assisted peer review focused on error detection.
- Multi-Layered Review uses three agents (Appendix, Literature, Review) over Claude Haiku and Sonnet models.
- Contradiction Benchmark plants 1,164 errors across 257 papers from ACL, AISTATS, CVPR, ICML, NeurIPS.
- MLR catches 73.43% of main-claim contradictions with 4-review ensemble, versus 14.81% for best baseline.
- On real withdrawn papers, exact-match detection drops to 16.11%, exposing synthetic-vs-real gap.
- Paper on arXiv and OpenReview.
Researchers from Sakana AI and the National University of Singapore have built a benchmark that tests whether large language model reviewers can find specific errors in research papers. Their paper, Beyond Imitation: A Framework and Benchmark for LLM-Assisted Peer Review, was accepted at Transactions on Machine Learning Research. It also introduces Multi-Layered Review, an agentic system designed to assist human reviewers.
Across 1,164 planted contradictions in 257 papers, a four-run Multi-Layered Review ensemble detected 73.43% of errors affecting the central claim, compared with 14.81% for the strongest prior system. Performance dropped on documented withdrawal issues, where the system found the exact problem in 16.11% of cases and a substantially similar concern in 26.07%. The gap shows both the value of controlled evaluation and the distance between benchmark performance and dependable real-world review.
Similarity cannot verify a review
Rising submission volumes and limited expert time have driven interest in automated review assistance. Existing evaluations often compare generated reports with human-written reviews, using textual overlap or agreement with reviewer scores. Those measures are noisy because careful reviewers can identify different valid concerns, and fluent models can reproduce academic review conventions without checking whether a paper’s claims follow from its evidence.
The authors isolate contradiction detection, one of peer review’s most labor-intensive tasks. A contradiction occurs when a claim conflicts with another claim, method, result, or piece of evidence in the same paper. Planting a known conflict creates a concrete target, allowing evaluators to measure whether a generated review identified the defect.
The benchmark covers verification rather than the full scope of peer review. It does not directly measure novelty assessment, field-specific judgment, clarity, ethical concerns, or the value of a reviewer’s suggested revisions.