Skip to main content
Back

LASR Labs Finds AI Transcripts Reveal True Power When Benchmarks Max Out

A new study shows that model transcripts from saturated benchmarks still carry capability signal, and pairwise LLM judging extracts it 5.3x better than scores alone.

Paper, Full Access
  • New paper by Hamada et al. extracts capability signal from saturated benchmarks using model transcripts.
  • Pairwise LLM judging between two correct solutions cut prediction error 5.3x versus accuracy-only baseline.
  • Tested across four domains: science, math, coding, and cybersecurity, using historical model releases.
  • Reasoning traces and tool-call channels carry most of the capability signal; removing them 3-4x's error.
  • Protocols fail under distribution shift, underestimating early-2025 reasoning models on math tasks.
  • Code and HTML paper released, letting teams extend lifetime of existing eval suites.

When several frontier models approach perfect accuracy, a benchmark can no longer rank them reliably. A new study from researchers affiliated with LASR Labs, the ELLIS Institute Tübingen, UK AISI, and collaborating institutions finds that correctly solved transcripts still contain enough signal to predict performance on harder, held-out problems.

Authors Morio Hamada, Jehyeok Yeon, Matteo Migliarini, Justin Olive, and Magda Dubois tested the idea through historical replay. They treated older benchmarks as though they had just saturated, then measured whether protocols analyzing successful solution traces could identify the stronger models.

Perfect Scores Hide Useful Evidence

Near-perfect accuracy removes the resolution needed to compare leading systems. That limits a benchmark’s value for model selection, capability monitoring, and safety cases, which use structured evidence to support deployment decisions. Policies triggered by capability thresholds face the same problem: a saturated score cannot show whether a model has crossed a new threshold.

Each benchmark attempt can yield reasoning, tool calls, intermediate outputs, and a final answer. Models with identical scores may differ in how they decompose a task, verify results, or choose tools. The paper evaluates whether those behavioral differences reveal capabilities that binary correctness no longer captures.