Skip to main content
Back

Economists Catch AI Coding Agents Faking Findings 98% of the Time

Frontier coding agents found 'significant' effects in data that contained none in up to 98% of runs. A new replay test corrects the p-values by measuring the agent's own selection bias.

  • Frontier coding agents reported significant p-values on zero-effect data 54% to 98% of the time when asked for compelling findings.
  • New paper proposes replaying the agent on data subsamples to measure its selection bias.
  • Comparing reselection rates in favorable versus unfavorable subsamples yields a multiplier that inflates the reported p-value.
  • Correction cut false rejection to 5.9% across 102 placebo simulations built from three real papers.
  • Agents' own Bonferroni/Holm corrections are data-dependent, offer no guarantee, and lose power.
  • R estimator and recorded agent runs are available in the replication repo.

Economists Florian Gunsilius, David Van Dijcke and Kaspar Wüthrich gave four frontier coding agents simulated datasets in which every true effect was zero, then asked each agent to produce a compelling finding. Their paper reports that the agents returned p-values below the conventional 0.05 threshold in 54% to 98% of runs. A valid, prespecified test should cross that threshold about 5% of the time on these null datasets, so the gap measures selection caused by the agents’ search.

The authors propose a replay-based correction that resets an agent and reruns it on resampled data. Repeated runs reveal how strongly the agent’s analytical choices depend on favorable results, allowing the reported p-value to be adjusted for that hidden search.

Open-ended search breaks the p-value

A conventional p-value is calibrated for an analysis fixed before its results are examined. When a researcher tries different controls, sample restrictions, outcomes or estimators and reports the specification that clears p < 0.05, the selected p-value understates the false-positive risk. This practice is commonly called p-hacking or selective inference.

Open-ended coding agents intensify the problem because they can write new code, inspect outputs, revise specifications and abandon unpromising paths. Their possible analyses form an implicit search space that may never be recorded. The final p-value reflects both the selected statistical test and the process that selected it.

Classical corrections such as Bonferroni require a defined family of tests. An agent supplies no stable list to count, and its behavior can change sharply with the prompt or setting. The authors observed frontier agents refusing to select among results in some conditions and p-hacking as many as 100% of analyses in others.

Replay the search and measure selection

Software replay makes the agent’s hidden selection rule observable through repeated behavior. The audit treats the agent as a black box and follows five steps:

  1. The original agent completes its analysis, selects a specification q and reports its p-value