Skip to main content
Back

GPT-6-Astra Solves 49 of 50 Unpublished Physics Problems for Under $3

A new benchmark of 50 unpublished research problems finds frontier agents solving nearly every one for under three dollars each, with little help from the web.

Paper, Full Access
  • TheoPhysEval introduces 50 unpublished research-level theoretical physics problems with verifiable answers.
  • GPT-6-Astra in Codex solved 49/50 problems at under $3 each on the realistic track.
  • Opus 5.5 with Claude Code CLI reached 46/50; Gemini 3.8 Flash hit 34/50.
  • Open-weight DeepSeek-V4.1-Flash and GLM 5.3 trailed at 27/50 and 21/50.
  • Blocking web access barely hurt accuracy, suggesting heavy memorization of the physics literature.
  • Dataset and paper are publicly released for reproduction and extension.

A group of 34 physicists evaluated five AI agent configurations on 50 newly written, research-level theoretical physics problems. The top configuration, GPT-6-Astra running in Codex, solved 49 of 50 problems with web access and 46 without it. The authors report an average cost below US$3 per problem.

Called TheoPhysEval, the benchmark tests whether an agent can independently complete a difficult, well-posed calculation using a coding harness and standard tools. Its results provide evidence that current agents can automate a substantial share of bounded theoretical work, while leaving question selection, research planning, and scientific judgment outside the evaluation.

Fresh problems cut leakage risk

Many scientific benchmarks rely on textbook exercises, exam questions, or published papers that may appear in model training data. TheoPhysEval reduces that risk with previously unpublished problems written for the evaluation, each with a verifiable closed-form or numerical answer. Fresh questions cannot remove contamination from established methods or related literature, but they make direct memorization of the prompts less likely.

The authors estimate that a specialist could solve a typical problem within a day, while researchers from adjacent fields would generally need longer. The collection spans 11 areas rather than concentrating on a single specialty.

Distribution of TheoPhysEval problems
Field Problems
Classical physics5
Stochastic processes and biological physics5
Nonequilibrium statistical mechanics3
Lattice theories3
Quantum phases of matter5
Electronic structure theory4
Quantum field theory