Oxford Researchers Built CUSP to See if AI Can Predict Science's Future
A new benchmark from Oxford, Stanford, and Sakana AI tests whether frontier AI can predict scientific breakthroughs — and the results are humbling
Takeaways
- CUSP benchmark: A new 4,760-event benchmark tests whether frontier AI can predict scientific breakthroughs before they happen, across biology, chemistry, physics, medicine, and AI.
- Strong recognition, weak forecasting: GPT-5.4 scores 82% on explaining known mechanisms but timing predictions overshoot by ~14 months on average.
- Not a data problem: Performance barely changes whether events are before or after the model's training cutoff, ruling out knowledge gaps as the main cause.
- High-impact discoveries are hardest: The gap between partial and full information grows specifically for high-citation advances — the ones that matter most.
- Systematic overconfidence: Models show strong response biases and poor uncertainty calibration, making them unreliable as scientific forecasting tools.
- Open source: Dataset on Hugging Face and code on GitHub with a quickstart notebook.
There is a growing assumption baked into the AI hype cycle: that models trained on the entirety of human scientific knowledge should, in theory, be able to see where science is headed next. A new paper from researchers at Oxford, Stanford, the Allen Institute for AI, and Sakana AI puts that assumption to a rigorous test , and finds it wanting. The paper introduces CUSP, a benchmark designed to measure whether frontier AI can genuinely forecast scientific progress, not just recognize it in hindsight.
CUSP stands for Cutoff-conditioned Unseen Scientific Progress, a multi-disciplinary, event-level benchmark that evaluates scientific forecasting in AI systems through feasibility assessment, mechanistic reasoning, generative solution design, and temporal prediction. The key word is cutoff-conditioned: the benchmark is explicitly designed to test what a model can predict about discoveries that happened after its training data ends.
Why this question matters now
AI is increasingly embedded in scientific discovery, yet whether it can anticipate scientific progress remains unclear. That gap between capability and understanding is becoming consequential. Labs are deploying AI systems to accelerate drug discovery, materials science, and physics research. If those systems are confidently wrong about which research directions will pan out, the cost is not just wasted compute , it is misdirected scientific effort.
AI systems have already surpassed graduate-level performance on some scientific benchmarks, such as GPQA Diamond, where leading models now exceed PhD-level experts. But acing a multiple-choice exam about known science is very different from predicting what science will discover next. CUSP is built to probe exactly that gap.
What the benchmark actually tests
CUSP is a research dataset for evaluating how well AI systems handle temporally grounded questions about scientific papers under a historical cutoff. Given a model's training cutoff date, CUSP asks: can the model anticipate real scientific discoveries that happened after that date?
The benchmark covers 4,760 real breakthroughs across biology, chemistry, physics, medicine, and AI, with verified knowledge cutoffs, and four forecasting tasks per event: feasibility, mechanism, solution design, and timing. Each task type probes a different layer of scientific reasoning:
- Binary: Did this finding actually happen? (Yes/No)
- Perturbed Binary: A negated version of the same question, used to probe whether models are just pattern-matching on surface cues rather than reasoning
- MCQ: A 4-choice question about the specific outcome of a discovery
- Free-Response (FRQ): An open-ended prompt asking the model to design a solution, scored by an LLM judge on a 0-10 scale
- Date Prediction: Predict the publication month of the discovery, scored by how close the estimate is
The perturbed binary task is a particularly clever design choice. By flipping the polarity of a question and checking whether the model's answer flips accordingly, the benchmark can detect systematic response bias , models that just say "yes" to everything, for instance.

The results are a reality check
Across 6 frontier models, the pattern is consistent: strong recognition , GPT-5.4 at 82% on mechanism , but near-chance forecasting, with timing predictions overshooting by roughly 14 months. In other words, models are good at explaining how a known discovery works, but terrible at predicting whether and when a future one will happen.
Performance is highly heterogeneous across domains, with the timing of AI progress more predictable than advances in biology, chemistry, and physics. This makes intuitive sense: AI progress follows more legible scaling trends, while biological and chemical discovery is messier and harder to extrapolate.
The most striking finding is about training data. A natural hypothesis would be that models fail to predict post-cutoff events simply because they haven't seen them. CUSP controls for this directly , and the results undercut that explanation. Performance is largely insensitive to whether events occur before or after the training cutoff, suggesting these limitations cannot be explained solely by knowledge exposure in training data.
Under controlled information access, additional pre-cutoff knowledge improves performance but does not close the gap to full-information settings, which becomes more pronounced for high-citation advances. The breakthroughs that matter most , the ones that get cited heavily , are precisely the ones models are worst at anticipating.
Models also exhibit systematic overconfidence and strong response biases, indicating unreliable uncertainty estimation. This is arguably the most dangerous failure mode: a model that doesn't know what it doesn't know.
The deeper finding
Current AI systems can identify plausible approaches and benefit from more knowledge, but fail to accurately forecast whether, when, and how discoveries will materialize, exhibiting biases and poor uncertainty calibration. This highlights a fundamental gap between knowledge access and the ability to reliably anticipate the trajectory of scientific discovery, suggesting that true scientific forecasting requires capabilities beyond mere information retrieval.
This is the real headline. The bottleneck is not data , it is something more fundamental about how models reason under genuine uncertainty about the future. Retrieval-augmented generation and larger context windows will not fix this. The gap grows specifically for high-impact discoveries, which are by definition the ones that break from established patterns.

What this is useful for
CUSP is primarily a diagnostic tool. It gives the research community a principled way to measure progress on scientific foresight as models improve. The benchmark also includes a CUSP Time Capsule component: a set of prospective tasks that shows models converge on shared expectations for future scientific and AI progress, but vary in the projected pace and magnitude of advancements, such as CO2 emissions and AI benchmark scores.
Practically, the benchmark is relevant for anyone building AI-assisted research tools , literature review agents, hypothesis generators, or automated experiment planners. If your system needs to make forward-looking claims about research directions, CUSP is the right stress test to apply. The dataset and evaluation code are publicly available on Hugging Face and GitHub, with a quickstart notebook to get running quickly.
from datasets import load_dataset
ds = load_dataset("SeanWu25/CUSP", data_files="CUSP_final.jsonl", split="train")
print(ds[0].keys())What it cannot do
CUSP measures forecasting ability but does not explain why models fail. It also focuses on event-level prediction , individual discoveries , rather than longer-horizon trend forecasting. The benchmark's coverage skews toward high-impact, well-cited work by design, which means it may not capture how models perform on niche or emerging subfields. And because scoring the free-response task relies on an LLM judge, there is an inherent circularity when the judge and the evaluated model come from the same family.
The paper's conclusion is measured but clear: AI is a collaborator in science, not an oracle. The models that exist today can help researchers identify plausible directions and reason about mechanisms , but the job of deciding which bets to place on the scientific frontier remains stubbornly human.