Skip to main content
Back

Databricks' OfficeQA Pro Exposes Frontier AI Failing Real Enterprise Work

Databricks' OfficeQA Pro benchmark exposes a brutal gap: frontier AI agents score under 50% on real enterprise document tasks that any financial analyst could handle

  • New benchmark: Databricks releases OfficeQA Pro, 133 hard questions over 89,000 pages of U.S. Treasury Bulletins spanning ~100 years.
  • Frontier models fail badly: Claude Opus 4.6, GPT-5.4, and Gemini 3.1 Pro Preview score under 5% from memory and under 12% with web search.
  • Best agent tops out at 54%: Even with the full corpus, no agent surpasses 50% accuracy on raw PDFs; best result is 66.9% with oracle pages + Databricks parsing.
  • Parsing is the biggest lever: Switching to Databricks' ai_parse_document yields a 16.1% average relative gain — larger than most model upgrades.
  • Key failure modes: Temporal revision tracking, visual chart reading, and multi-step analytical reasoning remain unsolved for all tested agents.
  • Open-source: Benchmark, dataset, and scoring code available on GitHub and Hugging Face.

OfficeQA Pro, a new benchmark from Databricks AI Research, is making a pointed argument: the AI models acing PhD-level olympiad problems are still failing at the kind of document-heavy analytical work that fills enterprise workflows every day. The results are humbling. When given a 89,000-page corpus of U.S. Treasury Bulletins and asked 133 targeted questions, the best frontier agents top out at under 50% accuracy. No agent cracks 50% on the full corpus, and the gap between benchmark hype and business reality has never been more clearly measured.

The Problem With "Smart" AI in the Office

Most AI benchmarks test things like abstract reasoning, math olympiads, or PhD-level trivia. Unlike existing benchmarks that focus on abstract capabilities, OfficeQA proxies for the economically valuable tasks enterprises actually perform. The specific capability being tested here is what the authors call Grounded Reasoning: the end-to-end pipeline of navigating a large, messy document archive, finding the right pages, parsing tables correctly, and then computing a precise numerical answer.

This is different from what most benchmarks measure. GDPval represents an important step towards evaluating economically valuable tasks, but does not fully capture the demands of real-world enterprise workflows. Tasks are designed for a closed-world setting where the full context (fewer than two artifacts on average) is provided directly in the prompt, which neglects the challenge of faithful retrieval across large enterprise collections. OfficeQA Pro is explicitly designed to stress-test that retrieval challenge.

The Corpus: A Century of Financial Chaos

The OfficeQA Pro corpus consists of U.S. Treasury Bulletins, which were published monthly from 1939–1982 and quarterly thereafter. Each bulletin is 100–200 pages long and consists of written analysis, numerical tables, figures, and charts. That adds up to 89,000 pages and over 26 million numerical values.

Example Treasury Bulletin table showing customs duties and taxable imports from 1939-1941

The corpus is deliberately nasty in ways that mirror real enterprise archives:

  • Format drift: The corpus transitions from scans of physical documents to digital-native PDFs in 1996, introducing variability in character recognition quality and layout consistency.