Skip to main content
Back

Can Small Decision Models Verify Benchmarks? We Tested Two Against Three LLM Judges

Jev ranked defective MMLU questions about as well as Claude Sonnet 4.6 at one-sixtieth the cost, but no model was reliable enough to accept or reject items unreviewed.

Model, Full Access
  • On MMLU, Jev 1.14 achieved an AUROC of 0.811, nearly matching Claude Sonnet 4.6's 0.806 (difference ~0.004, 95% CI -0.016 to +0.024), while costing about $0.027 per 1,000 judgments versus Claude's $1.727 (roughly 64x cheaper).
  • On the 300-task SWE-bench sample, Decider outperformed Claude on "tests reject valid fixes" (0.827 vs 0.754, +0.073) and "exclude task" (0.834 vs 0.770, +0.064), both with positive 95% confidence intervals.
  • Of 87 severely underspecified SWE-bench issues, none of the five tested systems (including GPT-5.4 and GPT-5.4 mini) flagged more than 5% at the default 0.5 probability threshold, despite Decider reaching an AUROC of 0.707 on this category.
  • In the auto-accept/escalate/auto-reject policy test targeting 80% defect recall across 166 defective MMLU cases, Jev and GPT-5.4 fell short (77.2% and 78.6% recall), while models meeting the target still auto-accepted around 18-20% of defects.
  • Only Decider qualified for automatic rejection, catching 32 of 37 rejected cases (86% precision) in the enriched sample, but this dropped to an estimated 44% precision once reweighted to the actual MMLU-Redux population of 5,668 questions with 338 defects.

You compare two coding agents using benchmark scores. One performs better, so you choose it for your next project. But how much can you trust that result if the benchmark itself contains errors?

A coding agent can fix the behavior described in a GitHub issue and still fail the benchmark because its tests enforce requirements the issue never specified. OpenAI found examples of this problem while auditing SWE-bench. Other benchmarks contain incorrect answer keys, ambiguous questions, and grading criteria that reject valid solutions.

These errors affect how we evaluate and compare AI models. Identifying them requires checking task instructions, expected answers, and grading rules. As AI agents begin generating benchmarks automatically, this verification work becomes another bottleneck.

One possible approach is to use smaller decision models to identify which benchmark items need further inspection. Unlike generative LLMs, models such as Jev and Decider return structured judgments and probabilities that could help prioritize review.

To investigate, I tested five systems, including Jev, Decider, GPT-5.4, and Claude Sonnet 4.6, on two expert-annotated datasets: MMLU-Redux and SWE-bench task-validity annotations.

The results were mixed. Jev achieved nearly the same defect-ranking accuracy as Claude on MMLU at substantially lower inference cost. Decider performed better than Claude on two SWE-bench validity labels. But the models' confidence scores did not establish a reliable basis for automatic benchmark acceptance or rejection under the policies tested.

Why benchmark verification needs more than a correct answer

A benchmark consists of tasks, expected outcomes, and grading criteria. A model receives a task, produces an answer or action, and receives a score.

For a multiple-choice benchmark, the grader usually compares the selected option against a reference answer. This assumes the question has a valid answer and that the answer key is correct.

For a coding benchmark, the grader runs tests against the submitted patch. A solution passes if it satisfies the required assertions. But those assertions can be incomplete, overly restrictive, or inconsistent with the issue description.

Consider the scikit-learn task identified in OpenAI's SWE-bench audit.

The original GitHub issue reported that TfidfVectorizer ignored its copy parameter. A reasonable solution would address the parameter's behavior.

However, the benchmark's test required the function to produce a DeprecationWarning with a particular message. The issue never specified that behavior, and the message was established during a pull-request discussion unavailable to the agent.

Consequently, the grader could reject an implementation that addressed the issue but did not reproduce the exact warning.

This illustrates three different verification problems:

  • Execution validity: Does the grader run and return a result?
  • Grader correctness: Does it accept valid solutions and reject invalid ones?
  • Task validity: Does the benchmark measure the capability it claims to measure?

Deterministic tests can establish execution properties and many behavioral requirements. They cannot establish every semantic relationship between task instructions, intended behavior, and scoring rules.

LLM judges provide another approach. They inspect tasks, reference answers, and rubrics, then assess whether the evaluation is valid. Meta's AutoBenchmark uses this method to check construct validity, correctness, feasibility, and usefulness before accepting generated benchmarks.

But LLM judges also make errors. Their verdicts can depend on wording, the evidence supplied, and how the task is framed.

This creates a possible role for decision models that answer narrower questions about benchmark quality.

How decision models evaluate benchmark items

Unlike conventional generative LLMs, which produce responses token by token, models such as TypeSafe AI's Jev return structured decisions and probabilities.

Jev supports three question types:

  • Choice: Select an answer from predefined categories and return their probabilities.
  • Noul: Evaluate a yes/no proposition and return its probability.
  • Score: Assign a value using an ordered scoring rubric.

For benchmark verification, a system could provide the task, reference answer, and relevant evidence, then ask:

Is the reference answer incorrect?

Does the task contain multiple defensible answers?

Which defect category best describes the problem?

The model returns probabilities for those judgments rather than a generated explanation.

These probabilities can also determine which cases enter a review queue. For example, a case with a high predicted defect probability might receive priority over one with a low probability.