Skip to main content
Back

Anthropic's Claude Opus 4.8 Beats Its Predecessor but Trails GPT-5.5 by 12 Points

Datacurve adds Claude Opus 4.8 to DeepSWE, revealing a 6% accuracy gain over Opus 4.7 at lower cost — but GPT-5.5 still leads the leaderboard by 12 points

  • Opus 4.8 added to DeepSWE: Datacurve benchmarked Claude Opus 4.8 on DeepSWE, showing a 6% accuracy gain over Opus 4.7 at lower average cost per task.
  • GPT-5.5 still leads: GPT-5.5 scores 70% vs Opus 4.8's 58% (max), while costing half as much ($6.61 vs $12.58) and finishing tasks twice as fast.
  • Efficiency is the real story: Opus 4.8 [high] scores 51% at ~$3.98/task, beating Opus 4.7 [max] at 54%/$18.19 — more performance per dollar across effort levels.
  • Honesty improvements matter for agents: Opus 4.8 lets code flaws pass unremarked 4x less often than 4.7, critical for long-running autonomous workflows.
  • Claude's recurring failure mode: DeepSWE's qualitative analysis shows Claude consistently implements one branch of multi-part prompts and forgets the other — two-thirds of its failures follow this pattern.
  • Same pricing, available now: Opus 4.8 is live on claude.ai, Claude Code, and the API at $5/$25 per million tokens — unchanged from Opus 4.7. Full leaderboard here.

Datacurve has added Claude Opus 4.8 to its DeepSWE leaderboard, and the numbers tell a nuanced story. On the default high thinking effort setting, Opus 4.8 scores 6% higher than Opus 4.7 at its xhigh setting, while also lowering the average cost per task. That is a genuine efficiency win. But it also puts the new model in sharper relief against GPT-5.5, which still leads the leaderboard by a significant margin.

What DeepSWE actually measures

Before diving into the numbers, it helps to understand why DeepSWE results are worth paying attention to. Anthropic released Claude Opus 4.8 alongside a wave of benchmark claims, but most public coding benchmarks are starting to saturate at the frontier. DeepSWE was built specifically to separate models that cluster together on those older tests.

The benchmark has four properties that distinguish it from alternatives like SWE-Bench Pro:

  • Contamination-free tasks: Every task is written from scratch, not adapted from existing commits or pull requests. The solutions have never appeared in any model's training data.
  • Real-world complexity: Prompts are roughly half the length of SWE-Bench Pro's, yet reference solutions require 5.5x more code and around 2x more output tokens to produce.
  • Broad coverage: 113 tasks across 91 repositories in TypeScript, Go, Python, JavaScript, and Rust.
  • Reliable verification: Verifiers are hand-written to test observable software behavior, not implementation details. DeepSWE's false positive rate is 0.3%, compared to 8.5% on SWE-Bench Pro.

That last point matters more than it sounds. Anthropic's evaluations showed Opus 4.8 was around four times less likely than its predecessor to allow flaws in code it generated to go unremarked. A benchmark that grades incorrectly a third of the time would obscure exactly that kind of improvement.

The Opus 4.8 numbers on DeepSWE

Opus 4.8's progress is mainly that it gets above Opus 4.7 max while being stronger and cheaper. Opus 4.8 [max] scores 58%, while Opus 4.7 [max] scores 54%; at the same time, Opus 4.8 [max] averages $12.58 per task, below Opus 4.7 [max] at $18.19. That is a meaningful efficiency gain: more accuracy for less money.

The effort-level breakdown is worth understanding. Opus 4.8 moves from medium to high to max, with scores of 47% → 51% → 58%. But max average cost jumps from high's $3.98 to $12.58, average output grows from 48k to 136k tokens, and runtime moves from around 21 minutes to 44 minutes. In other words, the model has a wide operating range, and the default high setting is the sweet spot for most workloads.

Here is how the full leaderboard looks as of the latest update:

Model Pass@1 Avg Cost Avg Time
GPT-5.5 [xhigh] 70% $6.61 21 min
Claude Opus 4.8 [max] 58% $12.58 43 min
GPT-5.4 [xhigh] 56% $4.38 27 min
Claude Opus 4.7 [max] 54% $18.19 39 min
Claude Sonnet 4.6 [high] 32% $5.52 42 min

Where GPT-5.5 pulls ahead

Opus 4.8 improved from 54% to 58% over Opus 4.7 , a solid gain. But GPT-5.5 at 70% leads by 12 points, while costing half as much ($6.61 vs $12.58) and completing tasks 2x faster (21 min vs 43 min). The gap is not just accuracy; it is the cost-per-solved-task ratio that hurts most for production workloads.

The token efficiency story is stark. GPT-5.5 achieves this while being roughly 3x more token efficient than Opus 4.8 at max: 47k output tokens versus 136k. For anyone running agents at scale, that difference compounds quickly.

DeepSWE's qualitative analysis also surfaces a recurring failure pattern specific to Claude. On tasks that enumerate parallel behaviors ("support both sync and async," "handle both line comments and block comments"), Claude tends to implement one branch correctly and forget to mirror the changes to the other. Roughly two-thirds of Claude's missed-requirement failures on DeepSWE fit this "one branch shipped" pattern. GPT-5.5 has the lowest rate of missing stated behaviors of any configuration on the leaderboard.

What Opus 4.8 is genuinely better at

The DeepSWE numbers are not the whole picture. Anthropic said Opus 4.8 outperformed competitors on a number of key benchmarks, including agentic coding, reasoning, financial analysis, and knowledge work. On Anthropic's own Super-Agent benchmark, Claude Opus 4.8 is the only model to complete every case end-to-end, beating prior Opus models and GPT-5.5 at parity on cost.

The honesty improvements are also notable for agentic use cases. Opus 4.8 lets flaws in its own code pass unremarked roughly four times less often than Opus 4.7, and produces dishonest summaries of agentic coding work about seventeen times less often than Claude Sonnet 4. For long-running autonomous workflows where a human is not watching every step, a model that accurately flags its own uncertainty is meaningfully safer to deploy.

Anthropic retakes the lead on GDPval-AA, the primary evaluation for agentic performance on knowledge work tasks. Opus 4.8 scored an 1,890 Elo, reflecting an implied win rate of approximately 67% against GPT-5.5. So the competitive picture is genuinely split: GPT-5.5 wins on long-horizon coding tasks under a standardized harness; Opus 4.8 wins on knowledge work and agentic reliability.

How to use it and what it costs

The model can be used through claude.ai, Claude Code, and the Claude API, with the API name claude-opus-4-8. Pricing for regular usage is unchanged from Opus 4.7: $5 per million input tokens and $25 per million output tokens. Fast mode costs $10 per million input tokens and $50 per million output, and runs at 2.5x speed.

The practical guidance from the DeepSWE data is straightforward: max effort is the expensive last gear , use it for high-value, high-failure-cost, long-horizon exploration tasks, not as the default for every daily issue. The default high setting delivers most of the accuracy improvement at a fraction of the cost.

The new model comes just 41 days after Opus 4.7 was released, a much faster upgrade cycle than normal for Anthropic. The fast turnaround may have something to do with the chilly reception to Opus 4.7, which some users found disappointing. Whether the efficiency gains in Opus 4.8 are enough to close the gap with GPT-5.5 on pure coding tasks remains an open question , but the DeepSWE leaderboard will keep updating as both labs ship.