Giving a coding task to the same model but with different harnesses can change its inference cost by as much as 5x without producing a comparable change in task success. That is the key finding of HarnessTax, a new study that tests how much the software surrounding an AI model affects coding-agent performance and cost.
The harness is the layer that determines the model’s instructions, tools, memory and context management, and how the agent executes a task. When choosing between Claude Code, Codex CLI, Pi and other coding agents, you’re choosing two things at once: a model and the harness.
The study suggests three useful rules of thumb:
- Harness choice can move cost much more than aggregate success
- A minimal harness can compete with feature-rich alternatives
- A model does not necessarily perform best in the harness built by its own provider
For developers, the key lesson is to evaluate the model and harness as one system instead of accepting the default pairing.
What HarnessTax tested
The researchers tested a total of 21 model-harness combinations, composed of seven models and three harnesses: Claude Code, Codex CLI and Pir. They randomly selected 30 tasks from SWE-bench Lite, which tests an agent’s ability to fix software issues, and 30 from Terminal-Bench 2.0, which tests command-line tasks.
The researchers measured both task success and token cost based on API pricing. This makes it possible to see whether one harness causes the same model to consume more tokens instead of mixing that effect with differences in product pricing.
Finding 1: The harness can move the bill more than the score
On SWE-bench Lite, Claude Fable 5 solved 97.8% of attempts with Claude Code and 96.7% with Pi. But an average attempt cost $1.33 with Claude Code and $0.67 with Pi. That means you’re paying roughly double the cost for an observed success difference of 1.1 percentage points.
Interestingly, both Claude Code and Pi averaged around 15 turns per task. However, each Claude Code turn carried more information. Across all seven models, its mean initial context was more than 10 times Pi’s, driven by longer instructions and larger tool definitions.
This overhead matters because the model has to process the harness instructions and tool descriptions alongside the actual task. Token cost is therefore not only a property of the model you choose. The model-harness combination determines how much inference you need to buy to get a result.
Finding 2: A simple harness can be competitive
Pi gives the agent four basic tools: read, write, edit and bash. Yet its configurations reach the study’s cost-success Pareto frontier on both benchmarks, meaning that it gives the highest accuracy for a given budget.
That does not mean that simpler harnesses are always better. These experiments cover two public benchmarks and do not measure every reason developers might choose a richer product, such as integrations, permissions, memory, hooks or workflow features.
But the results make a simple harness a useful baseline. You can add additional scaffolding when it fixes a failure or provides a feature that matters for the application.
Finding 3: The vendor default is only a starting point
The study shows that models do not necessarily work best inside their provider’s harness. Across six Anthropic and OpenAI models and two benchmarks, an alternative harness recorded the highest observed success rate in nine of 12 comparisons.
For example, on Terminal-Bench 2.0, GPT-5.6 Sol scored 83.3% with Pi at around $0.42 per attempt, compared with 78.9% and about $0.76 with OpenAI’s Codex CLI. This does not prove that Pi is generally better for GPT models, especially given the sample size of experiments. But it suggests that native pairing should not be considered as the ultimate best choice.
What it means for developers
HarnessTax also exposes a problem with how developers choose agent stacks today.
“Many harness decisions nowadays are sort of made by: preference, tribal knowledge, word of mouth,” Melissa Pan, co-author of the paper, told AlphaSignal. “It is very unclear whether a particular harness is more accurate, more cost efficient, more this or that on certain metrics for certain workload.”
A practical evaluation should therefore treat “model × harness × workload” as the unit of comparison. Start with a representative sample of your own tasks, run several combinations against it, and track task success, cost and latency.
A harness that looks inexpensive per attempt might become expensive if failed attempts require retries, so cost per successful task is often more useful than cost per run.
The right tradeoff depends on the application. For high-volume routine work, doubling inference cost for a small change in accuracy makes little sense. For a task where failure is expensive, even a modest reliability improvement might justify a higher bill.
Also don’t put too much trust in aggregate success rates. Two configurations can achieve similar overall scores while failing on different tasks. If your workload contains many of the cases where one configuration breaks, the aggregate benchmark can hide the difference that matters most.
A useful process should be to build a representative task set, test several model-harness combinations, measure success, cost and latency, and inspect the failed cases. Choose the least expensive configuration that clears your reliability requirements, and rerun the evaluation when models or harnesses change.
Harnesses are also an optimization layer
There is a broader implication for the developer community. Frontier model training requires resources that few organizations have. “Harness effort is one of the few ways that people can improve performance without touching the model,” Pan said.
“For research, [the study] gives us hope that we have a chance to win in the harness space without having the ability to co-train with SOTA models,” Pan said. “This is very exciting for people who are doing work in the open including academia and the developer community.”
HarnessTax shows why that layer deserves more scrutiny from developers as well. The practical question is no longer only which model to use. It is which model-harness combination provides the reliability your application needs at a cost you can justify.