How to Test a Decision Model in Your Coding Loop
Jev and a local model matched my reference on 12 of 12 Claude Code fixes to a synthetic API. A filename rule matched 3 and still caught the one real bug!
Takeaways
- Jev and winnow:e4b matched the reference on all 12 Claude Code fixes, Claude in a plain message matched 9, and a filename rule matched 3.
- Only one of the 12 fixes left a bug behind, and all four pickers caught it, so the test measures fit and says nothing yet about catching more bugs.
- The 7.6-microsecond filename rule caught that bug too, even though its targeted pick missed the reference.
- A 0.50 cutoff set with
--min-confidencewould have stopped 9 of winnow:e4b's 36 picks, all matching the reference, and none of Jev's, whose lowest confidence was 0.62. - On a first look, a plain Claude call costs $0.0105, about 150 times Jev's per call, since the $0.0017 cached price only came from resending the same packet.
Before a decision model runs any test suite in your coding loop, you need to know whether it picks the right one. I tested that on 12 Claude Code fixes, and the answer depended on what I counted as right.
This is part 2 of a two-part series on decision models in coding workflows, and it shows how to test one before you trust it. Part 1 showed where the call sits in a Claude Code loop and how to add it with next_check.py, the wrapper that makes the call.
A decision model picks one choice from a list of options you provide it, and assigns a probability to each option.
In our example, whenever a fix passed its required tests, the model had to choose one of four actions: trigger one of three specialized test suites (targeted, integration, or security), or select inspect_first, a fallback that halts automation for manual review when the evidence is too thin to choose.
My test app was a synthetic document API, built only for this experiment. An AI agent seeded 12 bugs into it, Claude Code fixed each one, and every fix passed the nine required tests that run on every change.
Then four pickers answered that question for every fix: TypeSafe's hosted Jev, the open winnow:e4b model on my Mac, Claude in a plain message, and a rule that reads file names.
Jev and winnow:e4b matched my reference on all 12 fixes, Claude matched nine of 12, and the filename rule matched only three.
But the 7.6-microsecond filename rule still caught the only bug a fix left behind, and so did every other picker.
So I'd score two things separately: whether the model picks the expected suite, and whether that suite actually catches failures. Until the model proves on your own codebase that it catches real bugs your simple rules miss, keep it in shadow mode, where you log the recommendations in the background without executing them.
Next comes how I ran the test, then seven steps to run it on your own fixes, each one shown on my 144 answers, and the strongest case against the whole result.
And everything behind those numbers is public: the project on my GitHub has next_check.py, the 12 saved packets, and all 144 answers, and the Decision Model Field Guide lets you browse every answer fix by fix and try the cutoff calculator.
How I tested decision models on 12 Claude Code fixes
I tested one moment in a Claude Code loop: right after Claude fixes a bug and the required tests pass. Each picker answered one question with four options:
| Option | What it means here |
|---|---|
targeted_tests | More edge cases for the changed behavior inside one component |
integration_tests | Contracts and persistence across the API, service, export, and database |
security_review | Access-control tests run as two different users |
inspect_first | The evidence is too thin to choose |
I ran everything in shadow mode, so none of the answers changed the code.
An AI coding agent built and ran the test harness on my Mac, meaning the scripts that seed the bugs, run Claude Code, ask the pickers, and score them. It worked under a spending cap I set and logged every step.
The test API runs on FastAPI and SQLite, with two test users so ownership bugs can be tested. Into it, the agent seeded 12 bugs: four edge cases inside one component, four that cross components such as the API and the database, and four access-control holes.
Claude Code (Claude Sonnet 5.5) fixed each one, and all 12 fixes passed the nine required tests on the first try: a compile check, a seven-test smoke suite, and the bug's own regression test.
To give every picker the necessary context without polluting it with raw agent logs or the entire codebase, the harness packaged the state into an evidence packet for each fix. It holds the task, the API contract, the diff, the required-test results, and each extra suite's description, command, and test names.
Every picker got the same packet three times, which made 144 answers (4 pickers, 12 fixes, 3 repeats), and none was invalid. Only the filename rule saw less, reading just the file paths, and Claude answered each time in a fresh conversation with no tools.