Putting a number on a coding-agent harness

mini-harness is a coding-agent harness I wrote in about 1,800 lines of Python: an agent loop, nine tools defined as Pydantic models, read-before-write gating on file edits, a Docker sandbox for generated code, subagents, context compaction, a terminal UI. It had sixteen tests that drove the whole loop against a fake model server, and it had no numbers. A harness with no numbers is a description.

So I ran it on the exercises from Aider's polyglot benchmark — Exercism problems in Python, Go and Rust, each with a stub, an instructions page and its own test suite. The harness gets the instructions, the stub's name and the test command. The score is whether the exercise's tests pass afterwards. No model grades anything, so the number is the same number every time it is run.

The result

With claude-haiku-4-5: 24 of 25 exercises, 96%. Python 12 of 12, Go 11 of 12, Rust 1 of 1. Median exercise: ten turns, about twelve cents, under a minute. The per-exercise report is in the repository.

That is the headline, and it is the less interesting half.

Where the money went

The whole run cost $7.33. Four exercises cost $4.60 of that. go/dominoes ran for 52 turns and $1.62 before failing; go/connect took 40 turns and $1.50 and passed; rust/variable-length-quantity and go/pov were the other two. The remaining twenty-one exercises averaged thirteen cents.

The reason is not mysterious once you see it. Every turn of an agent loop re-sends the entire conversation, so an exercise that loops for fifty turns does not cost five times what an exercise that finishes in ten costs — it costs closer to twenty-five times. Cost grows with the square of the turn count. A harness that allows three hundred turns per task, as mine did in its benchmark profile, has no ceiling on what one task can spend.

No unit test would have told me this. The tests check that the loop terminates, that a tool call with bad arguments comes back as a correctable error, that an edit to an unread file is refused. They cannot check what a task costs, because cost is a property of the model's behaviour on a real problem, and the fake server does not loop. The benchmark is the only instrument that measures it.

The harness now has a per-task turn cap, and the benchmark driver reports turns and cost next to pass/fail for every exercise. With the cap at forty, the two late runaways stopped at $1.50 and $0.77 instead of wherever they would have gone.

Two things the scoring had to get right

Rust's tests are ignored by default. Exercism marks every Rust test after the first with #[ignore], so a plain cargo test runs one test, reports ok, and exits zero. My first Rust score was a clean 100% that measured nothing. The driver now runs cargo test -- --include-ignored, which is what the published benchmark does. A grader has to be verified too.

The reference solution has to go. Each Exercism exercise ships with an example solution under .meta/. The driver deletes it from the copy the agent sees, and marks any run that edits a test file as a failure whether or not the suite then passes.

What the number is and is not

It is not comparable to the leaderboard. That runs all 225 exercises across six languages with much larger models; this ran a 25-exercise subset in three languages with a small one. What it says is narrower and, for a harness, more useful: given the same model, does the loop around it get the problem solved, and what does that cost. Both answers are now in a file rather than in my head.