Eighty-eight tests that were not there

Field note. One arm, three repositories, and every number that came with it.

The result

Three third-party Go libraries. Not ours, not chosen for weakness — chosen because they are small, widely used, and have real test suites.

21 files audited across 3 repositories
156 survivors      — mutants the projects' own tests did not catch
105 proven-missed  — survivors corral then KILLED with a test it wrote and ran

Updated 2026-08-26, later the same day. The first version of this note reported 15 files / 121 survivors / 88 proven-missed. See the correction at the end: six more afero files became gradable once a timeout was raised, and the numbers above are the fuller run. Nothing was retracted — every earlier figure still stands, there is simply more of it.

Proven-missed is the only number here worth much. A survivor on its own is ambiguous — it might be an equivalent mutant, a change no test could ever catch, which is mutation testing’s oldest measurement problem. Proven-missed resolves it by construction: corral wrote a test, executed it, and watched it kill the mutant. That test exists. It compiled, it ran, it passed against the real code and failed against the planted bug.

So the claim is not “your tests missed 121 things.” It is: here are 88 executable tests that catch bugs these suites do not.

How it was run

Three seats, one model each:

One pinned corral build for every arm. Each row records its repository, its commit, that build’s revision, and the model in each seat, so any line above can be re-run and checked.

The whole arm cost 634,788 input and 12,610 output tokens — the writer seat only. The generator plants dozens of mutants per file and is by far the heaviest consumer; it ran locally, at no API cost. Wall-clock was about three and a quarter hours, serialized.

What it could not do

spf13/afero graded every file it could legitimately grade — 12 of 12, out of 16 candidates. We first reported this as “12 of 16” with four failures, which was unfair to the tool in a way we only caught by reading the four:

• gcsfs/gcs.go and sftpfs/sftp.go live in separate Go modules (gcsfs/go.mod, sftpfs/go.mod), so go test ./... from the repository root cannot reach them at all. corral’s preflight refused before spending anything, because a test it wrote there would never be collected — and a gap it cannot prove is a gap it will not claim.
• lstater.go and symlink.go are pure interface declarations (type Lstater interface { … }). There is no behavior a mutant could violate, and the goal deriver correctly returned “none”.

Its 0.56 is still a kill rate over the audited files, not over the whole repository, and we report it with its denominator either way.

One file reported “tried and missed.” memmap.go scored 0.79 with five survivors, and the writer authored a sound test that killed none of them. That is recorded as a distinct outcome from “no gaps found” and from “the writer failed,” because collapsing the three would let a tool claim a clean bill of health it never earned.

The re-run is a good illustration of why: in the first pass afero.go and util.go both read “tried and missed”; in the fuller run they proved 1 and 2 gaps respectively. Same files, same commit, same models. “Tried and missed” is a statement about one run, not about the code.

One file was clean. match.go killed every mutant. It is in the table at 1.00 because a result that only ever finds problems is not a measurement.

The number nobody should quote alone

google/uuid was audited five times today, at the same commit, with the same models. null.go scored:

kill rate      0.48   0.29   0.40   0.37
proven-missed    16     15     18     20

Same file. Same commit. Same seats. The mutants are regenerated on every run, so each audit is a sample, not a verdict — a different draw from the same distribution.

This is the most important caveat on the page. Any single kill rate above, including ours, carries that spread. We publish the variance because we measured it, and because a tool that reports one confident number per run while behaving like this would be misleading by construction.

What this does not show

It does not show decorrelation. corral can measure whether two models miss the same mutants, but that seat only exists in single-file mode — so nothing in this run speaks to it. The one measured coefficient we have (Jaccard 0.750 over 13 survivors, two frontier models) came from a different mode entirely and remains n=13.

It is also not a cross-language claim. Three Go repositories, one arm, one writer model. A fourth repository was dropped because it predates Go modules and its suite cannot resolve inside the sandbox — corral refused to grade it rather than report a zero.

Where help would matter

The panel is the bottleneck, and it is a solvable one.

• Pairing. corral pairs a source file to its test by convention. Two of four repositories we first tried yielded zero auditable files — minitest/minitest names tests test/test_foo.rb where the walk expected foo_test.rb, and debug-js/debug has one root-level test file covering everything. The first is fixed. The second needs a source→test map, which corral accepts but nobody has written for a real project.
• Languages. Go’s enforced foo_test.go convention pairs almost perfectly. Ruby, JavaScript and Python do not, and every unpaired file is a file corral cannot see.
• Repositories. If you maintain something with a real suite and you want to know what it misses, that is the experiment we want to run — and the result is yours whichever way it goes.

corral is source-available under Elastic-2.0. The numbers above are reproducible from the commits and build revisions recorded with them; if you re-run one and get something different, we would rather hear about that than not.

Correction: the first pass under-reported afero

This note first published spf13/afero at 6 of 16 files, and said eight files “failed with executor errors — the local model daemon timing out under load.” That was true, and the diagnosis was incomplete in a way worth naming.

The cause was not load and not the repository. It was a 300-second per-request HTTP timeout, a default chosen against hosted models, applied unchanged to a 9B model running locally on one 16GB consumer GPU. afero’s larger files generate dozens of mutants per call, and prefill on that hardware is slow enough to cross five minutes. Raising AGENT_LLM_TIMEOUT_SECONDS to 900 and re-running the same commit, with the same models and the same corral build, graded six more files:

The remaining four of the 16 candidates are not failures at all — two are in separate Go modules the test command cannot reach, and two are pure interface declarations with nothing to violate. See the section above.

That was a second thing this note got wrong, and in the opposite direction from the first: we published four correct refusals as four failures. A report that understates its own accuracy is still not telling the truth — it just errs toward looking worse, which is easy to mistake for rigor. corral now prints a one-clause explanation beside every ungradable file saying whether it is the tool failing, the invocation, or a file with nothing to audit.

Three things we would rather say plainly than bury:

No earlier number was wrong. Each row reported what that run measured. The panel was smaller than it should have been, and the reason was a knob, not the code under audit.

The failure named the wrong thing. An operator saw executor-error and context deadline exceeded — the transport, not the cause. Nothing said “raise the request timeout.” A tool that refuses to grade should also say what would let it grade, and this one did not.

“Tried and missed” is per-run, not per-file. afero.go and util.go both read “tried and missed” in the first pass and proved 1 and 2 gaps in the second. That is the same variance the section above documents, showing up in a different column.

The headline moved from 88 proven-missed to 105. We are leaving the original figures visible above rather than overwriting the page, because a result that quietly improves between readings is indistinguishable from one that was tuned until it looked good.