A judge may not certify herself. So I made corral certify corral.

Field note. corral exists to answer one question — can something other than the author confirm the work is fit? There’s a comment buried in the code that states the principle plainly: a judge may not certify herself. Which raises an uncomfortable question I’d been avoiding: can corral certify corral?

The obvious experiment I’d been putting off

Every part of this tool is built around nemo iudex in causa sua — no one may be judge in their own cause. A different model plants the faults than grades them. The kill-rate is measured by execution, never self-reported. The whole point is that the thing under test doesn’t get a vote on whether it passed.

So the honest experiment writes itself: point corral at corral. Specifically, at internal/adequacy/score.go — the file that computes kill rates. If corral is going to lecture the world about grading your own exam, it had better survive grading its own grader.

I ran it. And it came back with a verdict that made no sense:

status:        NEEDS-REVIEW (dev suite killed 20/20 mutants)
dev_kill_rate: 0.00
survivors:     0

Killed 20 of 20, zero survivors — and a kill rate of zero. Twenty out of twenty is a hundred percent. Those numbers cannot both be true. The tool that certifies software was contradicting itself in its own signed output.

Six bugs wearing one mask

Here’s the part worth the whole note. That contradiction wasn’t one bug. It was six, each one hiding behind the last, and they all looked identical because of the first one I’ll describe.

The mask. When the dev suite can’t even pass on the unmutated code — a build failure, a broken environment, anything before grading can begin — the scorer fail-closes: nothing gets graded. But the driver, handed an empty result, computed “killed” as mutants minus survivors — twenty minus zero — and printed a triumphant, entirely fabricated killed 20 of 20. Every kind of failure produced the exact same misleading line. It was a single lie that made four different problems impossible to tell apart. This is the one that scared me most: a tool whose only job is honesty was manufacturing a number.

Peel it off, and underneath:

One. The jail runs offline, by design — the code under audit can’t phone home. But go.mod pins a toolchain (1.26.5) newer than the jail’s Go (1.26.4), so go tried to download the newer toolchain, failed with no network, and the baseline build died. corral could not audit corral because corral’s own version pin was newer than the jail’s compiler, and nobody had ever hit it because the demos used toy code with no such pin.

Two. Even past that, the whole-repo build needs external modules the offline jail’s empty cache can’t fetch. Toy targets have zero dependencies; a real repo has dozens.

Three. The one that stung: the test command I passed contained a regex — -run '^TestScore$|^TestScore(...)' — full of shell metacharacters. corral joined the argv into a string and ran it through sh -c, which happily re-parsed my $, |, and () into pipes and subshells and shredded the command. Half my fault for the regex; half corral’s, for treating an argv like a shell script.

Four. With the shell fixed, it got further — and hit a cgo wall. Auditing one file inside a monorepo, corral’s compile-check for the authored test was whole-repo go vet ./..., which drags in every package — including an unrelated tree-sitter binding elsewhere in the tree whose C headers go mod vendor quietly prunes. The check failed on a header that has nothing to do with the file under audit. Scope it to the audited file’s own package and it’s gone.

Five. And the subtlest. On a genuinely hard target — an unexported interface, awkward test fakes — the test-writer model sometimes returned nothing: an empty result. corral fed that emptiness into its own repair prompt (“here is your broken test: «», fix the compile error”), which begets more emptiness and burns the whole retry budget in a degenerate loop, so a later good attempt never got a slot. On an empty result, re-prompt fresh instead of trying to repair nothing.

Six failures, most of them nested inside the one before. I only found any of them after I fixed the mask — because until then every single one printed the same fabricated killed 20 of 20 / 0.00 and told me nothing. Each fix I shipped let the run get one step further and hit the next wall.

The critic, still hallucinating on cue

One more, for the record. corral’s decorrelated critic — a second, cheaper model that reads the tests cold and flags the vacuous ones — looked at our own score_test.go and confidently declared a good test worthless. It had “read” an errJail test as a fakeJail test, invented a failure mode the code plainly doesn’t have, and pronounced with total confidence. Flat wrong. And it kept being wrong: on a later run it flagged the same test three times, and one of those flags is the single best argument I have for why a cheap critic can never be trusted. It reasoned, out loud, mid-flag, all the way to the correct answer — and then filed the finding as high severity anyway:

“…Actually, re-reading: the error WILL be triggered. The test correctly expects the error to propagate. This test is NOT vacuous…” — reported as [high].

A judge that argues itself to “not vacuous” and raises the objection regardless is exactly the judge you cannot let near a verdict. And it never got near one. The critic is fenced off — advisory, never gating, recorded as an unverified opinion; the signed number comes from execution alone. The architecture assumes the cheap judge will sometimes hallucinate, so three hallucinations in a row touched nothing. Watching that safety net do its job on our own code was, honestly, the most reassuring moment of the whole exercise.

The number

Once the jail could actually build and grade, corral audited its own kill-rate scorer and returned a real verdict: below its own bar. Depending on which tests you count — score.go’s five dedicated unit tests, or the whole package’s — it lands somewhere between the mid-forties and mid-seventies percent, needs-review either way, with a fistful of execution-proven gaps in the coverage of the very code that measures coverage.

And then it did the thing I’m actually proud of. Once the last two bugs were fixed and the test-writer got a clean shot, it wrote one of the missing tests for me — a real, compiling boundary test for clampMutantTimeout, a function no existing test had ever touched — and handed it back to add to score_test.go.

I wanted a clean 90. What I got is better. A tool that only ever passes things is a rubber stamp. A tool that fails its own most sacred file, hands you the specific gaps, and then writes one of the missing tests — that’s the thing actually working. The physician did not heal thyself. The physician found what was wrong with thyself, wrote it down, and then wrote one of the missing tests.

What shipped

All six are fixed and merged — each as its own small, reviewed change, because a certifier that cut corners fixing itself would be the exact hypocrisy the whole tool exists to catch.

• The mask is gone: when the baseline can’t pass, corral says COULD-NOT-GRADE with the real reason instead of inventing a tally.
• The jail pins its toolchain resolution (a clear “requires a newer Go”, not a silent offline hang) and vendors external deps so an offline audit actually builds.
• The test command is argv-safe (no more shell shredding your -run regex).
• The compile-check is scoped to the audited package, so a monorepo audit doesn’t trip on unrelated cgo.
• An empty model result re-prompts fresh instead of looping on nothing.

Six walls, six fixes, and — the point — it disclosed every one of them out loud rather than papering over it. That’s the same discipline the tool is selling.

The thesis was never that corral is above judgment. It’s that nothing is — including corral. The only way to earn that claim is to turn the instrument on yourself and publish what it finds, especially the unflattering parts. So: real gaps in our own scorer, six bugs in our own jail, one of them a lie the tool was telling in its own signed output — and, at the end, a test it wrote for its own code.

A judge may not certify herself. We didn’t. We let the tool do it, and it convicted us. That’s the whole idea.