What is actually new here, and what is not
Field note. A tool whose whole argument is “prove it, don’t assert it” should not get to call itself new without showing its work. This is the showing.
The run that raised the question
On 2026-09-04 we pointed corral at psf/requests
at commit 414f051 — a real library, a real suite, nobody’s fixture — with
gemini-3.6-flash planting faults, deriving goals and writing tests, and
claude-haiku-4-5 reading the suite as the critic. It ran for four hours and
fifteen minutes.
The report is honest about what that bought: sixteen candidate files, eight
audited, a kill rate of 0.49 over those eight, and eleven gaps proven by
execution — ten of them in auth.py, where the pool wrote a test that
killed all ten survivors, and one in _internal_utils.py. Six of the eight
files are marked [TIMED OUT — pool did not converge]: each hit the thirty
minute per-file budget, and each hit it in the same phase — not while grading
the developer’s own tests against the faults (four to eleven minutes), but
while authoring tests to prove the survivors were catchable (eighteen to
twenty-six minutes), with a third to three quarters of each file’s survivors
never attempted. Three more files could not be graded at all, because the
suite’s own baseline fails without network access; the report says so rather
than scoring them. The whole run was 232 model calls — 1.9 million tokens
in, of which 0.9 million were served from cache, and 69 thousand out. At any
current flash-tier price that is pocket money. The money was never the
constraint. The clock was.
Four hours for eight files is a nightly job, not a pull-request check. That number made us ask a question we had been avoiding: is this slow because we are doing something nobody has done, or because we are re-deriving something people did fifteen years ago?
The half that is not new
Mutation testing — plant a fault, run the tests, count what dies — is a 1970s idea with mature tools: PIT for Java, Stryker for JavaScript, mutmut for Python. The kill rate as a metric is theirs. Coverage-based test selection — run only the tests that execute the mutated line — is theirs too, and corral’s version of it is a re-derivation, not an invention. Their whole engineering history is the problem we just hit, and they solved it more thoroughly than we have: PIT kills a mutant at the first failing test, ordered by which is likeliest to fail, and caches per-mutant results across runs so an unchanged line is never re-graded. Corral does neither yet. Both are published. The right move is to take them, and we will.
So the cost problem is not why a tool like this was never built. Those tools were built. They are also rarely run, and the reason is exactly our requests number: a whole-repository mutation run is something a team schedules overnight and reads on Monday. What was built stayed niche.
Signed attestations, hash-linked records and public transparency logs are not ours either; Sigstore’s Rekor is where our statements go, and it was there first.
The half that is new
Two things in the requests report could not have been produced by any of those tools, and as far as we can find, by anything else.
A gap proven by execution rather than reported. Every mutation tool
produces survivors and stops. A survivor is a claim: this fault got past
your tests. It might be a real gap, or an equivalent mutant no test could
ever catch, and the tool cannot tell you which. Corral’s writer seat authors
a test against each survivor, runs it on the unmutated code to prove it is
sound, plants deliberately broken source at the test’s own path to prove the
runner actually collected it, and then runs it against the fault. Only a
survivor killed that way is called a gap. The rest are disclosed as what they
are — tried and missed, test unsound, or, in the requests report, 15 seats ungraded — never attempted, so the proven count is over the rest. That
last sentence is one no existing tool can write, and it is the sentence the
whole product turns on. Nobody could write it before a model could write the
test.
A measurement of how alike two models are, instead of an assumption. The
second-model-as-reviewer pattern is now everywhere; every serious agent
pipeline runs one model as a check on another. We have not found anyone who
measures how correlated the two seats’ blind spots actually are — same
faults, same file, same commit — and publishes the number. Corral’s
--shadow-writer-model runs two writers against the same survivors and
reports the Jaccard overlap of their misses. The first result,
0.750 over 13 survivors
between frontier models from two different labs, cuts against the industry’s
working assumption: of everything either writer missed, both missed three
quarters of it. Distinct vendors bought far less independence than people
were pricing in. We think that number is a first of its kind. We would be
glad to be shown an earlier one.
The stance, which is the part with no competitor
There is a third thing, and it is not a feature. Corral ships with no
default models: every seat is named by the operator or the run is refused.
On a CI runner, the checkout under audit may not choose its own auditors —
its .corral/models.json is ignored, out loud. The record of a run is
hash-linked to a public log, and when the tool finds defects in itself, those
go into the same record with the same signature: the
reviews that found a
fabricable “proven” gap and a published trust anchor are on this site because
the tool’s own rule applies to the tool.
None of that is hard to build. It is hard to want to build, and this is our actual answer to “why not before.” The people who could ship this in six weeks — the model vendors, the agentic-IDE companies, the test-generation startups — have the wrong incentive. A vendor sells the model that would be graded. A company that sells “we write your tests” cannot also sell “your tests are weak, here is the proof.” The buyer who wants an unflattering, signed number about their own suite is a compliance-shaped buyer, and that is a small, slow market. Nemo iudex in causa sua is not only the design principle. It is why the seat was empty: nobody who could sit in it was allowed to.
What we will and will not claim
Not new: mutation testing, coverage-based selection, the kill rate, signed attestations, transparency logs. We use them and we say so.
New, to the best of our knowledge: a gap proven by an executed, collected, sound test, with everything unproven disclosed by name; and a published measurement of writer-seat decorrelation on a fixed mutant set.
A stance nobody with a commercial model has a reason to take: named seats, no defaults, the audited code barred from picking its judges, the tool’s own defects in the tool’s own signed record.
And a debt: the grading loop is fifteen years behind PIT, and the authoring loop — the new one, the one that timed out on requests — has no cost engineering at all yet, because nobody has ever had to make that phase fast. That is the next thing to build, and when it is built, the number that proves it will be a run on the same commit of the same repository, signed, beside this one.
The number that proves it
Same commit, same herd, same timeout, the same afternoon — after the budget,
the alone authored pass, and the confidence terms landed. Generated from the
two runs’ ledgers; the page this is spliced from, with the reading guide, is
docs/design/before-and-after.md.
Two runs of corral certify --repo on psf/requests@414f051, same herd, same 30-minute per-file timeout, 2026-09-04. Generated from the two runs’ own ledgers by scripts/gen-before-after.py; never hand-edited.
| run | files audited | converged | proven gaps | mutants graded | time in audited files | model calls | tokens in / out |
|---|---|---|---|---|---|---|---|
| before (main @ 925dddc) | 8 | 2 of 8 | 11 | 255 | 3h28m | 232 | 1.9M / 68.6k |
| after (main @ d951419) | 3 | 2 of 3 | 27 | 112 | 1h12m | 136 | 2.3M / 89.7k |
| primed (main @ 7f49ddf, –prior on the after-run’s record) | 3 | 3 of 3 | 30 | 112 | 1h14m | 117 | 2.0M / 76.8k |
Time in audited files sums each audited file’s own phases plus the one selection pass; the runs’ clock times — 4h15m before, 1h20m after, 1h22m primed, from the launcher logs — are longer by the files the scan probed and then could not grade (three baseline failures before, none after) and by setup nothing attributes to a file.
The kill rates below are not a before/after of requests’ tests: the exam changed (the mutants column says how — a flat five per seat became a complexity-derived budget), so a rate over one exam is not comparable to a rate over the other. What is comparable across the runs: wall clock, whether a file converged, the gaps proven by execution, and the width of each rate’s 95% interval. The before-run’s reach reads not recorded because mutant spans were not stored until #248.
before (main @ 925dddc)
| file | mutants | kill rate (95% interval) | survivors | proven | reach | dev pass | authored | total | |
|---|---|---|---|---|---|---|---|---|---|
| _internal_utils.py | 10 (flat, 5 per seat) | 0.90 (0.60–0.98, n=10) | 1 | 1 | not recorded | 8m37s | 2m21s | 12m24s | converged |
| _types.py | 20 (flat, 5 per seat) | 0.25 (0.11–0.47, n=20) | 15 | 0 | not recorded | 7m25s | 22m45s | 31m37s | timed out |
| adapters.py | 37 (flat, 5 per seat) | 0.49 (0.33–0.64, n=37) | 19 | 0 | not recorded | 11m38s | 18m19s | 31m37s | timed out |
| api.py | 39 (flat, 5 per seat) | 0.08 (0.03–0.20, n=39) | 36 | 0 | not recorded | 3m56s | 25m51s | 31m05s | timed out |
| auth.py | 34 (flat, 5 per seat) | 0.71 (0.54–0.83, n=34) | 10 | 10 | not recorded | 2m00s | 3m16s | 6m01s | converged |
| cookies.py | 37 (flat, 5 per seat) | 0.32 (0.20–0.49, n=37) | 25 | 0 | not recorded | 10m52s | 18m58s | 31m37s | timed out |
| structures.py | 40 (flat, 5 per seat) | 0.72 (0.57–0.84, n=40) | 11 | 0 | not recorded | 10m28s | 19m38s | 31m37s | timed out |
| tests/testserver/server.py | 38 (flat, 5 per seat) | 0.47 (0.32–0.63, n=38) | 20 | 0 | not recorded | 19m13s | 10m32s | 31m22s | timed out |
after (main @ d951419)
| file | mutants | kill rate (95% interval) | survivors | proven | reach | dev pass | authored | total | |
|---|---|---|---|---|---|---|---|---|---|
| adapters.py | 40 (complexity) | 0.59 (0.43–0.74, n=37) | 15 | 15 | 14/19 symbols, 28/53 decisions | 17m01s | 3m29s | 22m17s | converged |
| models.py | 40 (complexity) | 0.47 (0.33–0.63, n=40) | 21 | 0 | 35/51 symbols, 42/149 decisions | 24m02s | 5m49s | 31m37s | timed out |
| utils.py | 40 (complexity) | 0.66 (0.49–0.79, n=35) | 12 | 12 | 29/44 symbols, 40/143 decisions | 9m56s | 5m43s | 17m26s | converged |
primed (main @ 7f49ddf, –prior on the after-run’s record)
| file | mutants | kill rate (95% interval) | survivors | proven | reach | dev pass | authored | total | |
|---|---|---|---|---|---|---|---|---|---|
| adapters.py | 40 (complexity) | 0.65 (0.49–0.78, n=37) | 13 | 8 | 13/19 symbols, 25/53 decisions | 17m07s | 3m19s | 22m03s | converged, primed (37 prior edits) |
| models.py | 40 (complexity) | 0.69 (0.54–0.81, n=39) | 12 | 11 | 32/51 symbols, 54/149 decisions | 19m18s | 8m38s | 29m51s | converged, primed (40 prior edits) |
| utils.py | 40 (complexity) | 0.67 (0.50–0.80, n=36) | 12 | 11 | 34/44 symbols, 43/143 decisions | 10m33s | 8m58s | 21m23s | converged, primed (35 prior edits) |
The file both runs audited
--top 3 chose different files before and after (#244 stopped evidence widening from adding files past the bound, and stopped tests/utils.py from out-ranking the library file it was named after), so only these appear in both:
| file | mutants | kill rate (95% interval) | survivors | proven | authored phase | total | ||
|---|---|---|---|---|---|---|---|---|
| adapters.py | before | 37 (flat, 5 per seat) | 0.49 (0.33–0.64, n=37) | 19 | 0 | 18m19s | 31m37s | timed out |
| adapters.py | after | 40 (complexity) | 0.59 (0.43–0.74, n=37) | 15 | 15 | 3m29s | 22m17s | converged |
| adapters.py | primed | 40 (complexity) | 0.65 (0.49–0.78, n=37) | 13 | 8 | 3m19s | 22m03s | converged, primed (37 prior edits) |
Cumulative reach — what the prior bought
Decision points a fault landed on, per run and across both, from the recorded mutant spans of the after-run and the primed run against the extractor’s decision spans. If the prior had done nothing, the union would sit near the larger of the two; it sits near their sum.
| file | decision points | after-run reached | primed run reached | both runs together |
|---|---|---|---|---|
| adapters.py | 53 | 28 | 25 | 38 |
| models.py | 149 | 42 | 54 | 79 |
| utils.py | 143 | 40 | 43 | 68 |
By model
Per seat, from scan_model_calls. The after-run adds a shadow writer (claude-sonnet-5) that attacked the same survivors as the primary writer in the same run — the only comparison between two models that is controlled. Its per-file outcome is on the run log (the challenger writer … proved N of M survivor(s)); the ledger records the pair’s overlap only when the union of both writers’ misses reaches the minimum the coefficient needs, and on these files it did not (both writers proved nearly everything), so the Jaccard column is honestly empty rather than a number over two misses — until the primed run’s adapters.py, where the two writers’ misses reached the minimum and the coefficient was computed: both missed 1 of the 5 either missed, Jaccard 0.200.
| run | seat | model | calls | tokens in / out | model wall clock |
|---|---|---|---|---|---|
| before | mutant-generator | gemini-3.6-flash |
54 | 77.2k / 20.6k | 16m16s |
| before | test-critic | claude-haiku-4-5 |
41 | 944.1k / 17.1k | 2m49s |
| before | test-writer | gemini-3.6-flash |
137 | 908.4k / 30.9k | 37m05s |
| after | mutant-generator | gemini-3.6-flash |
24 | 65.9k / 10.2k | 6m05s |
| after | test-critic | claude-haiku-4-5 |
16 | 419.6k / 5.8k | 1m14s |
| after | test-writer | gemini-3.6-flash |
48 | 746.4k / 8.4k | 10m54s |
| after | test-writer-shadow | claude-sonnet-5 |
48 | 1.0M / 65.3k | 10m50s |
| primed | mutant-generator | gemini-3.6-flash |
24 | 143.4k / 9.9k | 7m25s |
| primed | test-critic | claude-haiku-4-5 |
15 | 437.1k / 5.7k | 1m07s |
| primed | test-writer | gemini-3.6-flash |
41 | 640.8k / 7.0k | 9m29s |
| primed | test-writer-shadow | claude-sonnet-5 |
37 | 781.8k / 54.2k | 9m05s |
The requests run: 2026-09-04, psf/requests@414f051, herd
gemini-3.6-flash (generator, writer, goal-deriver) + claude-haiku-4-5
(critic), shadow off, --top 3 --substrate workspace, 30-minute per-file
timeout. 8 of 16 candidates audited, kill rate 0.49, 11 proven gaps, 6 files
timed out, 3 baseline-failed, 2 ungoaled; 232 calls, 1.9M tokens in (0.9M
cached), 68.6k out. The report also audited tests/testserver/server.py as
a subject, which it should not have; that was fixed the same day.