Real audits, recorded. Each is corral certifying a change by execution: a decorrelated cross-vendor herd plants faults in the code, checks whether the developer's own tests catch them, and signs a tamper-evident verdict — no one judging their own cause. Some clear the bar and certify; some leave too many survivors the suite didn't kill and are sent back — corral discloses those for a human to judge, it doesn't rule them defects. Both are here, honestly — the gate showing its work either way.
Every one was exported through the same deny-list + human-manifest privacy gate as the landing hero, and is offline-verifiable from its signed record. Open the tests tab in any replay to see the verdict, the code under review with the surviving fault highlighted, and the suite it graded. Pick a card to replay it on the corral canvas.
Certify version4.go from google/uuid — one of the most-used Go libraries
there is — against the package’s own test suite, by execution. The goal: New /
NewRandom return a valid RFC-4122 version-4 UUID (122 random bits, the version
nibble set to 4, the variant bits set to RFC 4122).
Claude Sonnet 5 planted 20 goal-violating mutants across the file (sharded four ways); google/uuid’s own suite, run in the jail against every one, killed 13 of 20 — a 65% kill-rate, and 7 survived. The gate returned NEEDS-REVIEW and signed the verdict. This is the point worth sitting with: a library this widely trusted, with a real test suite, still leaves a third of the planted faults uncaught.
The decorrelated critic (Haiku 4.5) put its finger on the shape of the gap:
TestRandomUUID calls New() many times and asserts each result has version 4
and the RFC-4122 variant — but it never verifies the 122 bits are actually
random. So a mutant that fixes or narrows the randomness (while keeping the
version/variant bytes correct) sails straight past. The suite checks the
structure of a UUID thoroughly and its randomness not at all. (The critic’s
read is unverified advice, marked as such; the 65% is what the jail measured.)
It’s the “your tests suck” thesis on code nobody would call badly tested. corral doesn’t grade google/uuid as bad — it grades it by execution and hands back exactly which faults its suite can’t see. Open the tests tab to watch a surviving mutant highlighted against the code the suite passed anyway.
Certify a file from more-itertools — more_itertools/recipes.py, 67
functions we didn’t write, from a well-loved zero-dependency library — against the
library’s own test suite (tests/test_recipes.py), by execution. The
generation fanned out across shards so every function got probed, not just
whichever one a single generator happened to pick.
Gemini 3.5 Flash planted 20 goal-violating mutants across the file; the library’s own suite, run in the jail against every one, killed 18 of 20 — a dev kill-rate of 90%, measured by execution, not asserted. That cleared the bar (0.8), so the gate returned CERTIFIED and signed the verdict — offline-verifiable from the record.
Two of the twenty mutants survived — the suite passed despite them. Corral records both rather than rounding up to “spotless,” but it makes no claim that they’re defects. A survivor is either a real untested edge or an equivalent mutant — an injected change with no observable effect, which no test can catch by construction — and corral only calls a survivor a proven gap when the pool authors a compiling test that actually kills it (it didn’t here). So this is not “more-itertools missed two tests”: it’s the tool catching 90% of injected faults against a genuinely strong suite, and disclosing the two it couldn’t kill, unadjudicated, for a human to judge. Open the tests tab to see them highlighted against the code the suite passed anyway.
The decorrelated test-critic here is a stronger, different-tier model — Gemini 3.1 Pro — reading the suite cold while a lighter model did the planting and writing. Its job is to flag tests that don’t actually test anything. It flagged one, and — the point — it holds up under execution:
test_full_permutation, the if i == r “not permuted” check is dead:
i is a range object and r is a tuple, and in Python 3 range == tuple
is always False. The check can never fire. The test still has a real
assertEqual around it, so it isn’t vacuous — but that one guard is inert, and
the critic scoped its wording precisely to the check. Accurate, not overstated.That precision is the point. An earlier run of this same file, with a lighter
same-vendor critic, produced a confident hallucination — it flagged
test_negative_take as vacuous, insisting islice silently swallows a negative
count. It doesn’t: take(-3, …) raises ValueError, so the test passes for
exactly the right reason. A stronger, decorrelated critic makes no such mistake.
Either way, corral treats the critic’s word as what it is: unverified advice that never gates the signed verdict — only a jail and an exit code certify. But watching a more reliable model produce more reliable advice is the whole helper turn: better decorrelation makes your tests stronger from both ends, and every claim on this tape is one you can re-run yourself.
Certify a password validator — valid iff length ≥ 12 and it contains an uppercase letter, a lowercase letter, a digit, and a symbol — against a Python suite that only ever feeds one valid password (and one too-short one), so it never exercises the four character-class rules.
Claude Sonnet 5 planted 5 goal-violating mutants; the developer’s own suite, run in the jail against every one, killed 0 of 5. Every survivor is a dropped character-class check the length-only test can’t see. A test-writer (Sonnet 5) then authored a test that killed all 5 survivors — proving the gaps are real and catchable, not equivalent mutants. The gate returned NEEDS-REVIEW and signed the verdict: it will not certify a suite that guards a fraction of the spec. The decorrelated critic (Haiku 4.5) independently flagged both tests as vacuous — and here it was right: neither can fail on a character-class mutation, and that’s exactly what the execution showed.
The same blind spot as the Go password recording — a length-only test that sails past every character-class fault — now in Python, from corral’s own known-adequacy eval corpus. It’s the whole loop in about forty seconds: find the gap by execution (a 0% kill-rate, measured, not asserted), prove it (a written test that kills every survivor), and grade the suite honestly (needs-review, signed). Open the tests tab to watch a surviving fault highlighted against the code the suite passed anyway.
Certify a change to internal/fence/fence.go — the wrapper that fences
untrusted content (ingested documents, agent-written memory, reported findings)
so a consuming agent can’t mistake it for its authoritative task — by
execution. The question isn’t “does the code look right”; it’s “does this
file’s own test suite actually guard the security property: that untrusted
content can never forge or close the fence?”
This is the audit gate, not a builder — nemo iudex in causa sua, no one is judge in their own cause. A decorrelated cross-vendor herd put the suite to the test: Claude Sonnet 5 planted the faults (and drafted the exposing test), and a different model — Gemini 3.5 Flash — graded whether the developer’s tests caught them. The model that plants the exposing fault is never the model that judges the suite. That separation is enforced, not hoped.
Open the tests tab (the audit lens) to see the whole thing in one place: the signed verdict, the code under review with the surviving fault highlighted, and the developer’s suite corral graded. Every catch on this tape is proven by a test that actually ran in the jail and killed a fault — never a model’s word.
corral certify verify). You don’t trust it; you re-run it.corral certify is the shipped atom; the adversarial pool that produced this
tape is experimental and off by default. And one run is a data point, not a
ranking — fence.go’s suite is genuinely strong, so whether a fault survives
is partly a coin flip. The bug-catching numbers only mean something at volume,
which the eval harness accrues across a whole corpus before any of it gets
published. We’ll always tell you which floor you’re standing on.
Certify lib/text/levenshtein.rb from threedaymonk/text — a real, pure-Ruby
edit-distance implementation (UTF-8 aware) with a minitest suite — against its
own tests, by execution. The goal: distance(a, b) returns the minimum number of
single-character insertions, deletions, or substitutions to turn a into b.
Run head-on, the suite looks pristine: 35 tests, 5,241 assertions, 100% passing. Then corral graded it by execution. Claude Sonnet 5 planted 5 goal-violating mutants in the algorithm; the library’s own minitest suite, run in the jail against each, killed only 3 of 5 — a 60% kill-rate with 2 survivors. That is below the bar (0.8), so the gate returned NEEDS-REVIEW and handed the two surviving faults back.
This is the gap the whole tool exists to expose: a green, assertion-heavy suite that still lets two goal-violating edits through undetected. Passing count is not adequacy — and corral says so by execution rather than by vibe. Open the tests tab to see the two survivors highlighted against the code the suite passed anyway.
The decorrelated critic (Haiku 4.5), reading the suite cold, flagged something
tangential but telling: a helper in the test file itself
(LevenshteinGeneratedDataTest#substitute) references an undefined local — se
where seq was meant — which raises a NameError on the path that hits it. It’s
the critic noticing a latent break inside the tests that grade the code. That
finding is advisory, marked unverified, and never part of the signed 60% —
only the jailed kill-rate certifies — but it’s a second, independent reason to
look twice at this suite.
Same loop as the Go and Python audits, in a third language (Ruby, minitest): plant faults, run the dev’s own suite in a jail, grade by the kill-rate — no self-report. Here the honest answer was not yet — and that refusal, on a suite that passes 100% of its own assertions, is the point.
Certify src/index.ts from vercel/ms — the tiny, ubiquitous
duration-string library (ms('2 days') → 172800000, and back) — against its
own Jest suite, by execution. This is the first recording of a real
TypeScript project audited with its real, dependency-heavy test toolchain.
ms’s test run pulls in a 337 MB node_modules (Jest, its TS transform, the
lot). corral’s jail seeds a copied workspace with a size cap, and 337 MB blows
straight through it — so until recently this audit failed at the seed stage,
before a single test ran. The fix, shipped just before this run: corral
bind-mounts dependency directories read-only into the jail instead of copying
them. You can see it in the run’s first line — deps: bound 1 dir(s) read-only (node_modules). Deps must be present (vendored, exactly as CI installs them);
corral binds them, never installs them. That’s what lets it audit real-world
JS/TS code and not just zero-dependency toys.
Claude Sonnet 5 planted 5 mutants; ms’s own Jest suite killed 4 of 5 — 80%, 1 survivor, NEEDS-REVIEW, signed.
Getting an honest number here took one correction worth telling on ourselves. corral certifies by exit code — it runs your test command and trusts what the process returns. ms’s Jest config enforces a 100% coverage threshold, so the command exits non-zero even when every test passes (coverage sits at 98.4%). On the first run that made the exit code meaningless: every mutant “failed” for the coverage gate, not for catching a bug, and the kill-rate signal was garbage. Re-running with coverage off — so the exit code reflects test pass/fail — gave the real 80%. The takeaway is a genuine edge of execution-gated auditing: a suite whose exit status is dominated by a coverage (or lint) gate confounds a tool that reads exit codes. corral is honest about what it can and can’t infer from a process’s return.
Real TypeScript, a real Jest suite, real vendored dependencies — the exact shape of code the earlier toy recordings couldn’t prove out — audited by execution to a signed verdict. Open the tests tab to see the surviving fault against the code ms’s suite passed anyway.
The cross-recording metric for audits isn't a findings table — it's thebug-catching scorecard: which model actually catches bugs,proven by execution, per (model, role). It accrues as audits run — a cell stays provisional until it has enough of them, so we don't publish a confident number off a handful — and lives atcorral scorecard / /api/bugcatch, offline-verifiable from each signed record. Fugu's routing scores come out of a trained black box; this comes out of a test that really ran and really killed a fault.