Would you trust your house if the same person architected it, built it, inspected it, and certified it?
Nemo iudex in causa sua — no one may be judge in their own cause.
Corral is an auditing engine. It judges code it did not write — by execution and by adversary — onto a record nobody can edit. Accountability, paper trail, analysis, certification: everything but building.
Two extensions carry the rule: certify breaks your code on purpose and runs your own tests against every fault; review hands your code to a model that has never seen it and makes every claim prove itself by script, with a second model arguing the other side. Both write signed, hash-linked entries to a branch in your repository — run as a GitHub Action, GitHub becomes the record, and any DuckDB reads it straight from the branch.
Below is a real run, replaying now: a decorrelated cross-vendor herd put internal/certify/certify.go to its own tests — gemini-3.6-flash planted 78 faults, and the project's own test suite was run against every one of them in a sandbox. No model graded anything: the suite's exit code decided, and it caught 43, missed 35. A different model, gemini-3.7-flash, reviewed the run alongside it — the critic advises and is never allowed to decide. Signed NEEDS-REVIEW, and the herd handed back a test that catches the gap.
We ran our own gate on our own signing code and published the result: NEEDS-REVIEW. 43 of 78 planted faults killed, 35 survived — and the writer handed back tests proving 30 of them catchable. The corrective-retry loop is visible mid-tape: an authored test failed on clean code and was reissued with the failure fed back.
“the signed audit statement is tamper-evident: any change to the verdict, the subject hashes, or the model names must break the ed25519 DSSE signature, and verification must fail closed”
Graded by
internal/certify/certify_test.go — the library’s own tests
The decorrelated herd
Planted the faults
gemini-3.6-flash
Wrote exposing tests
gemini-3.6-flash
Critiqued the suite different model
gemini-3.7-flash
What happened, in a jail
Planted 78 goal-violating mutants across the file.
Ran the developer’s own suite against every one — it killed 43 of 78 (a 55% kill-rate), measured by execution, not asserted.
Verdict: NEEDS-REVIEW — signed and offline-verifiable, and the herd handed back a compiling test that kills the gap.
What it does — and doesn’t — claim
35 survivors disclosed, unadjudicated: an injected mutant the suite didn’t kill — a real untested edge, or an equivalent mutant nothing can catch. Corral hands it to a human to judge; it doesn’t rule the code buggy.
Certified by execution — run in a jail, not taken on a model's word·Decorrelated — the judge is never the author; enforced, not hoped·Signed & offline-verifiable — tamper-evident, human-gated
Corral is the reactive audit you put downstream of every AI coding tool — agnostic to whoever, or whatever, wrote the change. It mutates the code to plant real faults and checks whether the developer's own tests catch them — a test that catches nothing proves nothing — while a decorrelated critic, a different model than the one that wrote the exposing test, reads the suite for theater. The verdict is measured by execution in a jail, never taken on a model's word. Any model, local 7B to frontier, mixed in one run. Human-gated and fail-closed; every run recorded, replayable, and signed into a record you verify offline. (Honest floor: corral certify is the shipped atom; the adversarial pool is experimental and off by default; the broader engine is ahead — we'll always tell you which.)
Both extensions, pointed at flask at 36e4a824 — a library we did not write — and both ending on the same frame: corral's own record, verified from a fresh clone. These are recordings of actual runs. The pipeline applies pacing and captions and nothing else; the first caption of each video says exactly what was applied.
certify — 56 seconds
Goals derived from app.py, then 40 deliberate faults planted in it and flask's own test suite run against every one of them in a jail. It caught 29 and slept through 11 — a kill rate of 0.72. A second model then wrote a test for each survivor and ran it alone against its fault: 10 of the 11 became demonstrated gaps, bugs the suite misses with the test that catches them attached. Ends with the signed verdict entered into Sigstore's public log (index 2759598612, the one you watch being issued) and the chain verified.
review — 84 seconds
Claude Code reviews src/flask/sessions.py cold and Codex argues the other side. Five claims come back — and the three the reviewer declared reproduced had scripts that did not demonstrate them, so corral demoted its own reviewer on the record: 0 reproduced, 5 standing as read. Then a person checks the top claim by hand — on an app with no secret key, session["a"]=1 raises the documented error but session |= {"a": 1} is silently accepted and the write dropped — and rules it real. The defect stood; the overclaim did not.
One run each, not a grade. Faults are generated fresh every time, so the same file scores differently on different days — this app.py has come back at 0.62, 0.72 and 0.75 across runs. That spread is the reason corral reports a proven gap, which does not drift, rather than asking you to gate on a kill rate, which does. Both runs above are entries on the public branch; query them and you will get these numbers and other runs' too.
Both close on the same frame: a fresh clone of corral/ledger, corral ledger verify, one SELECT over the entries, a push to MotherDuck, and the seats ranked by the record they left. How it works →
It runs on your pull requests
A principle nobody enforces is a slogan. The way corral enforces this one is a GitHub Action: it installs itself, audits only the files a pull request actually touched, and writes the verdict to the run page verbatim — including the lines that say what a number does not mean. Two steps, and there is no corral for you to install:
- uses: actions/checkout@v4
with:
fetch-depth: 0 # required — the diff needs a merge base
- uses: pdbethke/corralai@v1.0.0-rc.13
with:
test-command: "go test ./..."
anthropic-key: ${{ secrets.ANTHROPIC_API_KEY }}
max-proven-missed: "0" # fail on a DEMONSTRATED gap
attest: "true" # publish a receipt anyone can verify
max-proven-missed rather than a kill-rate threshold, and the difference matters. A kill rate is a proportion of freshly generated faults, so it moves between runs on code nobody touched — we have watched an unchanged file score 0.375 and then 0.525, and a deliberately weakened suite score higher than it had a moment earlier. Set a bar near a healthy value and it reddens good work until someone switches it off. A proven gap does not drift: it is a survivor the herd then killed with a test it wrote and ran — a specific demonstrated bug your suite misses.
attest publishes the verdict as a signed in-toto statement through GitHub's attestation API, keyless, chained to the repository and workflow rather than to a key on a disposable runner. A reviewer verifies it without trusting the log, or you:
It is written before the gate's exit code is honoured, so a failing audit still leaves a receipt. A verdict you only keep when it flatters you is not evidence.
What it said about this repository
We pointed it at the file that starts our own program, cmd/corral/main.go, as a gated check on a real commit. It planted 40 deliberate faults. Our suite caught 10 and slept through 30 — a kill rate of 0.25. Then the test-writer authored a test we hadn't written and used it to kill one of those survivors, so that gap is demonstrated rather than alleged. The whole run took 11m12s on a 2-core hosted runner.
That is a bad number, published by the tool that produced it, on a run we could not quietly not-publish. A gate its own author can talk his way past is furniture.
Scoped to the diff, on purpose
An audit costs roughly (mutants × your suite's whole runtime) per file — it scales with how long your tests take and how many files the PR touched, not with the size of the change. So the default is the PR's diff, top bounds how many files one run can spend on, and a whole-repo scan is something you opt into deliberately.
Without min-kill-rate, a graded file exits 0 no matter how weak it scored — read the report for the number. Set it and any single file below the threshold fails the run; a well-tested file elsewhere in the PR cannot average out a weak one. The bar is the operator's, never the tool's.
The bill lands on the repository that runs the workflow — your runner minutes, your API key. Fork pull requests skip on their own because GitHub withholds secrets, and the shipped example is label-gated on top of that, so no pull request starts a paid job merely by existing. Reaching for pull_request_target to "fix" that would run a contributor's code with your secrets.
A CI runner is a throwaway machine: it boots, it measures your code, it is destroyed. The usual way to keep what it learned is to stand up a database — a server, a schema, credentials in CI, a bill — to hold what is really just an append-only list of small facts. Corral doesn't stand anything up. Every verdict is committed to an orphan git branch in your own repository, corral/ledger. It shares no history with main, never appears in your diffs, and lives somewhere every runner already has credentials for. GitHub stops being just CI and becomes the record.
GitHub Actionwrites the entry→corral/ledgersigned, hash-linked→DuckDBreads it as tables→Rekorwitnesses it publicly→MotherDuckshares it across repos
DuckDB reads the branch in place
Entries are gzipped JSON, and DuckDB reads JSON over HTTPS — so the record is in an analytical engine from the first entry, with nothing installed and nothing running. No clone, no server, no schema to adopt. Point any DuckDB at the raw files:
SELECT kind, count(*) AS entries
FROM read_json_auto('scans/*.json.gz')
GROUP BY 1 ORDER BY 2 DESC;
The same query runs against the URLs straight off the branch, from any DuckDB client — the CLI, Python, R, or the WebAssembly build in a browser tab. Then the questions a single run can never answer are one GROUP BY each: which files have been weak across every scan, which model catches bugs in which language, what an audit costs per file, whose review claims held once execution checked them.
Signing proves integrity — the verdict is byte-for-byte the one produced. It does not prove independent authenticity, because the key is still ours. --transparency closes that gap: it uploads the signed statement to Sigstore's Rekor, a public append-only log run by people with no stake in the audit, and prints the receipt, and records it on the ledger entry. The flask audit on corral's own branch carries index 2759598612— read it off the entry, fetch it from Sigstore, and check it matches, without asking us for anything.
The entry is public and permanent: once logged it cannot be removed or edited, by anyone, including you. It carries the verdict — repo, commit, paths, kill rates, the models in each role — and never your source. And it fails open: an unreachable log prints one line and leaves the verdict untouched. A witness, not a gate.
The branch is checked out before the audit, so a fresh machine arrives holding everything the repository has learned about itself. Faults already planted on a file's exact bytes become the generator's prior, so it plants somewhere new instead of re-sitting the same exam. A verdict for a file whose bytes haven't changed is reused, and marked reused rather than re-billed. The review planner knows which scopes were covered and which have changed since. Every verdict says whether it was primed this way.
Each entry carries the hash of the one before it and is signed with your certify key. Edit an entry and its signature breaks; remove one and the next entry's link breaks. corral ledger verify walks the chain from a clean checkout and names which entry is wrong — a stranger checks the record against its own signatures, not against you. Nothing is edited in place: a wrong entry is retracted by a later entry that says so, and every reader leaves it out.
The branch is one repository's record. corral ledger push appends the same entries to a warehouse you own — a plain DuckDB file, or MotherDuck for the view across every repository that pushes. Rows are append-only, because overwriting is how a trend is lost. There is no hosted tier and nothing is collected: your key, your runner, your warehouse. Any DuckDB works, so MotherDuck is a destination rather than a requirement.
Fork it, mirror it, move to a self-hosted GitLab or Gitea, hand it to an auditor — the record goes along, because it is the repository. Each entry also names the audited party: author, committer and co-author trailers, so the record says who wrote the change beside who judged it. A person and an agent are named the same way.
Corral's own record is public at corral/ledger— audits, cold reviews, and the human rulings on them, in a branch you can query without cloning.
How the gate works
Point corral at a code change and the tests that are supposed to cover it. A decorrelated herd of models plants goal-violating faults in the code and runs your tests against each one in a jail — measuring the kill rate by execution, never by a self-report. A different model reads the suite cold and critiques it; any fault that survives gets a missing test written and proven to catch it. What comes out is a signed verdict: certified, or needs-review. No one may be judge in their own cause.
change + its tests
│
▼
[mutant-generator] plants goal-violating faults in the code
│ each fault ↓
▼
run YOUR tests against each fault, in a bwrap jail ──► kill-rate (never a self-report)
│
├── faults your tests killed → covered
└── survivors (a fault slipped past) → [test-writer] writes the missing
killing test, proven against the fault
│
▼
[test-critic — a DIFFERENT model reads the suite cold]
│
▼
SIGNED VERDICT ── certified (kill-rate clears the bar, no blocking gap)
└─ needs-review (a survivor still stands, or a human must adjudicate)
Decorrelated by construction
Run one LLM in every role and the "critic" shares the fault-planter's blind spots — the same model can't be surprised by its own bugs. Corral forces the three roles apart: the model that plants the faults, the model that writes the exposing test, and the model that critiques the suite are genuinely different models, enforced at assignment — CheckDecorrelation refuses a run where the critic and the test-writer are the same model. Each role can be pointed at a different vendor, so a Gemini fault-planter with a Claude critic is one run. Mix cheap local models with frontier ones: bring anything OpenAI-compatible, or a local model, on your own key — no vendor lock-in.
And decorrelate from whoever wrote the code. corral separates its own three roles; it cannot see the fourth participant — the model that authored the file under audit. If that was Claude and you audit with Claude, the lineage that wrote the code is planting the faults and grading the tests for it. Point the roles at a different vendor than your code came from.
A suite's grade isn't an opinion. The kill rate is mutation testing measured by the brain: your real command — go test, pytest, npm test, cargo test, anything — is run against each planted fault inside a bwrap jail, and the exit codes decide what died. A model never tells us it did well; a test that claims to catch a bug has to actually catch it, on a real run, on the record. Clear the bar with no blocking gap and the change is certified; fall short and it routes to a human as needs-review.
Survivors don't just get flagged — they get a test
A critic who only complains is furniture. When a planted fault slips past your suite, the test-writer writes the test you were missing and the brain proves it: runs the new test against the surviving fault and watches it fail, then against clean code and watches it pass. The gap comes back to you as proven_missed — a real, catchable bug and the exact test that would have caught it — not a lecture.
Every run ends in a signed record you can verify offline: the kill rate, the survivors, the written tests, and which model played which role — ProducedBy: [mutant-generator, test-critic, test-writer], each named. Because the verdict is attributed to a verified principal and the subject of the record doesn't control the ledger, it's a forensic artifact, not a screenshot. A judge may not certify her own cause, and she may not quietly edit the transcript either.
Every command corral runs — your tests against each mutant, the new test against the survivor — runs inside a bwrap jail: no network by default, workspace-confined, a secret-free environment. The kill rate is a fact about what happened in that jail, not a number a model asserted. Nothing about the grade rests on trusting the thing being graded.
The record says which model actually catches bugs in which seat, proven by execution, not self-reported. corral models rank reads that back and ranks the seats — a number you can't flatter your way up. It never staffs one: you name every seat, so no audit picks its own auditors. See below.
We pointed corral at real code across four languages — libraries we didn't write, plus our own — and graded each suite by execution: plant goal-violating faults, run the developer's own tests against them in a jail, sign the kill-rate. The first two rows are the same file twice: on a branch whose suite is green but thin it scored 75% and came back NEEDS-REVIEW; on main, one test richer, 90% and CERTIFIED. more-itertools certified at 90% too. The rest fell short of the bar and were sent back with their survivors disclosed — corral hands those to a human to judge, it doesn't rule them defects. Every verdict is a signed record you can replay and verify offline — pick any row.
Run it yourself — one command, off a single ANTHROPIC_API_KEY:
corral certify --local --code validator.py \
--goal "reject passwords under 8 chars, or missing a digit or an uppercase letter" \
--writer-model claude-sonnet-5 --mutant-model claude-sonnet-5 --critic-model claude-haiku-4-5 \
-- pytest
Every number above is a real signed record you can verify offline — read the write-up →
Quickstart
Install the CLI, hand it your key, and point it at a change plus the test that's supposed to guard it:
go install github.com/pdbethke/corralai/cmd/corral@v1.0.0-rc.13
export ANTHROPIC_API_KEY=sk-ant-...
corral certify --local --code path/to/file.go \
--goal "what it must guarantee" \
--writer-model claude-sonnet-5 --mutant-model claude-sonnet-5 --critic-model claude-haiku-4-5 \
-- go test ./...
Out comes a signed verdict — certified or needs-review — with the kill-rate, the survivors, and any written-and-proven killing test. Re-check the record offline, anytime:
corral certify verify record.json # re-runs the proof — no key, no network
The herd runs in a bwrap jail on Linux (--jail container for docker/podman elsewhere) and grades tests in six languages — Go, Python, Ruby, JavaScript, TypeScript, PHP — writing its killing test in your project's harness, so vitest, jest, pytest, minitest and PHPUnit work without configuration. Roles are decorrelation-enforced across models — the critic must differ from the writer or the run refuses — and each role can be pointed at a different vendor: a Gemini fault-planter with a Claude critic is a supported run, off your own keys.
Honest about the coverage behind that: Go and Python are exercised hardest — Go continuously in this repo's own CI, Python across repeated whole-repo scans of Flask. TypeScript, JavaScript, Ruby and PHP each rest on a single third-party repository so far, which shows the plugin works and is not evidence about the ecosystem. The evidence table is in the README →
Ready for more than one file? Install your project's dev dependencies first — the suite has to pass before corral can plant bugs against it — then point corral at the whole repo:
corral certify --repo . --substrate workspace \
--writer-model claude-sonnet-5 --mutant-model claude-sonnet-5 --critic-model claude-haiku-4-5 \
--derive-model claude-sonnet-5 \
-- python -m pytest # or your own test command
It measures its own models
Every audit is graded by execution, and the record says which model did what: which generator's faults survived, which writer's tests proved a gap, which critic's findings a person later confirmed — per seat, per language, never self-reported. corral models rank reads that evidence back from a warehouse you pushed to and ranks the seats. It prints a ranking; it never staffs one — every seat is named by you, and no audit picks its own auditors.
The optional brain daemon carries a second loop for a coordinated herd: recurring blind spots are swept into proposals, an operator approves or rejects them, and approved guidance becomes vetted memory and a versioned skill that a mission worker can search. That loop is brain-only — nothing in it is read by an audit, and corral does not yet measure whether a promoted lesson improves a later run. Mechanism, not a proven effect.
The knowledge corpus (CORRAL.md)
A repo that runs with corralai can carry its working knowledge as a markdown corpus in the repo itself: CORRAL.md at the root as the entry point, docs/corral/*.md as the corpus. The same corpus serves four readers — developers read it as onboarding docs, any developer's coding agent queries it conversationally over the optional brain, a mission worker on that brain can search it, and it grows the way code does — through ordinary pull requests, where code review is the trust gate for knowledge exactly as it is for code.
The learning loop closes the circle: skills the herd proposes and a human approves land in the same corpus — herd-discovered knowledge and developer-written knowledge accumulate in one place, under one review gate, readable by humans and queryable by every agent that joins the brain. An audit does not read it: corral certify builds its prompts from the goal and the code, never from the corpus — so nothing a repo ships can steer its own grading.
The optional daemon, for a fleet
Everything above runs with no server: an audit on a laptop or a CI runner, rows in a DuckDB you own. The brain is the optional daemon underneath a coordinated herd — remote workers, a live console, the human-gated proposal loop — and nothing in it is required for a verdict or read by one. If you want it: approved skills and guardrail hooks sync across a whole fleet — corral sync pulls every changed skill/hook, so what one machine learns, every machine's panel can do. Publishing to the fleet is superuser-only (a worker proposes, it can't publish), and hooks are staged for human review rather than auto-applied, because executable guardrails should never silently activate.
Independent brains can federate their signed records, so one gate can re-check another's verdicts offline without ceding control to a shared master. Because the record store takes a DSN, pointing it at MotherDuck instead of a local DuckDB file is a config flip rather than a port — which is what makes fleet-wide analytics and an "ask the fleet" natural-language oracle over the audit trail an option rather than a rewrite. Auth was designed in from day zero: OIDC against any provider, a principal allowlist, signed delegation tokens for spawned subagents, and a human gate that even a superuser's delegation token can't bypass for admin writes.
You don't need any of this to use corral — the gate runs in CI and hands back a verdict. But when a number surprises you, the run is on tape. The cockpit is a post-mortem instrument: scrub back through a finished audit, see which model played which role, and read the exact fault that survived next to the code your suite passed anyway. Every claim on this page is checkable against that tape rather than against marketing copy — it's the same replay code you'd point at your own runs.
The cockpit replaying the passwd-gappy audit. On the canvas, the decorrelated herd — a mutant-generator, a test-writer, and a separate test-critic — not a builder in sight. One scrub bar drives the whole thing.Click the mutant-generator's task and the surviving fault opens: the planted bug — len(p) < 8, quietly weakening the length rule — highlighted against the original. The dev's tests passed anyway.The verdict, in plain language: the suite killed 4 of 8 planted faults, 4 survived, and the record is signed — naming which model played which role. Not a green check; a receipt.
Nothing about a finished audit is thrown away: every fault the herd planted, every test run against it, every kill and every survivor, each command an agent actually ran, and the event log itself survive indefinitely. A signed verdict — certified or needs-review — sits at the end of a record you can list, open, and re-run. Replay is read-only: it reconstructs the whole audit from durable rows and plays it back on the same corral canvas, at up to 16×. And one scrub bar moves the whole cockpit — canvas, plan, files, and console all snap to the same instant in the run, so pausing live traffic and stepping through history is a single control, not four.
Then go deeper than the timeline: click any task — a row in the plan or a node on the canvas — and its whole story opens. What the herd was told to do, every command that ran under it, which agent (and which model) did it, what triggered it (the faults and upstream tasks it depended on), and what came next (the downstream work it unblocked). Every link is clickable, so you walk the causal chain a step at a time — the replay isn't a video, it's a queryable record of why the verdict came out the way it did.
A files lens rebuilds the directory tree the herd touched, each path colored by the agent that claimed it, filling in as you scrub — the code under audit and the tests grading it appearing side by side. And with story capture on, each agent's own reasoning streams verbatim in the console, interleaved with its commands: the italic lines scrolling past are the model thinking out loud. Scrub back to where a fault slipped past the suite and watch exactly which test failed to kill it — the replay is a debugger for your tests, not a demo reel.
This hero IS that player — the exact same code, replaying a real audit.
