So Corral doesn't build your code — it certifies a change by execution: a decorrelated herd grades the tests meant to guard it, in a jail, and signs a tamper-evident verdict. Not a builder — a gate.
Below is a real run, replaying now: a decorrelated cross-vendor herd put more_itertools/recipes.py to its own tests — gemini-3.5-flash planted 20 faults and a different model, gemini-3.1-pro-preview, graded whether the suite catches them. It caught 18, missed 2. Signed CERTIFIED.
Planted 20 goal-violating mutants across the file.
Ran the developer’s own suite against every one — it killed 18 of 20 (a 90% kill-rate), measured by execution, not asserted.
Verdict: CERTIFIED — signed and offline-verifiable.
What it does — and doesn’t — claim
2 survivors disclosed, unadjudicated: an injected mutant the suite didn’t kill — a real untested edge, or an equivalent mutant nothing can catch. Corral hands it to a human to judge; it doesn’t rule a well-loved library buggy.
The decorrelated critic flagged 1 possible dead test — advisory, unverified, and structurally barred from gating the signed verdict. In `test_full_permutation`, `i` is a `range` object (`range(15)`) and `r` is a `tuple` returned by `random_permutation`. In Python 3, `range` objects are never equal to `tuple` objects, so `i == r` is…
Certified by execution — run in a jail, not taken on a model's word·Decorrelated — the judge is never the author; enforced, not hoped·Signed & offline-verifiable — tamper-evident, human-gated
Corral is the reactive audit you put downstream of every AI coding tool — agnostic to whoever, or whatever, wrote the change. It mutates the code to plant real faults and checks whether the developer's own tests catch them — a test that catches nothing proves nothing — while a decorrelated critic, a different model than the one that wrote the exposing test, reads the suite for theater. The verdict is measured by execution in a jail, never taken on a model's word. Any model, local 7B to frontier, mixed in one run. Human-gated and fail-closed; every run recorded, replayable, and signed into a record you verify offline. (Honest floor: corral certify is the shipped atom; the adversarial pool is experimental and off by default; the broader engine is ahead — we'll always tell you which.)
A principle nobody enforces is a slogan. The way corral enforces this one is a GitHub Action: it installs itself, audits only the files a pull request actually touched, and writes the verdict to the run pageverbatim — including the lines that say what a number doesnot mean. Two steps, and there is no corral for you to install:
- uses: actions/checkout@v4
with:
fetch-depth: 0 # required — the diff needs a merge base
- uses: pdbethke/corralai@v0.3.5
with:
test-command: "go test ./..."
anthropic-key: ${{ secrets.ANTHROPIC_API_KEY }}
min-kill-rate: "0.7" # opt-in teeth: fails the run, per file
What it said about this repository
We pointed it at the file that starts our own program,cmd/corral/main.go, as a gated check on a real commit. It planted 40 deliberate faults. Our suite caught10 and slept through 30 — a kill rate of0.25. Then the test-writer authored a test we hadn't written and used it to kill one of those survivors, so that gap is demonstrated rather than alleged. The whole run took11m12s on a 2-core hosted runner.
That is a bad number, published by the tool that produced it, on a run we could not quietly not-publish. A gate its own author can talk his way past is furniture.
Scoped to the diff, on purpose
An audit costs roughly (mutants × your suite's whole runtime)per file — it scales with how long your tests take and how many files the PR touched, not with the size of the change. So the default is the PR's diff, top bounds how many files one run can spend on, and a whole-repo scan is something you opt into deliberately.
Without min-kill-rate, a graded file exits 0 no matter how weak it scored — read the report for the number. Set it and any single file below the threshold fails the run; a well-tested file elsewhere in the PR cannot average out a weak one. The bar is the operator's, never the tool's.
The bill lands on the repository that runs the workflow — your runner minutes, your API key. Fork pull requests skip on their own because GitHub withholds secrets, and the shipped example is label-gated on top of that, so no pull request starts a paid job merely by existing. Reaching for pull_request_target to "fix" that would run a contributor's code with your secrets.
Point corral at a code change and the tests that are supposed to cover it. A decorrelated herd of models plants goal-violating faults in the code and runs your tests against each one in a jail — measuring the kill rate by execution, never by a self-report. A different model reads the suite cold and critiques it; any fault that survives gets a missing test written and proven to catch it. What comes out is a signed verdict: certified, or needs-review. No one may be judge in their own cause.
change + its tests
│
▼
[mutant-generator] plants goal-violating faults in the code
│ each fault ↓
▼
run YOUR tests against each fault, in a bwrap jail ──► kill-rate (never a self-report)
│
├── faults your tests killed → covered
└── survivors (a fault slipped past) → [test-writer] writes the missing
killing test, proven against the fault
│
▼
[test-critic — a DIFFERENT model reads the suite cold]
│
▼
SIGNED VERDICT ── certified (kill-rate clears the bar, no blocking gap)
└─ needs-review (a survivor still stands, or a human must adjudicate)
Decorrelated by construction
Run one LLM in every role and the "critic" shares the fault-planter's blind spots — the same model can't be surprised by its own bugs. Corral forces the three roles apart: the model that plants the faults, the model that writes the exposing test, and the model that critiques the suite are genuinely different models, enforced at assignment — CheckDecorrelation refuses a run where the critic and the test-writer are the same model. Each role can be pointed at a different vendor, so a Gemini fault-planter with a Claude critic is one run. Mix cheap local models with frontier ones: bring anything OpenAI-compatible, or a local model, on your own key — no vendor lock-in.
And decorrelate from whoever wrote the code. corral separates its own three roles; it cannot see the fourth participant — the model that authored the file under audit. If that was Claude and you audit with Claude, the lineage that wrote the code is planting the faults and grading the tests for it. Point the roles at a different vendor than your code came from.
A suite's grade isn't an opinion. The kill rate is mutation testing measured by the brain: your real command —go test, pytest, npm test,cargo test, anything — is run against each planted fault inside a bwrap jail, and the exit codes decide what died. A model never tells us it did well; a test that claims to catch a bug has to actually catch it, on a real run, on the record. Clear the bar with no blocking gap and the change is certified; fall short and it routes to a human as needs-review.
Survivors don't just get flagged — they get a test
A critic who only complains is furniture. When a planted fault slips past your suite, the test-writer writes the test you were missing and the brain proves it: runs the new test against the surviving fault and watches it fail, then against clean code and watches it pass. The gap comes back to you asproven_missed — a real, catchable bug and the exact test that would have caught it — not a lecture.
Every run ends in a signed record you can verify offline: the kill rate, the survivors, the written tests, and which model played which role — ProducedBy: [mutant-generator, test-critic, test-writer], each named. Because the verdict is attributed to a verified principal and the subject of the record doesn't control the ledger, it's a forensic artifact, not a screenshot. A judge may not certify her own cause, and she may not quietly edit the transcript either.
Every command corral runs — your tests against each mutant, the new test against the survivor — runs inside a bwrap jail: no network by default, workspace-confined, a secret-free environment. The kill rate is a fact about what happened in that jail, not a number a model asserted. Nothing about the grade rests on trusting the thing being graded.
The signed verdicts feed a leaderboard: which model actuallycatches bugs in which role, proven by execution, not self-reported. Over time the herd routes each role to the model that's measured-best at it — fitness earned at the gate, a number you can't flatter your way up. See how the loop works below.
We pointed corral at real code across four languages — libraries we didn't write, plus our own — and graded each suite by execution: plant goal-violating faults, run the developer's own tests against them in a jail, sign the kill-rate. more-itertools' suite caught 90% andCERTIFIED. The rest fell short of the bar and were sent back with their survivors disclosed — corral hands those to a human to judge, it doesn't rule them defects. Every verdict is a signed record you can replay and verify offline — pick any row.
Run it yourself — one command, off a single ANTHROPIC_API_KEY:
corral certify --local --code validator.py \
--goal "reject passwords under 8 chars, or missing a digit or an uppercase letter" \
-- pytest
Every number above is a real signed record you can verify offline —read the write-up →
Quickstart
Install the CLI, hand it your key, and point it at a change plus the test that's supposed to guard it:
go install github.com/pdbethke/corralai/cmd/corral@latest
export ANTHROPIC_API_KEY=sk-ant-...
corral certify --local --code path/to/file.go \
--goal "what it must guarantee" -- go test ./...
Out comes a signed verdict — certified orneeds-review — with the kill-rate, the survivors, and any written-and-proven killing test. Re-check the record offline, anytime:
corral certify verify record.json # re-runs the proof — no key, no network
The herd runs in a bwrap jail on Linux (--jail container for docker/podman elsewhere) and grades tests in five languages — Go, Python, Ruby, JavaScript, TypeScript — writing its killing test inyour project's harness, so vitest, jest, pytest and minitest work without configuration. Roles are decorrelated across models by default, and each role can be pointed at a different vendor: a Gemini fault-planter with a Claude critic is a supported run, off your own keys.
Honest about the coverage behind that: Go and Python are exercised hardest — Go continuously in this repo's own CI, Python across repeated whole-repo scans of Flask. TypeScript, JavaScript and Ruby each rest on a single third-party repository so far, which shows the plugin works and is not evidence about the ecosystem.The evidence table is in the README →
It learns
Every audit is graded by execution, and the results feed agate-earned leaderboard: corral tracks which model actually catches bugs — per role, per language, measured from real kill-rates and survivors, never self-reported — and routes each audit's roles (fault-planter, test-critic) to the best-fit model for the job. Recurring blind spots and clusters of similar lessons are swept into proposals: an LLM drafts corrective guidance plus a reusable skill, and the operator approves or rejects it — from a Proposals tab (a live count badge) or corral-admin proposals. Approval promotes the guidance into vetted memory and a versioned skill artifact that later audits carry (fence-wrapped, clearly labeled, capped at 3). And the loop watches its own efficacy: if the same blind spot keeps surviving after promotion, a revision proposal reopens for the human to reconsider.
The knowledge corpus (CORRAL.md)
A repo that runs with corralai can carry its working knowledge as a markdown corpus in the repo itself: CORRAL.md at the root as the entry point, docs/corral/*.md as the corpus. The same corpus serves four readers — developers read it as onboarding docs, any developer's coding agent queries it conversationally, the herd itself searches it before working and extends memory as it learns, and it grows the way code does — through ordinary pull requests, where code review is the trust gate for knowledge exactly as it is for code.
The learning loop closes the circle: skills the herd proposes and a human approves land in the same corpus — herd-discovered knowledge and developer-written knowledge accumulate in one place, under one review gate, readable by humans and queryable by every agent that joins.
Afterwards: the post-mortem
You don't need any of this to use corral — the gate runs in CI and hands back a verdict. But when a number surprises you, the run is on tape. The cockpit is a post-mortem instrument: scrub back through a finished audit, see which model played which role, and read the exact fault that survived next to the code your suite passed anyway. Every claim on this page is checkable against that tape rather than against marketing copy — it's the same replay code you'd point at your own runs.
The cockpit replaying the passwd-gappy audit. On the canvas, the decorrelated herd — a mutant-generator, a test-writer, and a separate test-critic — not a builder in sight. One scrub bar drives the whole thing.Click the mutant-generator's task and the surviving fault opens: the planted bug — len(p) < 8, quietly weakening the length rule — highlighted against the original. The dev's tests passed anyway.The verdict, in plain language: the suite killed 4 of 8 planted faults, 4 survived, and the record is signed — naming which model played which role. Not a green check; a receipt.
Nothing about a finished audit is thrown away: every fault the herd planted, every test run against it, every kill and every survivor, each command an agent actually ran, and the event log itself survive indefinitely. A signed verdict — certified or needs-review — sits at the end of a record you can list, open, and re-run. Replay is read-only: it reconstructs the whole audit from durable rows and plays it back on the same corral canvas, at up to 16×. And one scrub bar moves the whole cockpit — canvas, plan, files, and console all snap to the same instant in the run, so pausing live traffic and stepping through history is a single control, not four.
Then go deeper than the timeline: click any task — a row in the plan or a node on the canvas — and its whole story opens. What the herd was told to do, every command that ran under it, which agent (and which model) did it, what triggered it (the faults and upstream tasks it depended on), and what came next (the downstream work it unblocked). Every link is clickable, so you walk the causal chain a step at a time — the replay isn't a video, it's a queryable record of why the verdict came out the way it did.
A files lens rebuilds the directory tree the herd touched, each path colored by the agent that claimed it, filling in as you scrub — the code under audit and the tests grading it appearing side by side. And with story capture on, each agent's own reasoning streams verbatim in the console, interleaved with its commands: the italic lines scrolling past are the model thinking out loud. Scrub back to where a fault slipped past the suite and watch exactly which test failed to kill it — the replay is a debugger for your tests, not a demo reel.
This hero IS that player — the exact same code, replaying a real audit.
Dev infrastructure for a fleet, not a toy for a laptop
Corral isn't scoped to one machine's audit brain. Approved skills and guardrail hooks sync across a whole fleet — corral syncpulls every changed skill/hook, so what one machine's herd learns, every machine's herd can do. Publishing to the fleet is superuser-only (a worker proposes, it can't publish), and hooks are staged for human review rather than auto-applied, because executable guardrails should never silently activate.
Independent brains can federate their signed records, so one gate can re-check another's verdicts offline without ceding control to a shared master. Because the record store takes a DSN, pointing it atMotherDuck instead of a local DuckDB file is a config flip rather than a port — which is what makes fleet-wide analytics and an "ask the fleet" natural-language oracle over the audit trail an option rather than a rewrite. Auth was designed in from day zero: OIDC against any provider, a principal allowlist, signed delegation tokens for spawned subagents, and a human gate that even a superuser's delegation token can't bypass for admin writes.
