Your first audit, in detail

This walks one real corral certify --local run start to finish, using the same password-validator example published in the field note “Terribly sorry to trouble you… but your tests, um, well. They suck, really”. Nothing here is staged for the docs — it’s the actual shape of a verdict this gate produced.

The code under review

Elastic-2.0

package passwd

import "unicode"

// Valid reports whether p is a valid password: length >= 12 AND it contains an
// uppercase letter, a lowercase letter, a digit, and a symbol.
func Valid(p string) bool {
  if len(p) < 12 {
    return false
  }
  var up, lo, di, sy bool
  for _, r := range p {
    switch {
    case unicode.IsUpper(r):
      up = true
    case unicode.IsLower(r):
      lo = true
    case unicode.IsDigit(r):
      di = true
    case unicode.IsPunct(r) || unicode.IsSymbol(r):
      sy = true
    }
  }
  return up && lo && di && sy
}

The developer’s own test

A test that only ever feeds a valid password:

func TestGappy_ValidLength(t *testing.T) {
  if !Valid("Abcdefgh1!xy") {
    t.Fatal("a 12-char password was rejected")
  }
}
func TestGappy_TooShort(t *testing.T) {
  if Valid("Ab1!xyz") {
    t.Fatal("a 7-char password was accepted")
  }
}

Green in CI. Looks reasonable — it checks a boundary (7 chars vs 12) and a happy-path accept.

Running it

export ANTHROPIC_API_KEY=sk-ant-...

corral certify --local \
  --code passwd.go \
  --test passwd_gappy_test.go \
  --goal "Valid must require length >= 12, an uppercase letter, a lowercase letter, a digit, and a symbol" \
  --out verdict.json \
  -- go test ./...

--code/--test may be absolute or relative paths. On Ubuntu 24.04, if bwrap won’t start (apparmor disables unprivileged user namespaces by default), add --jail container and export CORRALAI_EXEC_IMAGE=golang:1.26 to run the jailed check in Docker instead — no sudo required.

What happens inside the jail

1. Preflight. The Go toolchain is checked (Preflight() fails closed if go isn’t on the jail-visible PATH), and the sandbox backend is resolved — bwrap by default on Linux, refusing to fall back to unsandboxed if it can’t start.
2. Mutants. A mutant-generator model seeds goal-violating mutants into Valid — dropping the digit check, weakening the length bound, flipping the symbol test, and so on. On a file with several functions the generation fans out: one seat per group of functions (balanced by complexity), so every function is probed rather than just the one a single generator picked — --max-shards bounds the width, and it auto-sizes to your machine otherwise. A mutant that makes the suite hang (a broken loop bound, say) is killed fast and counted as caught, so a non-terminating mutant can’t stall the run.
3. Dev kill-rate. The existing test above is run, in the jail, against every mutant. It only ever calls Valid with a password that already satisfies every rule (or is simply too short) — so a mutant that loosens a rule (say, dropping the digit requirement) still passes: the input never exercised the rule it broke. Several mutants survive — a survivor is a planted mutant your suite didn’t catch. corral discloses it unadjudicated: it may be a real untested edge, or an equivalent mutant (a change nothing can observe, so no test could ever catch it). Which one is the human’s call, not the tool’s — corral won’t call an unproven survivor a real bug.
4. Test-writer. For each surviving mutant, a test-writer model (a different model than the critic, enforced — decorrelation isn’t optional) writes a targeted test; if its first attempt doesn’t compile, the compiler’s own error is fed back so it corrects rather than blindly repeating. When the jail confirms the test kills the mutant, that survivor is now a proven, catchable gap — not a fluke of the harness — and the test is handed back for you to add.
5. Test-critic. A decorrelated critic model reads the dev’s test file cold and flags what it would have said in review: the suite never tries an invalid password that’s the right length. Its findings are carried as an unverified second opinion — advisory, never gating the signed verdict, which is execution-proven only.
6. Shadow challenger. In parallel, a second (cheaper) model attacks the same regions — a same-ground, head-to-head measurement of one model against another. It’s recorded for comparison and never enters the verdict; the exam’s difficulty is set by the primary model alone, so certification means exactly what it did before. --shadow-model off turns it off.

Whatever happens, the run converges to a signed verdict — even if the herd can’t author a compiling test to kill a survivor, it routes to needs-review and hands back what it found rather than spinning.

The verdict

dev_kill_rate = 0.75   (killed 6 of 8 mutants, 2 survived)
status: needs-review   (below the 0.8 certify bar)

Below the bar, so the run routes to a human instead of auto-certifying — the honest failure mode. The kill-rate is measured, not asserted: it’s mutation testing executed in the sandbox, never the model’s self-report of how good the test is.

The signed record

The verdict is written to a local, tamper-evident ledger (the same certify chain corral certify uses for the non---local path) and printed to stdout — subject (repo/commit), the models that played each role, the kill-rate, and the status. Because you passed --out, the signed record is also written as a self-contained file, which re-verifies any time, fully offline (the run prints this line for you, with your key filled in):

corral certify verify verdict.json --pubkey "$(corral certify pubkey)" --allow-unanchored

--allow-unanchored is required: a --local record is signed by your own key but never submitted to a public transparency log, so verify makes you opt in to that weaker “signed-by-you, not third-party-witnessed” trust explicitly.

Try it on your own code

Point --code and --test at a real file and its test, describe the guarantee in --goal, and pass whatever check command actually exercises it (go test ./..., pytest, npm test, …). See Getting started for the jail-visibility gotcha (system-installed toolchain, not a --user/snap install) and the Ubuntu-apparmor fix if bwrap won’t start.