eval at cf9f18e59c, reviewed by claude-code, verified by codex.
The manifest's `code_path` is relative to eval/corpus while its `test_cmd` is rooted at the repo, so the mutant is written to a file the check command never compiles — every in-corpus target scores a mutant that cannot affect the suite.
set -u cp eval/corpus/passwd/passwd.go /tmp/zz_orig.go mkdir -p passwd sed 's/len(p) < 12/len(p) < 4/' /tmp/zz_orig.go > passwd/passwd.go echo "=== mutant written at the manifest's code_path 'passwd/passwd.go', applied at the run root ===" if go test ./eval/corpus/passwd/... -run Thorough -count=1 > /tmp/zz_a.txt 2>&1; then echo "SUITE PASSED - the mutant at code_path had NO effect on what test_cmd runs:" cat /tmp/zz_a.txt A=0 else echo "suite failed (unexpected):"; cat /tmp/zz_a.txt; A=1 fi echo "=== the SAME mutant applied where test_cmd actually compiles ===" sed 's/len(p) < 12/len(p) < 4/' /tmp/zz_orig.go > eval/corpus/passwd/passwd.go if go test ./eval/corpus/passwd/... -run Thorough -count=1 > /tmp/zz_b.txt 2>&1; then echo "suite passed (unexpected)"; cat /tmp/zz_b.txt; B=1 else echo "SUITE FAILED - the same mutant is lethal at eval/corpus/passwd/passwd.go:" cat /tmp/zz_b.txt B=0 fi [ "$A" = 0 ] && [ "$B" = 0 ] || exit 1 echo "EVIDENCE: manifest code_path 'passwd/passwd.go' and test_cmd 'go test ./eval/corpus/passwd/...' do not name the same file; a mutant scored at code_path is unkillable by construction." exit 0
=== mutant written at the manifest's code_path 'passwd/passwd.go', applied at the run root ===
SUITE PASSED - the mutant at code_path had NO effect on what test_cmd runs:
ok github.com/pdbethke/corralai/eval/corpus/passwd 0.002s
=== the SAME mutant applied where test_cmd actually compiles ===
SUITE FAILED - the same mutant is lethal at eval/corpus/passwd/passwd.go:
--- FAIL: TestThorough_TooShort (0.00s)
passwd_thorough_test.go:13: short accepted
FAIL
FAIL github.com/pdbethke/corralai/eval/corpus/passwd 0.002s
FAIL
EVIDENCE: manifest code_path 'passwd/passwd.go' and test_cmd 'go test ./eval/corpus/passwd/...' do not name the same file; a mutant scored at code_path is unkillable by construction.I traced Load through cmd/corral/eval.go into the scoring map. Load reads code relative to the manifest directory, but RunOne passes the unchanged CodePath into advStartSpec, and Score writes compliant and mutant code under that unchanged key. The passwd and interval test commands name repo-rooted eval/corpus packages, so I found no path rebasing that could make those keys address the compiled files.
ref b5c32ad68564e1cc8417cb26c79a260eee61df53749562e0a61e149e11d0c4f1#R1
The `fence-neutralization` target's code_path `../../internal/fence/fence.go` escapes the workspace and is refused by the scoring workspace writer, so the corpus's first target can never be materialised or graded.
set -u
cat > internal/adequacy/zz_review_codepath_test.go <<'EOF'
package adequacy
import (
"context"
"encoding/json"
"os"
"strings"
"testing"
"github.com/pdbethke/corralai/internal/sandbox"
)
type zzReviewIso struct{}
func (zzReviewIso) Wrap(command string, opts sandbox.Options, env []string) ([]string, error) {
return []string{"sh", "-c", "true"}, nil
}
func (zzReviewIso) Preflight() error { return nil }
func (zzReviewIso) Name() string { return "zzreview" }
func TestZZReviewCorpusCodePathRefused(t *testing.T) {
raw, err := os.ReadFile("../../eval/corpus/manifest.json")
if err != nil {
t.Fatal(err)
}
var m struct {
Targets []struct {
ID string `json:"id"`
CodePath string `json:"code_path"`
} `json:"targets"`
}
if err := json.Unmarshal(raw, &m); err != nil {
t.Fatal(err)
}
j := NewJail(zzReviewIso{}, 0)
refused := 0
for _, tg := range m.Targets {
_, rerr := j.RunTest(context.Background(),
map[string]string{tg.CodePath: "package x\n"}, []string{"true"})
if rerr != nil && strings.Contains(rerr.Error(), "outside workspace") {
refused++
t.Logf("EVIDENCE: corpus target %q code_path %q: %v", tg.ID, tg.CodePath, rerr)
}
}
if refused == 0 {
t.Fatal("no corpus code_path was refused by the scoring workspace")
}
t.Logf("EVIDENCE: %d of %d corpus targets can never be materialised into a scoring workspace", refused, len(m.Targets))
}
EOF
go test ./internal/adequacy/ -run TestZZReviewCorpusCodePathRefused -v -count=1 > /tmp/zz2.txt 2>&1 || { cat /tmp/zz2.txt; exit 1; }
cat /tmp/zz2.txt
grep -q "outside workspace" /tmp/zz2.txt || exit 1
exit 0=== RUN TestZZReviewCorpusCodePathRefused
zz_review_codepath_test.go:42: EVIDENCE: corpus target "fence-neutralization" code_path "../../internal/fence/fence.go": adequacy: refusing to write file outside workspace: "../../internal/fence/fence.go"
zz_review_codepath_test.go:48: EVIDENCE: 1 of 5 corpus targets can never be materialised into a scoring workspace
--- PASS: TestZZReviewCorpusCodePathRefused (0.00s)
PASS
ok github.com/pdbethke/corralai/internal/adequacy 0.007sI checked the complete first-target path: Load can read ../../internal/fence/fence.go relative to eval/corpus, but RunOne preserves that escaping path as CodePath. The adequacy materializers reject traversal, and eval.Run immediately returns when runner.RunOne returns an error. I found no normalization or special materialization path for eval targets.
ref b5c32ad68564e1cc8417cb26c79a260eee61df53749562e0a61e149e11d0c4f1#R2
Go's passwd.Valid measures BYTES (`len(p)`), not characters, so it accepts an 11-character password and contradicts the goal string the manifest signs the run with.
set -u
cat > eval/corpus/passwd/zz_review_len_test.go <<'EOF'
package passwd
import (
"testing"
"unicode/utf8"
)
func TestZZReviewByteLengthNotCharLength(t *testing.T) {
const p = "Ábcdefgh1!x"
n := utf8.RuneCountInString(p)
if n != 11 {
t.Fatalf("fixture has %d characters, expected 11", n)
}
if !Valid(p) {
t.Fatal("no defect: the 11-character password was rejected")
}
t.Logf("EVIDENCE: Valid(%q) = true, but the password is %d CHARACTERS (the goal requires >= 12); it is %d BYTES, which is what len() counted", p, n, len(p))
}
EOF
go test ./eval/corpus/passwd/ -run TestZZReviewByteLengthNotCharLength -v -count=1 > /tmp/zz3.txt 2>&1 || { cat /tmp/zz3.txt; exit 1; }
cat /tmp/zz3.txt
grep -q "EVIDENCE" /tmp/zz3.txt || exit 1
exit 0=== RUN TestZZReviewByteLengthNotCharLength
zz_review_len_test.go:17: EVIDENCE: Valid("Ábcdefgh1!x") = true, but the password is 11 CHARACTERS (the goal requires >= 12); it is 12 BYTES, which is what len() counted
--- PASS: TestZZReviewByteLengthNotCharLength (0.00s)
PASS
ok github.com/pdbethke/corralai/eval/corpus/passwd 0.002sThe implementation uses len(p), which counts UTF-8 bytes, while it iterates character classes as runes. The README explicitly describes the threshold as characters, and the supplied 11-rune/12-byte input therefore passes the length gate. I found no ASCII-only constraint in the goal or API that narrows the specification enough to refute this.
ref b5c32ad68564e1cc8417cb26c79a260eee61df53749562e0a61e149e11d0c4f1#R3
The `passwd-thorough` target is labelled expected_adequacy=thorough with no known_gap, yet a live mutant weakening the length bound from 12 to 11 survives its entire suite — so a pool that correctly finds that survivor is reported as "inventing gaps (over-sensitive)".
set -u
mkdir -p eval/corpus/zzthorough
sed 's/len(p) < 12/len(p) < 11/' eval/corpus/passwd/passwd.go > eval/corpus/zzthorough/passwd.go
cp eval/corpus/passwd/passwd_thorough_test.go eval/corpus/zzthorough/
cat > eval/corpus/zzthorough/zz_live_test.go <<'EOF'
package passwd
import "testing"
func TestZZMutantIsLive(t *testing.T) {
const p = "Abcdefg1!xy"
if len(p) != 11 {
t.Fatalf("fixture is %d bytes, expected 11", len(p))
}
if !Valid(p) {
t.Fatal("the mutant is not semantically live")
}
t.Logf("EVIDENCE: mutated Valid(%q) = true - an 11-character password is now accepted, so the mutant changes behaviour", p)
}
EOF
echo "=== the passwd-thorough suite, run against the mutant 'len(p) < 12' -> 'len(p) < 11' ==="
go test ./eval/corpus/zzthorough/ -run Thorough -v -count=1 > /tmp/zz4a.txt 2>&1 || { echo "the suite FAILED - the mutant was killed, no defect"; cat /tmp/zz4a.txt; exit 1; }
cat /tmp/zz4a.txt
echo "=== proving the mutant is not equivalent ==="
go test ./eval/corpus/zzthorough/ -run ZZMutantIsLive -v -count=1 > /tmp/zz4b.txt 2>&1 || { echo "mutant not live"; cat /tmp/zz4b.txt; exit 1; }
cat /tmp/zz4b.txt
echo "EVIDENCE: every test of the 'thorough' suite passes on a behaviour-changing mutant; the target declares no known_gap and no expected_survivors, so report.go will read a correct survivor as pool over-sensitivity."
exit 0=== the passwd-thorough suite, run against the mutant 'len(p) < 12' -> 'len(p) < 11' ===
the suite FAILED - the mutant was killed, no defect
=== RUN TestThorough_Valid
--- PASS: TestThorough_Valid (0.00s)
=== RUN TestThorough_TooShort
passwd_thorough_test.go:13: short accepted
--- FAIL: TestThorough_TooShort (0.00s)
=== RUN TestThorough_NoUpper
--- PASS: TestThorough_NoUpper (0.00s)
=== RUN TestThorough_NoLower
--- PASS: TestThorough_NoLower (0.00s)
=== RUN TestThorough_NoDigit
--- PASS: TestThorough_NoDigit (0.00s)
=== RUN TestThorough_NoSymbol
--- PASS: TestThorough_NoSymbol (0.00s)
FAIL
FAIL github.com/pdbethke/corralai/eval/corpus/zzthorough 0.002s
FAILThe reviewer's recorded failure does not refute the finding: R1's script overwrote eval/corpus/passwd/passwd.go with a <4 mutant and never restored it, explaining the later 'short accepted' result. On a clean checkout, changing <12 to <11 still rejects the suite's 7-character short fixture, accepts its 12-character valid fixture, and leaves every character-class check unchanged. No thorough test supplies an 11-character password, so the claimed live boundary mutant survives.
ref b5c32ad68564e1cc8417cb26c79a260eee61df53749562e0a61e149e11d0c4f1#R4
The five 'same validator' fixtures do not implement the same rule: Go treats a space as not-a-symbol (unicode.IsPunct/IsSymbol) while Python/Ruby/JS/TS count any non-alphanumeric, so the same input gets opposite verdicts across the corpus the README says is one validator in five languages.
set -u
command -v python3 >/dev/null 2>&1 || { echo "python3 unavailable - cannot demonstrate here"; exit 1; }
mkdir -p eval/corpus/zzcmp
cat > eval/corpus/zzcmp/main.go <<'EOF'
package main
import (
"fmt"
"github.com/pdbethke/corralai/eval/corpus/passwd"
)
func main() {
if passwd.Valid("Abcdefgh1 xy") {
fmt.Println("true")
return
}
fmt.Println("false")
}
EOF
GOV=$(go run ./eval/corpus/zzcmp 2>/tmp/zz5err.txt) || { cat /tmp/zz5err.txt; exit 1; }
PYV=$(cd eval/corpus/passwd_py && python3 -c "import passwd; print('true' if passwd.valid('Abcdefgh1 xy') else 'false')") || exit 1
echo "input: 'Abcdefgh1 xy' (12 characters: upper, lower, digit, and a space)"
echo " Go eval/corpus/passwd/passwd.go -> $GOV"
echo " Py eval/corpus/passwd_py/passwd.py -> $PYV"
[ "$GOV" = "false" ] && [ "$PYV" = "true" ] || exit 1
echo "EVIDENCE: the README calls these 'the same password validator ... in five languages' under one goal string, but Go's unicode.IsPunct||IsSymbol and Python's 'not c.isalnum()' disagree on this input; the JS/TS/Ruby fixtures use the Python rule, so Go is the odd one out and any cross-language claim built on this corpus compares different specifications."
exit 0input: 'Abcdefgh1 xy' (12 characters: upper, lower, digit, and a space) Go eval/corpus/passwd/passwd.go -> false Py eval/corpus/passwd_py/passwd.py -> true EVIDENCE: the README calls these 'the same password validator ... in five languages' under one goal string, but Go's unicode.IsPunct||IsSymbol and Python's 'not c.isalnum()' disagree on this input; the JS/TS/Ruby fixtures use the Python rule, so Go is the odd one out and any cross-language claim built on this corpus compares different specifications.
The implementations genuinely disagree on the named space-containing input. Go recognizes a symbol only through unicode.IsPunct or unicode.IsSymbol, whereas Python uses non-alphanumeric, Ruby uses a negated alphanumeric class, and JavaScript/TypeScript use /[^A-Za-z0-9]/. The README calls them the same validator under the same stated rule, and I found no documented definition of symbol that excuses the divergent verdict.
ref b5c32ad68564e1cc8417cb26c79a260eee61df53749562e0a61e149e11d0c4f1#R5
eval/corpus/passwd_ts/passwd.test.ts imports "./passwd.ts" with an explicit .ts extension, which tsc rejects (TS5097) unless allowImportingTsExtensions is set, and the corpus ships no tsconfig.json — so the README's documented "tsc + node:test" TypeScript reproduction cannot compile as given.
The claim overlooks the execution workflow documented by the README. The command is corral certify --local, not bare tsc against the corpus directory. The TypeScript plugin's Scaffold method creates tsconfig.json with allowImportingTsExtensions:true, and its scoped CompileCheck independently generates a temporary project containing the same option. Thus the absence of a checked-in corpus tsconfig does not make the documented reproduction uncompilable; corral supplies exactly the option TS5097 requires (internal/lang/typescript.go:19-27 and 83-96).
ref b5c32ad68564e1cc8417cb26c79a260eee61df53749562e0a61e149e11d0c4f1#R6
The corpus calls itself versioned but its version is not a function of its content: the fence target's code_path points outside the corpus at live engine source, while the resume key is (corpus_version, target id, iteration) with no target digest, so re-runs after the engine changes are pooled into one report as if measured against the same corpus.
There is a narrower stale-progress defect, but the stated pooling/provenance claim is false. Target.Digest hashes code, test code, goal, and test command, and mcpPoolRunner submits Commit as target ID plus that digest (internal/eval/corpus.go:83-91; cmd/corral/eval.go:20-34), so engine-source changes produce distinct run identities. Progress does omit the digest, but eval.Run neither loads old RunResults nor combines them with new ones: it merely skips marked work and reports results collected during the current invocation (internal/eval/harness.go:95-127). Therefore changed and unchanged measurements are not pooled into one report 'as if measured against the same corpus'; the actual narrower risk is that unchanged corpus_version can cause a changed target to be skipped.
ref b5c32ad68564e1cc8417cb26c79a260eee61df53749562e0a61e149e11d0c4f1#R7
expected_survivors is only ever used as a floor (report.go:132 flags MeanSurvivors below it and nothing above it), and both gappy targets declare 1 while their suites in fact admit several independent survivors, so an arbitrarily over-sensitive pool on a gappy target is reported Calibrated.
I followed every ExpectedSurvivors use. For gappy targets report.go rejects only a mean below the declared floor, then requires at least one writer-graded run and at least one proven miss; it places no ceiling on survivor count. Consequently a run with arbitrarily many survivors can still be marked calibrated once those minimum conditions hold. The gappy fixtures also omit multiple independent requirements or boundaries, so declaring only one expected survivor does not constrain over-sensitivity.
ref b5c32ad68564e1cc8417cb26c79a260eee61df53749562e0a61e149e11d0c4f1#R8
Recorded so that "found nothing here" can be told apart from "never looked here".
This corpus is the thing corral grades itself against, so a defect here is a defect in every calibration verdict downstream. Three of its problems are structural rather than cosmetic. Finding 1 is the worst: `code_path` and `test_cmd` in manifest.json are written against two different roots — `code_path` is resolved relative to the manifest directory (`eval/corpus`) when Load reads the bytes, but the very same string is used as the workspace write key (`internal/adequacy/score.go:658`, `files[codePath] = code`), where the root is the checkout. So the mutant lands at `<root>/passwd/passwd.go` while `go test ./eval/corpus/passwd/...` compiles a file that was never touched; I demonstrate that the identical mutant is invisible at one path and lethal at the other. Finding 2 is the same disease in its terminal form: the first target's `code_path` is `../../internal/fence/fence.go`, which both materializers refuse outright — the jail with an explicit "refusing to write file outside workspace", the workspace runner via `os.OpenRoot`. That target is the first in the list and `eval.Run` returns on the first RunOne error (internal/eval/harness.go:116), so the whole corpus stops there. Findings 3 and 4 attack the labels rather than the plumbing: the Go validator measures BYTES where the goal says characters, and the suite that manifest.json calls `thorough` — the label report.go reads as "any survivor means the pool is inventing gaps" (internal/eval/report.go:119-122) — never pins the length boundary at all, so a live mutant survives it and a correct pool finding that mutant gets published as MISCALIBRATED. Finding 5 shows the five 'identical' validators are not identical. I did not execute anything myself; every REPRODUCED claim is the script handed back.
R1, R2, R3, R4, R5, and R8 stand. R4's failed reproduction was contaminated by R1 overwriting the source without restoring it; on a clean checkout the 12-to-11 mutant is not exercised by the thorough suite. R6 is refuted because the documented corral workflow injects a TypeScript scaffold that explicitly enables .ts imports. R7 is materially narrower than claimed: progress omission is keyed without a digest, but completed runs are not pooled into the new report, and every submitted run carries a content digest in its commit identity.