← the record

Review 61ecf250c4b4

internal/adequacy at 954c320247, reviewed by codex:gpt-6-astra.

repo
git@github.com:pdbethke/corralai
commit
954c3202475865682e78d9f2d4c65bd2739e757c
scope
internal/adequacy · go
reviewer
codex:gpt-6-astra (codex-cli 0.156.1)
verifier
no verifier seat ran
substrate
workspace (a detached worktree at the commit; not a jail)
started
2026-09-24 00:14 UTC
audited party
Peter D Bethke · co-authored: Claude Opus 5.5 (1M context)
files shown
46 file(s)

Findings (2)

R1 medium reproduced confirmed by a person

Score deduplicates identical edits without considering WithCommandFor, so two identical mutants assigned different commands inherit one verdict even when their commands demonstrably disagree.

where
internal/adequacy/score.go:1001
tier
REPRODUCED (declared REPRODUCED)
the script that was executed
#!/bin/sh
set -eu
d=$(mktemp -d ./review-dedupe-XXXXXX)
trap 'rm -rf "$d"' EXIT HUP INT TERM
cat > "$d/main.go" <<'GO'
package main
import (
 "context"
 "fmt"
 "os"
 "strings"
 "time"
 a "github.com/pdbethke/corralai/internal/adequacy"
)
func must(err error) { if err != nil { panic(err) } }
func main() {
 root, err := os.MkdirTemp(".", "review-suite-"); must(err)
 defer os.RemoveAll(root)
 j := a.NewWorkspaceRunner(root, time.Second)
 strict := []string{"sh", "-c", "set -e; . ./value.sh; test \"$v\" -eq 1"}
 loose := []string{"sh", "-c", "set -e; . ./value.sh; test \"$v\" -ge 1"}
 mutants := []a.Mutant{
  {ID:"strict", Search:"v=1", Replace:"v=2"},
  {ID:"loose", Search:"v=1", Replace:"v=2"},
 }
 selectCmd := a.WithCommandFor(func(m a.Mutant) a.MutantCommand {
  if m.ID == "loose" { return a.MutantCommand{Cmd:loose, Tests:1} }
  return a.MutantCommand{Cmd:strict, Tests:1}
 })
 control, err := a.Score(context.Background(), j, nil, "value.sh", "v=1\n", mutants[1:], strict, selectCmd); must(err)
 combined, err := a.Score(context.Background(), j, nil, "value.sh", "v=1\n", mutants, strict, selectCmd); must(err)
 fmt.Printf("loose mutant alone: killed=%v survived=%v\n", control.Killed, control.Survived)
 fmt.Printf("same mutant beside identical strict mutant: killed=%v survived=%v duplicates=%d\n", combined.Killed, combined.Survived, combined.DuplicateMutants)
 if !control.CompliantPass || !control.CanaryKilled || strings.Join(control.Survived, ",") != "loose" || len(control.Killed) != 0 { os.Exit(1) }
 if !combined.CompliantPass || !combined.CanaryKilled || strings.Join(combined.Killed, ",") != "strict,loose" || len(combined.Survived) != 0 || combined.DuplicateMutants != 1 { os.Exit(1) }
 fmt.Println("DEFECT: the selected command survives this mutant, but deduplication reports it killed.")
}
GO
GOPROXY=off GOSUMDB=off GOTOOLCHAIN=local timeout 55s go run "$d"
what it printed
loose mutant alone: killed=[] survived=[loose]
same mutant beside identical strict mutant: killed=[strict loose] survived=[] duplicates=1
DEFECT: the selected command survives this mutant, but deduplication reports it killed.

human verdict

confirmed by pdbethke — real, reproduced: Score's dedupe ignored WithCommandFor, so a mutant that survived its own command inherited its twin's kill. The reviewer's script exited 0. Fixed in #346 (dedupe key is edit + grading command); TestScoreDoesNotShareAVerdictAcrossDifferentGradingCommands failed on main first.

ref 61ecf250c4b4e6265f13ac54d948522687773d29e03389e3b10177efa07893f2#R1

R2 medium reproduced confirmed by a person

Applying an overlay through an existing dangling symlink treats the symlink as nonexistent, then restoration deletes the symlink while leaving the newly created target containing mutant bytes.

where
internal/adequacy/workspace.go:292
tier
REPRODUCED (declared REPRODUCED)
the script that was executed
#!/bin/sh
set -eu
d=$(mktemp -d ./review-symlink-XXXXXX)
trap 'rm -rf "$d"' EXIT HUP INT TERM
cat > "$d/main.go" <<'GO'
package main
import (
 "context"
 "fmt"
 "os"
 "path/filepath"
 "time"
 a "github.com/pdbethke/corralai/internal/adequacy"
)
func must(err error) { if err != nil { panic(err) } }
func main() {
 root, err := os.MkdirTemp(".", "review-workspace-"); must(err)
 defer os.RemoveAll(root)
 link := filepath.Join(root, "source.txt")
 target := filepath.Join(root, "target.txt")
 must(os.Symlink("target.txt", link))
 before, err := os.Readlink(link); must(err)
 if _, err := os.Lstat(target); !os.IsNotExist(err) { panic("target must initially be absent") }
 fmt.Printf("before: source.txt -> %s; target.txt absent\n", before)
 j := a.NewWorkspaceRunner(root, time.Second)
 passed, runErr := j.RunTest(context.Background(), map[string]string{"source.txt":"MUTANT\n"}, []string{"sh", "-c", "test \"$(cat source.txt)\" = MUTANT"})
 _, linkErr := os.Lstat(link)
 content, targetErr := os.ReadFile(target)
 fmt.Printf("after: passed=%v error=%v symlinkAbsent=%v target=%q targetError=%v\n", passed, runErr, os.IsNotExist(linkErr), content, targetErr)
 if !passed || runErr != nil || !os.IsNotExist(linkErr) || targetErr != nil || string(content) != "MUTANT\n" { os.Exit(1) }
 fmt.Println("DEFECT: a read-only test command returned successfully, but restoration deleted the original symlink and retained mutant bytes.")
}
GO
GOPROXY=off GOSUMDB=off GOTOOLCHAIN=local timeout 55s go run "$d"
what it printed
before: source.txt -> target.txt; target.txt absent
after: passed=true error=<nil> symlinkAbsent=true target="MUTANT\n" targetError=<nil>
DEFECT: a read-only test command returned successfully, but restoration deleted the original symlink and retained mutant bytes.

human verdict

confirmed by pdbethke — real, reproduced: overlaying through a dangling symlink created the target and restore deleted the link, on pass=true, err=nil. The reviewer's script exited 0. Fixed in #346 (applyFiles refuses a dangling symlink before writing); TestWorkspaceRunnerRefusesToOverlayThroughADanglingSymlink failed on main first.

ref 61ecf250c4b4e6265f13ac54d948522687773d29e03389e3b10177efa07893f2#R2

Looked at and could not break (8)

Recorded so that "found nothing here" can be told apart from "never looked here".

Reviewer's opinion

Finding 1 shows that duplicate elimination can override the verdict of a mutant's selected grading command. Finding 2 shows that workspace restoration can delete an existing symlink and leave mutant content behind even when the command changes nothing. Both include executable reproduction scripts; I did not execute them, run tests, or modify this checkout. Coverage below is code inspection only.