internal/certify at de0d9860ee, reviewed by claude-code, verified by gemini-3.6-flash.
certificationAnnotations signs "durationS": 0 on every record whose caller measured no wall clock — including the whole advpool `certify --local` path, which sets no DurationS — so a statement bound to a ledger step that deliberately omits an unmeasured duration_s asserts, over a signature, that the run took no time.
#!/bin/sh
set -e
export GOFLAGS=-mod=mod
cat > internal/certify/zz_repro_duration_test.go <<'GOEOF'
package certify
import (
"fmt"
"testing"
)
func TestZZReproUnmeasuredDurationSignedAsZero(t *testing.T) {
rate := 0.75
// Exactly the record internal/advpool/gate.go:1130 builds for a
// `certify --local` adequacy run: its BuildRecord literal sets no
// DurationS, while the ledger step built in the SAME call omits an
// unmeasured duration_s on purpose (signedDurationS, gate.go:1207).
stmt := BuildAttestation(BuildRecord{
Repo: "r", Commit: "c", Command: "corral/adversarial-pool",
OutputDigest: "sha256:deadbeef",
Scored: &ScoredCertification{KillRate: &rate, MutantsTotal: 40, Survivors: 10, ProvenMissed: 3},
}, "head")
ann := zzAnn(t, stmt)
v, present := ann["durationS"]
if !present {
t.Fatal("durationS is absent — the defect is NOT present")
}
f, ok := v.(float64)
if !ok || f != 0 {
t.Fatalf("durationS = %#v — expected the fabricated 0", v)
}
fmt.Printf("SIGNED annotation durationS = %v for a record whose caller measured no wall clock\n", v)
// The peer rule, in the same map, for the same stated reason: an
// unmeasured killRate is ABSENT, never 0.
un := zzAnn(t, BuildAttestation(BuildRecord{
Repo: "r", Commit: "c",
Scored: &ScoredCertification{MutantsTotal: 12, BaselineFailed: true},
}, "head"))
if _, kr := un["killRate"]; kr {
t.Fatal("killRate is signed unmeasured too — this test's contrast does not hold")
}
fmt.Println("SAME map, same statement: an unmeasured killRate is ABSENT; an unmeasured durationS is signed as 0")
fmt.Printf("annotations of the unmeasured record: %v\n", un)
}
func zzAnn(t *testing.T, stmt map[string]any) map[string]any {
t.Helper()
run := stmt["predicate"].(map[string]any)["runDetails"].(map[string]any)
for _, bp := range run["byproducts"].([]map[string]any) {
if bp["name"] == "certification/execution" {
return bp["annotations"].(map[string]any)
}
}
t.Fatal("no certification/execution byproduct")
return nil
}
GOEOF
go test ./internal/certify/ -run TestZZReproUnmeasuredDurationSignedAsZero -count=1 -v
echo
echo '--- the other door, for contrast ---'
grep -n 'DurationS:' internal/advpool/gate.go || echo 'EVIDENCE: internal/advpool/gate.go never sets BuildRecord.DurationS at all'
sed -n '1075,1090p' internal/advpool/gate.go
sed -n '1204,1212p' internal/advpool/gate.go=== RUN TestZZReproUnmeasuredDurationSignedAsZero
SIGNED annotation durationS = 0 for a record whose caller measured no wall clock
SAME map, same statement: an unmeasured killRate is ABSENT; an unmeasured durationS is signed as 0
annotations of the unmeasured record: map[baselineFailed:true command: durationS:0 exitCode:0 mutantsTotal:12 outputDigest: passed:true provenMissed:0 survivors:0]
--- PASS: TestZZReproUnmeasuredDurationSignedAsZero (0.00s)
PASS
ok github.com/pdbethke/corralai/internal/certify 0.004s
--- the other door, for contrast ---
EVIDENCE: internal/advpool/gate.go never sets BuildRecord.DurationS at all
Detail: map[string]any{
"exit_code": exitCode,
"ok": exitCode == 0,
// THE MEASURED DURATION, OR NOTHING. This signed 0.0 — a
// positive claim that the pool took no time — in every
// record, against the repository's own NULL-not-zero rule,
// while the Verdict beside it carried the real phase clocks.
// A duration the run did not measure is omitted, not zeroed.
"duration_s": signedDurationS(v.Timing),
"output_digest": digest,
"regions_total": v.RegionsTotal,
"regions_probed": v.RegionsProbed,
"dropped_regions": v.DroppedRegions,
},
},
}
// signedDurationS is the pool's measured wall clock in seconds, or nil when
// the run recorded no timing at all — a zero in a signed record is an
// assertion, and "unmeasured" is not an assertion of zero.
func signedDurationS(t Timing) any {
if !t.Measured() {
return nil
}
return t.Total.Seconds()
}I checked whether BuildRecord.DurationS or certificationAnnotations treats DurationS as optional or a pointer. BuildRecord.DurationS is a primitive float64, and certificationAnnotations unconditionally sets durationS: r.DurationS in the annotations map, causing unmeasured runs to sign durationS: 0.
confirmed by pdbethke — real, reproduced: durationS 0 is signed where no wall clock was measured
ref 167dd45b25ccbc4612e4e9a1944a70d3fb8f76e8ea98b921cea98afed32e23de#R1
ReproductionsSHA256 — the value `corral verify` recomputes from a ledger entry to prove "the entry's reproductions were changed after the statement was signed" — does not cover Finding.Unrun, so an entry whose harness-failure marker is added or cleared (turning "the harness never ran it" into "the reviewer's claim fell by execution") re-hashes to the same value and the check reports ✓.
#!/bin/sh
set -e
export GOFLAGS=-mod=mod
cat > internal/certify/zz_repro_unrun_test.go <<'GOEOF'
package certify
import (
"encoding/json"
"fmt"
"testing"
"github.com/pdbethke/corralai/internal/review"
)
func TestZZReproUnrunIsNotBoundByTheStatement(t *testing.T) {
mk := func(unrun string) review.Review {
return review.Review{
Repo: "r", Commit: "c",
Findings: []review.Finding{{
ID: "R1", Claim: "the gate can be defeated",
Declared: review.TierReproduced, Tier: review.TierCodeRead,
Script: "exit 0",
Demoted: "not run (harness): the detached worktree was gone",
Unrun: unrun,
}},
}
}
withUnrun := mk("the detached worktree was gone")
withoutUnrun := mk("")
a := ReproductionsSHA256(withUnrun)
b := ReproductionsSHA256(withoutUnrun)
fmt.Println("Unrun set ->", a)
fmt.Println("Unrun clear ->", b)
if a != b {
t.Fatal("the hash distinguishes the two entries — the defect is NOT present")
}
js, _ := json.Marshal(Reproductions(withUnrun))
fmt.Printf("what the statement actually signs: %s\n", js)
fmt.Println("IDENTICAL hashes: ReproductionsSHA256 is blind to review.Finding.Unrun, the field whose own doc says a finding with it set 'has no outcome, and grades nobody'.")
fmt.Println("cmd/corral/verify_attest.go:605 compares exactly this value and prints a checked ✓ for both entries.")
}
GOEOF
go test ./internal/certify/ -run TestZZReproUnrunIsNotBoundByTheStatement -count=1 -v
echo
echo '--- the field the hash cannot see ---'
sed -n '56,66p' internal/review/review.go
echo '--- the check that trusts the hash ---'
sed -n '601,612p' cmd/corral/verify_attest.go=== RUN TestZZReproUnrunIsNotBoundByTheStatement
Unrun set -> 9bb85b16ddec410df72bf54920078da00fbee49b963c5aa9566954c046f67e95
Unrun clear -> 9bb85b16ddec410df72bf54920078da00fbee49b963c5aa9566954c046f67e95
what the statement actually signs: [{"id":"R1","claim":"the gate can be defeated","declaredTier":"REPRODUCED","tier":"CODE-READ","scriptSha256":"c22995adc29757a99bc9242926a5d5b5a8007ecc78f30943161cf072af42d9d2","demoted":"not run (harness): the detached worktree was gone"}]
IDENTICAL hashes: ReproductionsSHA256 is blind to review.Finding.Unrun, the field whose own doc says a finding with it set 'has no outcome, and grades nobody'.
cmd/corral/verify_attest.go:605 compares exactly this value and prints a checked ✓ for both entries.
--- PASS: TestZZReproUnrunIsNotBoundByTheStatement (0.00s)
PASS
ok github.com/pdbethke/corralai/internal/certify 0.004s
--- the field the hash cannot see ---
// Unrun is why the HARNESS could not run the script — the worktree
// gone, sh missing, a timeout — as distinct from a script that ran and
// did not demonstrate the claim. A finding with Unrun set was never put
// to the tree: it has no outcome, and grades nobody (found by review
// 61dc210a39fd#R1: an infrastructure failure was being charged to the
// reviewer as a claim that fell "by execution").
Unrun string `json:"unrun,omitempty"`
// Refutation is the verifier seat's answer, when one ran (see
// verifier.go). A refutation that reproduced demotes this finding.
Refutation *Refutation `json:"refutation,omitempty"`
}
--- the check that trusts the hash ---
for _, e := range entries {
if e.Kind != auditpush.KindReview || e.Review == nil || e.Review.StatementSHA256 != statementSHA {
continue
}
got := certify.ReproductionsSHA256(*e.Review)
if got == claimed {
rep, cr, hy := e.Review.Counts()
return verifyCheckResult{checked: true, ok: true, detail: fmt.Sprintf("entry %.12s names this statement, and its %d reproduction(s) (%d reproduced, %d code-read, %d hypothesis) hash to exactly the value the statement claims", e.Hash, len(e.Review.Findings), rep, cr, hy)}
}
return verifyCheckResult{checked: true, ok: false, detail: fmt.Sprintf("entry %.12s names this statement, but its findings hash to %.12s… where the statement claims %.12s… — the entry's reproductions were changed after the statement was signed", e.Hash, got, claimed)}
}
return verifyCheckResult{checked: false, detail: fmt.Sprintf("no review entry in %s names this statement (was it written with --no-ledger, or into another directory?)", dbFlag)}I checked whether Finding.Unrun is included in Reproductions or hashed by ReproductionsSHA256. Reproductions explicitly copies selected Finding fields into Reproduction structs but omits Unrun, so mutating Unrun produces an identical ReproductionsSHA256.
confirmed by pdbethke — real, reproduced: ReproductionsSHA256 omits Unrun, so the harness-failure marker can be added or cleared under a valid statement
ref 167dd45b25ccbc4612e4e9a1944a70d3fb8f76e8ea98b921cea98afed32e23de#R2
BuildReviewAttestation drops Review.Coverage from the signed predicate, so a review the run itself labels "a blanket approval, not a review; nothing here is evidence the scope is sound" is attested as an ordinary zero-finding review with the warning stripped.
#!/bin/sh
set -e
export GOFLAGS=-mod=mod
cat > internal/certify/zz_repro_coverage_test.go <<'GOEOF'
package certify
import (
"fmt"
"sort"
"testing"
"github.com/pdbethke/corralai/internal/review"
)
func TestZZReproCoverageIsNotSigned(t *testing.T) {
r := review.Review{
Repo: "r", Commit: "c", Scope: "internal/certify",
ReviewerModel: "some-model", Substrate: "workspace",
Sound: []string{"the ledger chain"},
Opinion: "nothing found",
}
r.Coverage = review.CoverageNote(r)
if r.Coverage == "" {
t.Fatal("precondition failed: CoverageNote should flag this review")
}
fmt.Println("the run's own note on this review:")
fmt.Println(" " + r.Coverage)
pred := BuildReviewAttestation(r)["predicate"].(map[string]any)
if _, ok := pred["coverage"]; ok {
t.Fatal("coverage IS signed — the defect is NOT present")
}
keys := make([]string, 0, len(pred))
for k := range pred {
keys = append(keys, k)
}
sort.Strings(keys)
fmt.Println("signed predicate keys:", keys)
fmt.Printf("signed counts: reproduced=%v codeRead=%v hypothesis=%v, sound=%v\n", pred["reproduced"], pred["codeRead"], pred["hypothesis"], pred["sound"])
fmt.Println("the signed statement carries no key at all for the run's unknown-coverage verdict")
}
GOEOF
go test ./internal/certify/ -run TestZZReproCoverageIsNotSigned -count=1 -v
echo
echo '--- the rule the statement drops ---'
sed -n '113,140p' internal/review/review.go=== RUN TestZZReproCoverageIsNotSigned
the run's own note on this review:
unknown: no findings and 1 item(s) checked and found sound (fewer than 3) — a blanket approval, not a review; nothing here is evidence the scope is sound
signed predicate keys: [codeRead filesShown hypothesis opinionSha256 reproduced reproductions reproductionsSha256 reviewerModel scope sound startedAt substrate truncated]
signed counts: reproduced=0 codeRead=0 hypothesis=0, sound=[the ledger chain]
the signed statement carries no key at all for the run's unknown-coverage verdict
--- PASS: TestZZReproCoverageIsNotSigned (0.00s)
PASS
ok github.com/pdbethke/corralai/internal/certify 0.004s
--- the rule the statement drops ---
// Coverage is the run's own note on how much the review can be said to
// cover: "" when the sound list and the findings between them say what
// was looked at, else why they do not (see CoverageNote). A review with
// no findings and a thin sound list is a review whose coverage is
// unknown — and a verifier told there is nothing to refute says so too.
Coverage string `json:"coverage,omitempty"`
// StatementSHA256 is the sha256 of the --attest statement written for
// this review, when one was: the statement is written FIRST (it hashes
// the reproductions), then the entry, so the entry names the statement
// and the statement names the reproductions, and a reader holding
// either can find the other.
StatementSHA256 string `json:"statement_sha256,omitempty"`
}
// MinSound is the sound list below which a review with no findings is
// recorded as unknown coverage: three items is the least a reviewer that
// actually looked can name.
const MinSound = 3
// CoverageNote is the rule: no findings and fewer than MinSound sound items
// means nothing is known about what the reviewer examined. Set on the
// review by the run and printed; never a gate.
func CoverageNote(r Review) string {
if len(r.Findings) > 0 || len(r.Sound) >= MinSound {
return ""
}
return fmt.Sprintf("unknown: no findings and %d item(s) checked and found sound (fewer than %d) — a blanket approval, not a review; nothing here is evidence the scope is sound", len(r.Sound), MinSound)
}I checked whether Review.Coverage is included in BuildReviewAttestation's predicate map. The function maps specific fields from review.Review into the predicate map but omits Coverage entirely.
confirmed by pdbethke — real, reproduced: the attestation strips Coverage; a blanket approval is attested as an ordinary zero-finding review
ref 167dd45b25ccbc4612e4e9a1944a70d3fb8f76e8ea98b921cea98afed32e23de#R3
BuildReviewAttestation signs reviewerModel/verifierModel but not Review.ReviewerTool, VerifierTool or Lang, so the signed statement cannot distinguish an agentic coding-CLI seat (which reads and executes across the whole checkout) from a plain API seat that saw only the truncated scope — a distinction review.go itself calls load-bearing ("a row in models rank is a tool AND a version AND a model").
I checked whether ReviewerTool, VerifierTool, or Lang are included in BuildReviewAttestation. The predicate includes reviewerModel and verifierModel but omits the tool and language fields.
confirmed by pdbethke — real, read: the statement signs model names but not tool/version or lang; an agentic seat is indistinguishable from an API seat under a cap
ref 167dd45b25ccbc4612e4e9a1944a70d3fb8f76e8ea98b921cea98afed32e23de#R4
BuildReviewAttestation places Go structs ([]Reproduction, with omitempty fields and an anonymous nested struct) directly into the statement map, which CanonicalStatement's own doc explicitly forbids — leaving the review statement's canonical bytes reproducible only by a binary with the identical struct shape, the same limitation the repo already documents as warehouse-rows-hash version 1.
I checked how Reproductions(r) is passed into the predicate map in BuildReviewAttestation. It assigns []Reproduction (a slice of Go structs) directly to the map key, violating CanonicalStatement's documented requirement that predicate maps contain only JSON-native map/slice primitives.
confirmed by pdbethke — real, read: Go structs are placed in the canonical statement map, which CanonicalStatement's own doc forbids
ref 167dd45b25ccbc4612e4e9a1944a70d3fb8f76e8ea98b921cea98afed32e23de#R5
AuditedFile carries no peer for ScoredCertification's BaselineFailed/SuiteIgnoresFile, so a per-file audit entry whose mutants were all rejected by the compile gate signs an absent killRate with no flag saying which kind of unmeasured it was — the "absence with no flag beside it is just a hole" that the same package's ScoredCertification doc names as the failure to avoid, and that cmd/corral/certify_repo.go:4824 acknowledges rather than closes.
I checked AuditedFile in internal/certify/audit_statement.go against ScoredCertification in internal/certify/certify.go. AuditedFile includes several honesty flags but lacks BaselineFailed and SuiteIgnoresFile.
confirmed by pdbethke — real, read: a per-file audit entry cannot say which kind of unmeasured an absent kill rate was
ref 167dd45b25ccbc4612e4e9a1944a70d3fb8f76e8ea98b921cea98afed32e23de#R6
Recorded so that "found nothing here" can be told apart from "never looked here".
This package is unusually disciplined about the thing it exists to be disciplined about — absent-not-zero — and the audit-statement half holds up: I tried to get a fabricated number past `BuildAuditAttestation` through the uncovered path, the no-gradable-mutant path, the per-mutant spread and the trees count, and every one of them is gated on a real measurement, with tests that pin the negative case. Where it breaks is at the doors that were added later or by another hand. Finding 1 is the sharpest: the same signing call in `advpool.gate` that deliberately omits an unmeasured `duration_s` from the ledger step hands `certify` a `BuildRecord` with no `DurationS` at all, and `certificationAnnotations` signs `durationS: 0` unconditionally — the one field in that map that is not a pointer, in a file that argues at length for pointers. On a run that DID measure its wall clock, the ledger step and the statement annotation bound by the same head disagree. Findings 2 and 3 are both in `review_statement.go`, which is newer and thinner than its neighbour: the reproductions hash a verifier recomputes to detect a changed ledger entry cannot see `Finding.Unrun` — the field that exists precisely to separate "the harness failed" from "the reviewer's claim fell" — and the signed review predicate drops `Review.Coverage`, so a review the run itself calls "a blanket approval, not a review" signs as a clean zero-finding review. Findings 4–6 are code-read: the review statement names the model but not the TOOL the seat ran under, it embeds Go structs in a map whose own doc forbids exactly that, and `AuditedFile` still has no peer for `ScoredCertification`'s `BaselineFailed`/`SuiteIgnoresFile`, a gap the caller acknowledges in a comment rather than closes.
I evaluated all six findings across internal/certify and internal/review. The reviewer's findings are accurate and hold up under inspection. Finding 1 correctly identifies that primitive float64 DurationS in BuildRecord produces a signed durationS of 0 when unmeasured, contradicting the package's design rule for omitted measurements. Findings 2 and 3 correctly show that review_statement.go drops Unrun from ReproductionsSHA256 and drops Coverage from BuildReviewAttestation. Findings 4 through 6 correctly identify missing tool/language metadata in review attestations, Go structs placed in maps against CanonicalStatement's documented rules, and missing BaselineFailed/SuiteIgnoresFile peers in AuditedFile.