← the record

Review 6c87da8a5a04

scripts at 441da802c8, reviewed by claude-code, verified by codex.

repo
git@github.com:pdbethke/corralai
commit
441da802c85aa4c6fd072d4367b4a276dfa1954d
scope
scripts · python
reviewer
claude-code (2.1.263 (Claude Code))
verifier
codex (codex-cli 0.152.0) different model than the reviewer different vendor: anthropic → openai
substrate
workspace (a detached worktree at the commit; not a jail)
started
2026-09-08 18:59 UTC
audited party
Peter D Bethke · co-authored: Claude Opus 5 (1M context)
files shown
21 file(s)

Findings (9)

R1 medium reproduced

build-warehouse-parquet.py's operator-facing withheld count still feeds export()'s -1 refusal sentinel into `total - kept`, so a refused export prints "-1 rows exported, N+1 withheld" — a privacy-accounting line describing rows that were never withheld — even though the fix directly above it guards the manifest value against exactly that sentinel.

where
scripts/build-warehouse-parquet.py:117
tier
REPRODUCED (declared REPRODUCED)
the script that was executed
#!/bin/sh
set -e
tmp=$(mktemp -d)
fake="$tmp/fake"
out="$tmp/out"
mkdir -p "$fake" "$out"

# A stand-in duckdb module: every extract query returns 0 rows (the documented
# "source store is empty on this machine" case), build.build_records has 7 rows,
# bugcatch_observations has 11.
cat > "$fake/duckdb.py" <<'PY'
class Error(Exception):
    pass


class _Res:
    def __init__(self, v):
        self._v = v

    def fetchone(self):
        return (self._v,)


class _Con:
    def execute(self, sql):
        s = " ".join(sql.split())
        if s.startswith("SELECT count(*) FROM ("):
            return _Res(0)
        if "build.build_records" in s:
            return _Res(7)
        if "bugcatch.bugcatch_observations" in s:
            return _Res(11)
        return _Res(0)

    def create_function(self, *a, **k):
        return None


def connect(*a, **k):
    return _Con()
PY

# Populated extracts already exist -> export() must REFUSE and return -1.
for t in audit_ledger bug_catches scans scan_files; do
  printf 'already-published-data' > "$out/$t.parquet"
done

PYTHONPATH="$fake" python3 scripts/build-warehouse-parquet.py --out "$out" > "$tmp/log.txt" 2>&1 || true

echo "=== scripts/build-warehouse-parquet.py output ==="
cat "$tmp/log.txt"
echo "=== end output ==="
echo

line=$(grep '^audit_ledger:' "$tmp/log.txt" 2>/dev/null || true)
if [ -z "$line" ]; then echo "no audit_ledger line produced"; exit 1; fi
line2=$(grep '^bug_catches:' "$tmp/log.txt" 2>/dev/null || true)

echo "FACTS OF THIS RUN:"
echo "  build.build_records          has 7 rows; export() REFUSED (returned -1); 0 rows withheld"
echo "  bugcatch_observations        has 11 rows; export() REFUSED (returned -1); 0 rows withheld"
echo
echo "WHAT THE SCRIPT TOLD THE OPERATOR:"
echo "  $line"
echo "  $line2"
echo

fail=0
echo "$line" | grep -q -- '-1 rows exported' || { echo "MISS: expected the -1 sentinel to leak into the line"; fail=1; }
echo "$line" | grep -q '8 withheld'  || { echo "MISS: expected the wrong '8 withheld' (7 + 1)"; fail=1; }
echo "$line2" | grep -q '12 withheld' || { echo "MISS: expected the wrong '12 withheld' (11 + 1)"; fail=1; }
[ "$fail" -eq 0 ] || exit 1

echo "DEFECT DEMONSTRATED:"
echo "  scripts/build-warehouse-parquet.py:116 guards the MANIFEST value against the"
echo "  -1 sentinel ('(total - kept) if kept >= 0 else 0'), and the comment above it"
echo "  says the sentinel 'printed total+1 withheld -- a number describing nothing"
echo "  that happened. Found by a cold review, 2026-09-08 (R2).'"
echo "  Line 117's print() still computes 'total - kept' unguarded. So does line 151."
echo "  The manifest is now right and the thing a human reads is still wrong."
exit 0
what it printed
=== scripts/build-warehouse-parquet.py output ===
REFUSED to overwrite /tmp/tmp.qW6D3saV92/out/audit_ledger.parquet: the query returned 0 rows and a populated extract already exists (source store is empty on this machine)
audit_ledger:  -1 rows exported, 8 withheld (repo not on the public allowlist)
REFUSED to overwrite /tmp/tmp.qW6D3saV92/out/bug_catches.parquet: the query returned 0 rows and a populated extract already exists (source store is empty on this machine)
bug_catches:   -1 rows exported, 12 withheld (repo not on the public allowlist)
REFUSED to overwrite /tmp/tmp.qW6D3saV92/out/scans.parquet: the query returned 0 rows and a populated extract already exists (source store is empty on this machine)
REFUSED to overwrite /tmp/tmp.qW6D3saV92/out/scan_files.parquet: the query returned 0 rows and a populated extract already exists (source store is empty on this machine)

wrote to /tmp/tmp.qW6D3saV92/out/ and site/src/data/warehouse-manifest.json — rebuild the site to pick them up
=== end output ===

FACTS OF THIS RUN:
  build.build_records          has 7 rows; export() REFUSED (returned -1); 0 rows withheld
  bugcatch_observations        has 11 rows; export() REFUSED (returned -1); 0 rows withheld

WHAT THE SCRIPT TOLD THE OPERATOR:
  audit_ledger:  -1 rows exported, 8 withheld (repo not on the public allowlist)
  bug_catches:   -1 rows exported, 12 withheld (repo not on the public allowlist)

DEFECT DEMONSTRATED:
  scripts/build-warehouse-parquet.py:116 guards the MANIFEST value against the
  -1 sentinel ('(total - kept) if kept >= 0 else 0'), and the comment above it
  says the sentinel 'printed total+1 withheld -- a number describing nothing
  that happened. Found by a cold review, 2026-09-08 (R2).'
  Line 117's print() still computes 'total - kept' unguarded. So does line 151.
  The manifest is now right and the thing a human reads is still wrong.

refutation — codex STANDS

The reproduced output matches the code: export() returns -1 on refusal, while both operator-facing print calls compute total-kept without the kept>=0 guard used for the manifest. I found no later correction of those printed values.

ref 6c87da8a5a04ecca1ba65c23ae81d6ac0fea7c2a1f8b19f8fd1467fe0a6a2fe9#R1

R2 medium not shown (exit 1) demoted

check-licensing.sh derives its entire file list from `git ls-files`, whose failure is indistinguishable from an empty repository, so with git unavailable the SPDX-header invariant reports "OK: all licensing invariants hold" and exits 0 over a tracked .go file that carries no header at all.

where
scripts/check-licensing.sh:20
tier
CODE-READ (declared REPRODUCED)
demoted
the script exited 1 — it did not demonstrate the claim
the script that was executed
#!/bin/sh
set -e

TARGET=cmd/corral-admin/client.go

# Remove the SPDX header from a TRACKED Go file (not a new untracked one, so the
# only variable under test is whether git can enumerate it).
python3 - "$TARGET" <<'PY'
import sys
p = sys.argv[1]
s = open(p, encoding='utf-8').read()
assert s.startswith('// SPDX-License-Identifier: Elastic-2.0\n'), 'fixture moved: ' + p
open(p, 'w', encoding='utf-8').write(s.split('\n', 2)[2])
print('stripped the SPDX header from', p)
PY

echo "=== first 3 lines of $TARGET now ==="
head -3 "$TARGET"
echo "=== (no SPDX-License-Identifier anywhere) ==="
grep -c 'SPDX-License-Identifier' "$TARGET" || true
echo

# CONTROL: with a working git, the gate is supposed to catch this.
echo "=== CONTROL: scripts/check-licensing.sh with git as-is ==="
ctl=$(bash scripts/check-licensing.sh 2>&1; echo "__rc=$?")
printf '%s\n' "$ctl"
echo

# Now make git unavailable, exactly as it would be on a runner/image without it
# or in an exported tarball.
stub=$(mktemp -d)
cat > "$stub/git" <<'EOF'
#!/bin/sh
echo "git: not available (stub)" >&2
exit 127
EOF
chmod +x "$stub/git"

echo "=== SAME TREE, git unavailable ==="
set +e
out=$(PATH="$stub:$PATH" bash scripts/check-licensing.sh 2>&1)
rc=$?
set -e
printf '%s\n' "$out"
echo "exit code: $rc"
echo

fail=0
printf '%s' "$out" | grep -q 'OK: all licensing invariants hold' || { echo "MISS: gate did not report OK"; fail=1; }
[ "$rc" -eq 0 ] || { echo "MISS: exit code was $rc, expected 0"; fail=1; }
printf '%s' "$out" | grep -q 'missing SPDX header' && { echo "MISS: gate did flag the file"; fail=1; }
[ "$fail" -eq 0 ] || exit 1

echo "DEFECT DEMONSTRATED:"
echo "  scripts/check-licensing.sh:20 enumerates the files to check with 'git ls-files'."
echo "  git exiting non-zero yields an EMPTY list; \$missing is then empty and the"
echo "  header invariant passes. The script runs under 'set -uo pipefail' with no -e"
echo "  and never inspects git's exit status, so there is no other signal."
echo "  This is the same defect the sibling gate names at scripts/check-security.sh:16 --"
echo "  \"The tool's ABSENCE used to read as clean ... A gate that cannot fail is not a"
echo "  gate\" -- fixed there for gofmt, and left in place for git, which is the ONLY"
echo "  thing deciding what either gate looks at."
exit 0
what it printed
stripped the SPDX header from cmd/corral-admin/client.go
=== first 3 lines of cmd/corral-admin/client.go now ===
package main

import (
=== (no SPDX-License-Identifier anywhere) ===
0

=== CONTROL: scripts/check-licensing.sh with git as-is ===

refutation — codex STANDS

The submitted script failed prematurely because set -e treats the deliberately failing control command substitution as fatal, but that does not refute the finding. check-licensing.sh lacks set -e and never checks git's status; failure of git ls-files leaves the while loop with no input, an empty missing value, and allows the final success message when the unrelated invariants hold.

ref 6c87da8a5a04ecca1ba65c23ae81d6ac0fea7c2a1f8b19f8fd1467fe0a6a2fe9#R2

R3 medium no script

export-golden-run.sh writes the mission `directive` (free-form operator text) into the published .meta.json sidecar without it ever passing the deny-list scan or the human-review manifest, and it is fetched after the operator's confirmation prompt, so the "not optional, not an assumption" privacy gate covers only the replay stream and not half of what is published beside it.

where
scripts/export-golden-run.sh:243
tier
CODE-READ (declared CODE-READ)

refutation — codex STANDS

The deny scan and manifest inspect only TMP_JSON. After confirmation, the script performs a fresh /api/history request, extracts directive from that response, and places it directly in META_JSON. I found no scan, manifest display, or confirmation covering this newly fetched directive.

ref 6c87da8a5a04ecca1ba65c23ae81d6ac0fea7c2a1f8b19f8fd1467fe0a6a2fe9#R3

R4 low no script

gen-cost-table.py's cost_line, declared a byte-for-byte replica of cmd/corral/cost_line.go's costLine and used to publish "the exact cost: lines corral scans show --timing prints", omits the Go version's " (N cached)" clause entirely, so any fixture whose model_calls carry a non-null cached_input_tokens generates a documented CLI line the CLI never prints.

where
scripts/gen-cost-table.py:167
tier
CODE-READ (declared CODE-READ)

refutation — codex STANDS

The Go formatter collects CachedInputTokens and appends a parenthesized cached total whenever at least one provider reported the measurement. The Python replica neither reads cached_input_tokens nor emits that clause, so its byte-for-byte claim fails for the stated input.

ref 6c87da8a5a04ecca1ba65c23ae81d6ac0fea7c2a1f8b19f8fd1467fe0a6a2fe9#R4

R5 low no script

add-spdx.sh skips any file containing the string "SPDX-License-Identifier" anywhere, while check-licensing.sh only accepts it as a comment inside the first five lines, so a file carrying the identifier in a string literal or below line 5 is flagged forever by the checker and never fixed by the fixer — the fixer/checker disagreement its own comment says must not exist.

where
scripts/add-spdx.sh:13
tier
CODE-READ (declared CODE-READ)

refutation — codex STANDS

The fixer uses an unanchored whole-file grep for SPDX-License-Identifier, whereas the checker requires an appropriately commented Elastic-2.0 identifier within the first five lines. Thus the named malformed placements are rejected by the checker but deliberately skipped by the fixer.

ref 6c87da8a5a04ecca1ba65c23ae81d6ac0fea7c2a1f8b19f8fd1467fe0a6a2fe9#R5

R6 low no script

check-security.sh gates only gofmt's PRESENCE, not its success: `bad=$(git ls-files '*.go' | xargs gofmt -l)` discards the exit status, so gofmt failing (an unparseable file, a path xargs split on whitespace, a missing file) produces empty output and the formatting gate reports clean.

where
scripts/check-security.sh:18
tier
CODE-READ (declared CODE-READ)

refutation — codex STANDS

The gofmt command substitution captures stdout but its nonzero status is not tested, and the script intentionally lacks set -e. Consequently an empty-output gofmt failure leaves bad empty and does not call note. Later gates might independently fail in a particular environment, but the formatting gate itself fails open as claimed.

ref 6c87da8a5a04ecca1ba65c23ae81d6ac0fea7c2a1f8b19f8fd1467fe0a6a2fe9#R6

R7 low no script

export-golden-run.sh interpolates $MISSION_ID and $MODELS_JSON unquoted into Python source passed to `python3 -c`, so a mission id or a model label originating from the brain's own JSON that is not a bare integer / contains a quote or backslash either breaks the export or executes as Python.

where
scripts/export-golden-run.sh:246
tier
CODE-READ (declared CODE-READ)

refutation — codex STANDS

MISSION_ID is inserted as raw Python syntax and MODELS_JSON is embedded inside a triple-single-quoted Python literal. Neither value is passed through argv or another non-code channel, and no validation excludes Python-significant content. The JSON parser guarantees valid JSON, not safe interpolation into Python source.

ref 6c87da8a5a04ecca1ba65c23ae81d6ac0fea7c2a1f8b19f8fd1467fe0a6a2fe9#R7

R8 low no script

sign-console-bundle.sh's header documents its output as a signature with "no trailing newline (matches the exact bytes GET /console/manifest.sig serves)", but cmd/sign-console-bundle's upsert returns `strings.Join(out, "\n") + "\n"`, so the committed file has always ended with a newline and the documented byte-equality claim is false.

where
scripts/sign-console-bundle.sh:41
tier
CODE-READ (declared CODE-READ)

refutation — codex STANDS

The shell header says the output file has no trailing newline, but cmd/sign-console-bundle's upsert unconditionally returns strings.Join(out, "\n")+"\n". The runtime trims the embedded file before serving it, so the served response can lack the newline, but that does not make the documented output-file claim true.

ref 6c87da8a5a04ecca1ba65c23ae81d6ac0fea7c2a1f8b19f8fd1467fe0a6a2fe9#R8

R9 low no script

capture-og-image.sh's post-launch safeguard checks `kill -0 $BRAIN_PID` rather than that our child owns the listening socket, so if the scratch brain died on bind failure and has not yet been reaped, the liveness probe succeeds and the capture shoots whichever brain actually owns port 9019.

where
scripts/capture-og-image.sh:145
tier
HYPOTHESIS (declared HYPOTHESIS)

refutation — codex STANDS

I found an up-front health check and a post-launch PID liveness check, but no verification that BRAIN_PID owns the listening socket. kill -0 establishes only process existence; it does not establish socket ownership, so it does not logically exclude the stated bind-failure/incumbent race. I could not refute the hypothesis from the code.

ref 6c87da8a5a04ecca1ba65c23ae81d6ac0fea7c2a1f8b19f8fd1467fe0a6a2fe9#R9

Looked at and could not break (15)

Recorded so that "found nothing here" can be told apart from "never looked here".

Reviewer's opinion

This is a scope of gates and generators, and the recurring shape is a fix applied at one door and not the next. Finding 1 is the sharpest example: a cold review on 2026-09-08 caught `total - kept` being fed the -1 refusal sentinel, wrote the correction into the manifest assignment, wrote a comment saying so, and left the `print()` on the very next line still computing `total - kept` — so the operator-facing privacy accounting of `build-warehouse-parquet.py` still reports `-1 rows exported, 8 withheld` when the source has 7 rows and nothing was withheld. Finding 2 is the same shape across files: `check-security.sh` grew an explicit "the tool's ABSENCE used to read as clean ... a gate that cannot fail is not a gate" fix for gofmt at this commit, while the file-list for every one of these gates still comes from `git ls-files`, whose failure is silently an empty list — with git unavailable, `check-licensing.sh` prints "OK: all licensing invariants hold" over a tracked .go file whose SPDX header I removed. Finding 6 is the third instance, inside the very gate that was just hardened: gofmt's exit status is discarded, so only its *presence* is gated, not its *success*. Finding 3 is the most consequential unreproduced one — `export-golden-run.sh` calls its deny-list scan and human manifest a non-optional privacy floor, then fetches the mission `directive` from a second `/api/history` call AFTER the human has already confirmed, and writes it verbatim into the published `.meta.json` without either check ever seeing it. Findings 4, 5 and 8 are all claim-vs-code: a Python "byte-for-byte replica" that dropped a branch of the Go it mirrors, a fixer and its checker that disagree about what a header is, and a header comment describing output bytes the tool no longer produces. The privacy regexes in `scrub-golden-run.py`, the golden-file discipline in `foreign-sweep.sh`, and the console trust anchor's refusal to default to the committed dev key all held up under pressure.

Verifier's opinion

The review withstands adversarial verification. R1 is directly demonstrated. R2's supplied reproducer aborts during its control phase, but code inspection still supports the underlying failure-open claim. R3–R8 accurately identify mismatches between stated guarantees and the implemented data flow or output. R9 remains a plausible race; I found no socket-ownership verification that refutes it.