internal/review at d56a788529, reviewed by gemini-3.6-flash, verified by claude-haiku-4-5.
extractJSON assumes the first opening brace '{' in LLM output begins the JSON payload, causing Parse and ParseRefutations to fail whenever prose or code blocks contain braces before the JSON object.
cat << 'EOF' > internal/review/repro_test.go
package review
import (
"testing"
)
func TestParseWithPrecedingBrace(t *testing.T) {
text := `I reviewed the scope {internal/review}.
{"opinion": "code looks okay", "findings": [], "sound": ["review.go"]}`
opinion, findings, sound, err := Parse(text)
if err != nil {
t.Fatalf("Parse failed unexpectedly: %v", err)
}
if opinion != "code looks okay" {
t.Fatalf("got opinion %q, want %q", opinion, "code looks okay")
}
if len(sound) != 1 {
t.Fatalf("got sound %v, want 1 item", sound)
}
_ = findings
}
EOF
output=$(go test ./internal/review -run TestParseWithPrecedingBrace 2>&1)
res=$?
rm -f internal/review/repro_test.go
if [ $res -ne 0 ]; then
echo "Defect demonstrated: Parse failed on text with preceding braces:"
echo "$output"
exit 0
else
echo "Defect not demonstrated"
exit 1
fiDefect demonstrated: Parse failed on text with preceding braces:
--- FAIL: TestParseWithPrecedingBrace (0.00s)
repro_test.go:13: Parse failed unexpectedly: review: the reviewer's reply is not the requested shape: invalid character 'i' looking for beginning of object key string
FAIL
FAIL github.com/pdbethke/corralai/internal/review 0.002s
FAILThe reviewer's reproduction script genuinely fails and demonstrates the defect. I attempted to trace through extractJSON (review.go:284-308) and confirm the logic: it calls strings.Index(text, "{") to find the first opening brace, then counts depth from that point. On the input 'I reviewed the scope {internal/review}.\n{"opinion": ..., it finds the '{' in '{internal/review}' at position ~21, then attempts to parse from there. The JSON fragment starting at that position is invalid (begins with 'i' from 'internal'), causing json.Unmarshal to fail with the exact error shown. The script is well-formed for a reproduction: it writes a test, runs it, cleans up, and exits 0 on failure. I verified the code path is exercised and the bug is real. I could not refute the finding because the defect genuinely exists in the code.
confirmed by pdbethke — real: extractJSON took the first '{' in the text; the verifier tried and could not refute it. fixed in feat/verifier-seat (fenced block first, else the first balanced object that parses), with a test the old code fails
ref 21821837da8ed6fb13a9a3d98e949eb757fd2b1fdbcb919169694a8a4930a9ec#R1
Recorded so that "found nothing here" can be told apart from "never looked here".
The internal/review package provides scope loading, prompt brief generation, and parsing/verification of model outputs. However, extractJSON in review.go naively locates the first opening brace '{' in the text stream to extract JSON payloads, causing Parse and ParseRefutations to fail whenever prose or code blocks containing braces precede the JSON payload in LLM responses (Finding 1). Scope filtering, tier demotion rules in Reproduce, and refutation verification in Verify are otherwise well-structured and tested.
The reviewer identified a real defect in extractJSON that fails when the LLM output contains prose with braces before the JSON payload. The provided reproduction script successfully demonstrates this failure. However, the script as written does not actually prove the claim in the way a REPRODUCED finding should: it shows Parse fails on input containing prose with braces, but the test case uses prose that happens to be about the scope '{internal/review}' — a description unlikely in real LLM output. A more compelling reproduction would use natural prose with arbitrary braces. That said, the defect is genuine and reproducible: extractJSON finds the first '{' and assumes it begins valid JSON, which fails when preceding text contains braces. The other items the reviewer listed as sound are accurately characterized.