It could break the code but not test it

Field note. Three runs, two labs, two languages, and a bug of our own that nearly became a finding about somebody else’s model.

The question

Corral’s adversarial pool has three seats. A mutant-generator plants violations in your code. A test-writer authors tests to kill the ones your suite missed. A test-critic reads the result for tests that pass without testing anything.

We advertise that you can fill those seats with local models, frontier models, or a mix. So: can a small local model actually hold a seat, or is local mode a demo?

The answer turned out to depend entirely on which seat.

The comparison that made it obvious

The cleanest evidence came from a run we set up to test something else. In it, qwen3.5:9b-q8_0 occupied both the generator and the writer seat — same model, same file, same process, same 45 minutes.

Target: internal/admission/admission.go, 117 lines, standard library only, with a real test suite written by a human.

As the generator, it worked:

dev-adequacy: the dev's OWN tests scored 41%
  (killed 9 of 22 graded mutants, 13 survived — bugs the dev's tests miss)

Twenty-two of its mutants compiled and were graded. Thirteen of them defeated a real, human-written test suite. That is not a model failing.

As the writer, in the same run:

test-writer could not produce a compiling test after 3 attempts
  — 13 survivor(s) found but not proven-killed

admission_corral_test.go:6:2:  "fmt" imported and not used
admission_corral_test.go:37:69: expected ')', found ','

Meanwhile the frontier challenger on the same 13 survivors:

the challenger writer (gemini-3.5-flash) proved 8 of 13 survivor(s)

One model, two seats, one run. Competent at breaking. Unable to build.

Ruling out the obvious objection: is it just Qwen?

A single family proves nothing about local models, so we swapped the writer seat to gemma4:12b — a different lab, a larger model — and changed nothing else.

It also failed, three attempts, and the failures are worth reading:

admission_corral_test.go:1:1:  expected 'package', found The
admission_corral_test.go:21:6: Lease redeclared in this block

The first is prose leaking above the package line. The second is the model redefining a type the package already exports.

Both are the same underlying miss: it wrote a self-contained answer instead of a file that composes with the code around it. Gemini, same run, same survivors, proved 13 of 15.

Two labs, two model families, same seat, same outcome.

Ruling out the other objection: is it Go?

Go is strict. go vet rejects an unused import, a redeclared type, wrong arity. A plausible story is that these models write fine tests and Go’s gate is simply unforgiving.

So we ran the same shape against Python — transform_parser.py, 114 lines, standard library only, 15 passing tests — where the compile check is py_compile, which validates syntax and nothing more.

We predicted the local writer would clear the weak gate and then fail at runtime against correct code. That prediction was wrong, and the way it was wrong is the point:

the pool's authored test graded soundly
  (CompliantPass=true CanaryKilled=true Total=3)
  but killed NONE of the 3 survivor(s)
  — a real 'tried and missed', not an ungraded run

qwen3.5:9b wrote a valid, sound, correctly-passing 15-case test suite. It parsed, it ran, it killed the canary, it passed against the unmutated code. It simply did not kill anything.

The tests it wrote explain why. test_thumb, test_large, test_all_presets, test_width_only — a textbook characterization suite, re-verifying the happy paths the developer’s tests already covered. It never went near the three survivors. Gemini, same three, killed two.

So Go’s strictness was not the barrier. It only changed where the failure became visible. Go rejects the local writer at the gate. Python lets it through to be measured — and the measurement says it tried and missed.

The result we deleted

The Gemma run above is the second one. The first produced three failures reading expected 'package', found 'EOF' — an empty file — and a tidy conclusion was available: a Google 12B model cannot author a Go test.

The signature was wrong for the hypothesis. EOF is not bad Go, it is no Go.

gemma4 turned out to be a reasoning model that routes its answer through a separate thinking field. When the token budget runs out mid-reasoning, the request still returns HTTP 200 with an empty content and no error at all. We already had a package handling exactly this for two other families. Gemma was not in it.

gemma4:12b   plain        thinking=424c  content=0c   done=length
gemma4:12b   think:false  thinking=0c    content=19c  done=stop

Three reproducible failures, a plausible story, and it was our bug. The fix was one line plus tests; the run was redone and only the redone one is reported here.

Anyone publishing model comparisons should sit with that. The wrong conclusion was reproducible, and reproducibility is not correctness.

What we are claiming, and what we are not

Claiming: across three runs, two model families, and two languages, small local models held the destructive seat and failed the constructive one — while a hosted model on identical inputs succeeded. Breaking and checking are separable skills, and small models have far more of the first.

Not claiming that these models are bad, that local inference is a dead end, or that this generalizes to every model or task. This is three runs on two small files. The Python survivor set was 3 — our floor for reporting anything — and deserves to be read as an existence proof, not a coefficient.

Two numbers we are not tidying away. Three runs on the same file produced dev kill rates of 0.41, 0.58, and 0.64, because mutants are regenerated each time; that spread is run-to-run noise and a caution against reading any single rate too closely. And the generator’s invalid-mutant rate ran from 2 of 12 in Python up to 24 of 60 in Go — a separate problem from the one this note is about.

Why this changes how we would deploy it

The practical reading is not “local models don’t work.” It is that seat assignment is the decision that matters.

The generator seat is where a small local model earns its keep. It is destructive, needs no compositional understanding of the surrounding package, and a bad output is free — the compile gate throws it away and labels it evidence about the generator rather than about your tests. In our Go run a 9B model in that seat produced thirteen bugs a human suite missed.

The writer seat is different. It has to compose with existing code, satisfy a compiler, and target a specific gap. That is where every local model we tried fell over, in both languages.

A mixed herd is not a compromise between a cheap tier and a good one. It is putting each model where its actual strength is: local models breaking things, a hosted model proving the break matters.

If you are running a local-only pipeline that both finds problems and writes the checks for them, it is worth asking which half you have actually verified — and whether anything in your setup would tell you if the checking half were quietly producing tests that pass without testing anything.