Terribly sorry to trouble you… but your tests, um, well. They suck, really

“Mr. Frampton, I understand that you, as it were… have… well, let me put it another way…”

Field note. Named with love after two critics: the late Roger Ebert, who knew the kindest thing you can do for bad work is to say so, precisely — and John Cleese, who knew you can say very nearly anything if you apologize sincerely enough on the way in. There is a third debt owed here, but it doesn’t announce itself until the end.

The green check is a liar

Ebert kept a stack of one-star reviews so good he published them as a book: Your Movie Sucks. The joke was never cruelty. It was precision — the exact, itemized reason a thing that looked finished didn’t work.

Your CI is running the same con in the other direction. A green check means your tests ran and passed. It says nothing about whether they’d notice if the code broke. A test that asserts nothing passes forever. A suite that only exercises the happy path waves through every bug that lives on the sad one. The check is green; the tests are furniture.

So I built the critic. Point it at a change and its own tests, and it does what Ebert did: it doesn’t take the poster’s word for it. It mutates the code — plants real, goal-violating bugs — and runs your tests against each mutant. If your tests kill the mutant, good. If the mutant strolls right past them, that’s a survivor: a bug your suite doesn’t test for. The kill-rate is your grade.

Then, because a critic who only complains is useless, it writes the test you should have had (and proves it catches the bug), and a second, deliberately different model reads your tests cold and files the pan.

And it files it politely. If Ebert supplies the standard, John Cleese supplies the bedside manner — the verdict lands the way the nicest man in the sketch delivers the worst possible news: terribly sorry to trouble you; lovely validator, really, top marks on the length check; it’s only that a password with no digit strolls straight through, and I did feel you’d want to know. I’ve taken the liberty of writing the test you were missing. No rush. You are allowed to tell a suite it’s broken to its face, provided you are gracious about it and hand over the fix on the way out. That politeness isn’t a bit — it’s the humility the whole design is built on. A gate whose first principle is no one may judge their own cause has no business being smug.

But Cleese was working a vein far deeper than manners, and it happens to be the exact reason this tool needs to exist. His whole career was built on one exquisite agony: a person who plainly perceives the obvious thing and cannot force it past their own lips. In the very first episode of Flying Circus, an interviewer has landed the scoop of the century — Arthur Frampton, the man with three buttocks — and, live on air, cannot bring himself to ask the one question the entire nation tuned in for. He circles it in strangled euphemism (“you have a certain, shall we say, something…”), physically unable to name what everyone can plainly see.

Freeze that frame — the one up top — and you have a diagram of software testing. The interviewer sits with his legs crossed away, hands knotted, his eyes fixed on the exit — his entire body turned from the very thing he came to discuss. Beside him the guest waits, patient and entirely willing to answer: the truth is right there, nameable, cooperative, asking only to be asked. The few polite feet between them that neither man will cross is your green check. Nothing is hidden. Nothing is hard. Someone simply has to say the thing — and no one in the room can. That is your code review, precisely. Your reviewer saw the test that asserts nothing. They did. And then they typed “LGTM,” because “this test doesn’t actually test anything” is a genuinely awkward sentence to say to a colleague you’ll stand next to at tomorrow’s standup. The flaw was obvious and the sentence was unspeakable, so the green check shipped. Corral’s one real advantage over your reviewer isn’t intelligence — it’s that it has no standup to attend, no feelings to bruise, and no fear whatsoever of being the pedant. It clears its throat and says the obvious thing, out loud, politely, every single time. It is the one who finally, courteously, finishes the interviewer’s sentence — “I have with me Mr. Arthur Frampton, who has a test that asserts nothing.”

And the sketch has one more turn that is, unnervingly, this entire tool. The interviewer eventually rounds on his guest — you can’t just come on here claiming something; our viewers want proof — and Frampton, who came on the show precisely to be seen, now demurs, too embarrassed to actually show the thing. Everyone in the room agrees proof is required. Nobody can bear to produce it. That is certify-by-execution’s whole reason for being: the proof that a bug is real and catchable is trivial to generate — you run the test against the bug and watch it fail — and yet in a room full of polite people it never gets generated, because demanding it and supplying it are equally mortifying. Corral has no such difficulty. It runs the missing test against the survivor, watches it die, and hands you the signed proof. Our viewers want proof; corral is the only member of the panel with no capacity for embarrassment, and so the only one who will ever provide it.

Here are three real runs. Local models, a jail, a signed record at the end. I fed it toy code with tests I wrote to be bad on purpose — this is a demo, not an audit of your repo — but the machine didn’t know that, and the numbers are real.

★★½ — “Competent, and misses the one thing you’d get paged for”

The code: a Divide(a, b) that returns an error on divide-by-zero. The dev’s test: Divide(6, 2) == 3. Ships. Green.

The verdict:

dev_kill_rate = 0.83   (killed 5 of 6 mutants, 1 survived)
survivor: the mutant that deletes the `if b == 0` guard
status: certified   —   proven_missed: 1

Eighty-three percent! On paper, a strong suite. And still — the one mutant that survived is the one that removes the divide-by-zero check, because Divide(6, 2) is still 3 whether or not the guard exists. Your test never divides by zero, so it can’t tell. The pool then wrote the test you didn’t (Divide(1, 0) must error), ran it against the survivor, and it died — proven_missed: 1. The gap is real and catchable; you just didn’t catch it.

This is the two-and-a-half-star review, and it’s the most useful one Ebert ever wrote: technically fine, forgettable, and quietly broken in the exact place that will page you at 3am. It even certified — barely — which is the honest, uncomfortable part: the tool doesn’t fail a suite for one gap. It signs the evidence and hands you the receipt.

★ — “Tests the door, ignores the locks”

A password validator with three rules: at least eight characters, a digit, an uppercase letter. The dev’s test checks one thing — that "Password1" is accepted.

dev_kill_rate = 0.75   (killed 6 of 8, 2 survived)
status: needs-review   (below the 0.8 bar)

Below the bar, so it routes to a human instead of certifying. Of course it does. A test that only feeds a valid password can’t notice when you weaken a rule — loosen the length check, drop the digit requirement — because the valid password sails through either way. You tested that the door opens. You never tried a bad key.

☆ — “The test calls the function and checks nothing”

And the zero-star. The one Ebert lived for. A “smoke test” — the kind that exists to make the coverage number go up:

func TestValidatePassword(t *testing.T) {
  _ = ValidatePassword("Password1")  // calls it, asserts nothing
}

dev_kill_rate = 0.50   (needs-review)

Fifty percent — and the only reason it isn’t zero is that half the mutants make the function panic on the input, so the test dies by accident, not by design. It never checks a return value. It tests that the function can be called.

And the second model — the critic, a different LLM than the one writing tests, so it can’t grade its own homework — read it and filed this, verbatim:

“The test ValidatePassword calls ValidatePassword without checking its input.”

That’s the pan. Cold, correct, and exactly what your reviewer should have said in the PR and didn’t.

Why this isn’t just an LLM with opinions

Because the grade is not an opinion. The kill-rate is mutation testing, run in a jail, by the brain — not self-reported by the model that wrote the test. “Your test asserts nothing” isn’t a vibe; it’s an objective measurement (a vacuous test kills ~zero mutants on purpose). The model that critiques is forced to be a different model than the one that writes — a judge may not certify her own homework. And the whole verdict is signed: a tamper-evident record you can verify offline, naming which model played which role. The divide-by-zero run above produced exactly that — Actor: corral-advpool, anchored, ProducedBy: [mutant-generator:qwen, test-critic:llama, test-writer:qwen]. Evidence, not a star rating.

The honest part (Ebert would insist)

This is experimental and off by default. The verdicts above are grounded, but I’ll tell you what wobbled: I ran it on small local models (a 7B and a 3B on a desktop GPU), and the 7B repeatedly wrote a corroborating test that wouldn’t compile — which is exactly the argument for the gate-earned routing underneath this thing: on real work you’d route the writer to a model that’s earned it, and the leaderboard learns that from the signed outcomes. The complete, signed verdict I showed is the divide run; for the weaker suites I’m quoting the dev-adequacy grade — the headline — because that’s the review of your tests, which is the whole point.

And the part I love most: exercising this feature found three real bugs in the feature. Its own hermetic test suite — the fakes, the mocks — passed clean and green. The moment I ran it against real code with a real model, it fell over three times: a fast-path that keyed on the wrong role, a dropped field, a fence it forgot to strip. The tool that grades your tests for testing nothing was, itself, shipping with tests that tested slightly less than they claimed. I fixed them in the open. I’d have deserved the pan.

Ebert gave zero stars to movies that insulted his intelligence. Your CI gives a green check to tests that insult yours. This is the difference between “it passed” and “it was worth passing” — and it turns out you can measure that, sign it, and hand someone the receipt.

And now the third debt, the one I promised at the top. The first language this gate ever graded in anger was Python — which is named, as Guido van Rossum will happily tell you, after Monty Python’s Flying Circus. So the critic that turns to your Python suite and says, with infinite courtesy, that it’s terribly sorry to trouble you but the tests rather suck — has been speaking in the exact voice of the circus the language was named for, this whole time. I would love to claim I planned that. I did not. It’s simply the sort of thing that turns out to be true if you build honestly for long enough. Ebert for the standard, Cleese for the manners, and the language itself finishing the joke.

Your tests suck. Terribly sorry. Here’s the proof.