On November 3 I shut down an LLM council I had built myself: launchd jobs booted out, MCP server unregistered, hooks removed. The reason was a pre-registered blind A/B against one fresh-context subagent, which ended 0 wins, 4 losses and 0 ties for the council across four real decisions.
The score is the short part. Most of what I want on record is the procedure: how the two arms were blinded, why I replaced the judge after the experiment had started and what I disclosed about that, how I stopped at 4 of a planned 10 and logged it as a deviation, and the fact that the winning arm was the more expensive one. n=4 says little about councils in general, and I'll mark where it stops.
The server ran 8 historical-figure personas, all on claude-sonnet-5. The "debate" was one call writing claims and rebuttals together, and the problem statement was cut to 400 characters. I also had a second home-built council layer, but the A/B tested only the 8-persona server.
A re-evaluation on October 27 had already removed most of its reasons to exist. Multi-angle generation, task decomposition and arithmetic support were now absorbed by current frontier models and by Claude Code's built-in subagents and workflows. Two older results pointed the same way:
One result pointed the other way. A human blind review on June 21 found 3-4 of 6 questions in the council's draft critique valid, so the critique function itself was alive. That was the one function left to test, so the experiment asked one thing: does the council's critique of a draft produce more valid points than an independent critique pass?
Codex reviewed the plan twice independently and agreed with it. A third vendor was out of quota and did not review it.
Arm B's brief asked for "the most plausible ways this draft is wrong, up to 5, each one line of claim + one line of how to check. Do not write what you agree with."
The decision rule, excluding ties:
A council output carries persona names and a debating style, so it can't be scored blind as it stands. Both raw outputs went through the same sonnet extractor, which kept at most 5 items, removed persona names and style, and reduced each item to one line of claim and one line of check.
The two lists were then randomly assigned the labels X and Y, with the seed stored in the ledger. The mapping was revealed only after scoring.
Each list came out at 5 items. I have no figure for what that cap removed from either arm's raw output.
The original judge was me, alone. I had ruled out LLM judges because an earlier experiment put LLM-judge agreement at a kappa of about 0.
The first decision, D1, broke that plan. Its items were infrastructure facts, launchd and the like, and I could not judge them without reading code. My scoring would have turned into gut feel or into following an LLM anyway. Claude could not take over either, because it had read D1's raw output before normalization and was not blind.
So I amended the protocol on October 28. Zero scores were recorded at that point, but the timing was not clean, and the amendment says so: it came after seeing a first Codex verification result for D1, X n=3 and Y n=3. The X/Y mapping was still undisclosed to me. The decision rule did not change.
The new judges were two verifiers that never see raw output or the mapping: Codex in read-only mode and a fresh sonnet subagent. They got the same brief and ran the check commands. Each item received one of three labels:
n: true and not covered by the draftk: true and already covered by the draftw: the core factual assertion is wrongAn item is confirmed only where both verifiers agree. Disagreements are undecided, and they leave both the win count and the wrong-rate denominator.
This reversed the no-LLM-judges rule I had started with, so the amendment carries its own falsifier: if pooled Cohen kappa is below 0.4 after 10 decisions, the experiment ends as unjudgeable.
These are confirmed items only. A is the council and B is the critique pass.
| Decision | Topic | A new | A wrong | A undecided | B new | B wrong | B undecided | Verifier agreement |
|---|---|---|---|---|---|---|---|---|
| D1, October 28 | retire file bus + wake job | 1 | 0 | 4 | 3 | 0 | 1 | 0.5 |
| D2, October 28 | move two scheduled jobs off the council's env | 2 | 1 | 2 | 5 | 0 | 0 | 0.8 |
| D3, November 2 | drop a weekly audit job | 2 | 1 | 0 | 4 | 0 | 1 | 0.9 |
| D4, November 3 | adopt settings from a biweekly scan | 1 | 2 | 1 | 4 | 0 | 1 | 0.9 |
B won every round. The wrong rate over confirmed items was 4/13 = 0.31 for the council and 0/17 = 0 for the critique pass.
Pooled kappa ended at 0.599, excluding abstentions. It was 0.107 on D1 alone, 0.346 after D2 and 0.534 after D3, so it started under the 0.4 line and climbed over it. The falsifier did not trigger, with one qualification: it was written for 10 decisions and I stopped at 4, so 0.599 is a reading at n=4 and the checkpoint the protocol named was never reached.
Here is what confirmed items looked like:
D1 is the weakest round. 4 of the council's 5 items stayed undecided and verifier agreement was 0.5, so that loss rests on few confirmed items. It doesn't change the outcome band: had D1 gone to the council, its ceiling would have been 7/10, still short of keep.
Why the critique pass did better, I did not test. The design compares two arms and can't say which part of the council lost the points.
| Decision | A time (s) | B time (s) | B tokens |
|---|---|---|---|
| D1 | 95.5 | 388 | 131,676 |
| D2 | 56.4 | 254 | 118,593 |
| D3 | 60.1 | 234 | 130,127 |
| D4 | 64.3 | 448 | 138,861 |
Arm B was slower in every round. The council's token count exists only for D1: 13,076, API metered, against 131,676 for arm B. A's tokens were not reported on the later quick runs, so I have no token comparison for D2 through D4. The arm I kept is the slower and heavier one, and the win was not cheap.
The experiment's own machinery sat on top of that. Normalization and verification added about 94-105k and 124-152k tokens per round, plus one Codex run.
On November 3, after seeing the results, I stopped at n=4, before the target of 10 and before the deadline. That is a deviation from the pre-registration and it is recorded as one. The decision rule itself did not change.
The reasoning:
The rule had not produced "retire" by itself. The retirement at n=4 was my decision, made with the results in hand, and that arithmetic is the justification I wrote down.
The ledger's tally tool still says incomplete, because n is below 10. The written early-stop section is the record of why.
Verifiers' access to forbidden paths was audited, but unevenly: Codex's logs were grepped, while the sonnet subagent self-reported. I wrote about why a reviewer's reach matters more than what its prompt leaves out in An Empty Prompt Is Not a Blind Review.
The server was retired on November 3 with final backups taken, and the repository was kept. Retirement also turned up something unrelated to the A/B: the backup job had been silently skipping since October 2, because it looked for a Docker container that wasn't running. The last dump was from October 1.
I kept the independent critique procedure and the pre-call controls and ledger.
The retirement has its own falsifier. If a critique pass misses 2 or more defects on important decisions that later surface, and the council is reproduced catching them on the same input, I reconsider.
Both raw outputs went through the same sonnet extractor, which kept at most 5 items, removed persona names and style, and reduced each item to one line of claim and one line of check. The lists were randomly labeled X and Y, the seed was stored in the ledger, and the mapping was revealed only after scoring.
I was the original judge, but the first decision's items were infrastructure facts I could not judge without reading code, and Claude had read that decision's raw output, so it was not blind. On October 28, with zero scores recorded, two verifiers that never see raw output or the mapping took over: Codex and a fresh sonnet subagent. The amendment came after seeing a first Codex result (X n=3, Y n=3), which is disclosed, and the decision rule did not change.
Keep was already unreachable by the third decision, and at 0-4 the council's ceiling was 6/10, which only meant hold and extend to n=20. If the arms were equal, 6 straight wins has probability 1/64, about 1.6%. Stopping after seeing results is still a deviation from the pre-registration, it is recorded as one, and the ledger's tally tool still says incomplete.
No. The critique pass took 234 to 448 seconds per decision against 56.4 to 95.5 for the council. On the first decision, the only round with a council token count, it used 131,676 tokens against the council's 13,076.
No. n=4 is not a statistical claim about councils in general. All four decisions were one person's infrastructure decisions, both arms used the same model grade, and the council ran at default depth and quick on later runs. Why the critique pass did better was not tested.