I recently ran a pre-registered experiment that ended with a verdict that took longer to accept than to write: undeterminable, terminate, do not re-run. Everything meant to protect you from fooling yourself had been followed — randomized assignment, a primary metric frozen before the first data point, an amendment appended on 8 July rather than slipped in, a falsifier condition written in advance to stop me from forcing a verdict out of too small a sample. All of that discipline produced one honest outcome: the sample the experiment could realistically reach was never going to be large enough for the effect it was built to detect, and no instrumentation quality would change that arithmetic.
This is that experiment, and the general shape underneath it — the failure mode it avoided, quietly reporting an underpowered comparison as if it had settled something, is not specific to model routing or to me. It shows up in A/B tests, benchmark submissions gated on a margin, any two-arm comparison run on a sample smaller than the effect it is chasing.
I route coding tasks between two tiers of the same model family — call them Sonnet and Opus — depending on how much judgment a task looks like it needs. That routing is a policy, not a measurement: I decide where each task goes, so any difference I see afterward is entangled with my own decision about which tier gets the harder work. Random assignment was intended to separate model behavior from my routing choices; in this study, the missing assignment-to-thread link prevented that comparison from supporting a causal conclusion.
So I pre-registered an experiment: for tasks matching a defined set of dispatch conditions, a coin flip would pick Sonnet or Opus instead of my judgment, logged as it happened. A frozen primary metric — how often the same file gets re-edited inside one run, a behavioral measure rather than a measure of task success — would say, after four weeks, whether the two tiers differed. Two secondary metrics, a tool-error rate and a validation-after-edit rate, rode along. Evaluation date, metric definitions, and unit of analysis were all written down before the first coin got flipped.
The study closed a day earlier than pre-registered, a minor deviation logged rather than absorbed silently. Four weeks in, the mechanics had otherwise worked exactly as designed: 177 draws, 90 to Sonnet and 87 to Opus, close enough to call the randomization clean, clearing the rule of thumb most people reach for when eyeballing a sample size, "at least 20 per arm" — and it was the wrong number to check.
The pre-registration had defined the real unit one level down from a draw: not which model got picked, but what happened in the work thread it produced — and connecting the two reliably wasn't possible. Draws had timestamps; work threads didn't, only their enclosing sessions did, averaging about sixteen threads each. Of 177 draws, 142 fell inside some session's window, but candidate sessions per draw ranged three to ten (six typical), zero landing on exactly one. A secondary label that might have substituted for a join key was almost as generic across the board.
Run the pre-registered unit of analysis through that join and the honest count of usable observations per arm was zero, not 87 or 90. A falsifier condition existed for exactly this: below a floor on the real unit of analysis, stop, report undeterminable, and hand the call back to a person rather than force a number out of data that can't support one. It fired — no difference, interval, or p-value got computed, since there were no correctly joined groups to compute one on.
A second check — whether the randomized assignment actually ran as recorded — fared no better: an eighteen-draw hand review, about ten percent of the total, found mostly legitimate cases but one mismatch, too small a sample to say whether that crossed the twenty-percent threshold flagging the design broken — and no causal claim got made anywhere, since only a confirmed link between assignment and outcome earns that right.
This matters more than the join-key bug — the fix isn't "add a thread ID and run it again." Before recommending whether to extend the study, I pulled a large pool of already-observed behavior for both models — 1,339 Sonnet turns, 2,766 Opus turns, ordinary usage — and computed the minimum detectable effect (MDE) the study's actual arm sizes could support: the smallest true difference a sample of 87 versus 90 could reliably detect at 0.05 significance and 80 percent power.
| metric | Sonnet mean | Opus 4.8 mean | observed gap | pooled SD | MDE (n=87/90) | arm n needed |
|---|---|---|---|---|---|---|
| same-file re-edit rate (primary) | 0.0546 | 0.0440 | +0.0106 | 0.1682 | 0.0709 | 3,953 |
| tool-error rate (secondary) | 0.0380 | 0.0225 | +0.0155 | 0.0742 | 0.0313 | 360 |
| validation-after-edit rate (secondary) | 0.0297 | 0.0360 | −0.0063 | 0.1802 | 0.0759 | 12,843 |
The primary metric's observed gap was 0.0106; the MDE at this sample size was 0.0709 — 6.7 times larger. At the pace draws were accumulating, about 22 per arm per week, reaching the needed 3,953 per arm would take roughly 180 weeks: three and a half years. The validation-rate row would need over eleven years; only tool-error was reachable on any sane timeline, around sixteen weeks — and that one was secondary. Promoting it now would just be picking the question after learning which one was affordable — exactly what a frozen primary metric exists to prevent.
The join-key bug had been hiding this arithmetic — tempting to treat a zero-observation study as a plumbing problem, fix it, re-run. But the join and the power problem were independent faults; a perfectly repaired join would still have needed about 45 times more observations per arm than four weeks produced. Nor was it a one-off: the same week, a separate project I run hit the identical wall from a different angle, its own gate thresholds smaller than its measurement noise — one shared failure, two systems, an observed effect roughly one-sixteenth of the pooled standard deviation. Compute the MDE at design time, before a single data point exists, and a four-week version of either study never gets built.
The pre-registration wasn't touched only once. On 8 July, an external review flagged a gap in how draws were recorded, and an amendment was appended — dated, alongside the original text, not folded silently in — adding three things: a human-readable label for each draw's task, a flag for test draws, and a guard meant to catch re-rolled draws.
None of the three did the job. The label was information for a person, not a key for a machine — free text, when the analysis needed a stable identifier or a timestamp at the level it actually measured, and neither got added. The test flag was used zero times in four weeks. The re-roll guard flagged forty-four events, 89 percent of them ordinary batch assignment misread as re-draws — nearly useless for the one thing it existed to catch.
The review that triggered the amendment had actually recommended something more drastic: stop, redefine the design, pre-register a fresh version from scratch. The amendment took the cheaper path — keep the design, add fields — and that choice didn't survive contact with the analysis weeks later. The mistake generalizes: "a person can read something useful here" and "the target analysis can be computed from this" are different claims, and checking only the first is how a team ends up with new data and no way to run the comparison it promised.
None of this reads as a success story, and it should not be written as one — an experiment that cannot answer its own question is not a good outcome, however cleanly it was run. But the discipline did something specific: it produced no answer, because nothing was going to at this sample size, while stopping a wrong answer from quietly becoming the record. A falsifier written down before any data existed is the only reason this ended as "undeterminable, here is exactly why" instead of a routine "no significant difference found" — a sentence that reads as a result to anyone who doesn't separately check whether the study could have found one at all.
The same verdict carried a second, easier question about a different system: had four weeks of a separate reporting pipeline changed a real operational decision even once? The ledger recorded two routing changes, although one was self-referential; under the strict independent reading, the remaining change still cleared the one-change bar and coincided with workflow self-inheritance falling from 48 to 2 percent. Same author, same week, same discipline, wildly different sample requirements: one question needed to observe a single event; the other needed to resolve a small continuous difference against noisy data, taking orders of magnitude more observations. Power isn't a property of how carefully something is measured — it's a property of what is being asked, and how much data that requires.
The falsifier that ended the study also required routing the extend-or-terminate call back to a person, with a recommendation attached rather than a unilateral decision buried in a log. Mine was to terminate, not open a follow-up of the same design: a perfectly repaired join hits the same wall, and the one reachable metric is secondary — promoting it now would just be the HARKing move the frozen primary metric exists to block.
I attached one condition, because a recommendation without a stated way to be wrong is an opinion wearing a falsifier's clothes: if what actually matters is whether the task got done correctly, not these behavioral proxies, a real difference could be far larger and the required sample much smaller. That number doesn't exist yet — already flagged, separately, as the biggest measurement gap in the project. So the honest next step isn't a bigger version of this experiment; it's building the measurement that would say whether trying again is worth four more weeks.
None of this is specific to model routing, or to me. It applies to any two-arm comparison: an A/B test, a benchmark submission gated on a margin, a model-comparison table finally randomized properly. Work out the smallest effect that would change a decision, then the smallest effect your realistically obtainable sample could distinguish from noise. If the second number is larger — sometimes much larger — the problem isn't instrumentation. Better logging, cleaner joins, a stricter pre-registration document: none of it moves that number. You are looking at a design that cannot answer its own question no matter how well you run it, and the honest move is saying so before spending weeks or months collecting data that was never going to resolve anything.
Terminating a study like that and writing down why is a result. It tells the next person not to walk the same road without first fixing the real constraint — usually the rate real data accumulates, not the pipeline recording it. The quiet failure runs the other way: running the same underpowered comparison anyway, getting a number that isn't statistically distinguishable from zero, and reporting it as "no difference" instead of "no answer." Pre-registration, frozen metrics, and a falsifier condition don't prevent an underpowered design from being built. They make sure that when the sample is too small, the study says so out loud instead of dressing up a null result as a finding.
Q. Why did a properly randomized, pre-registered experiment end undeterminable instead of reporting a result?
Because the pre-registered falsifier fired: the real unit of analysis could not be reliably joined back to the randomized draws that produced it, so the honest count of usable observations per arm was zero. Reporting a difference anyway would have meant computing a number from data that could not support one, and the falsifier existed to block exactly that.
Q. The study had 177 randomized draws — why wasn't that a big enough sample?
177 draws split 90 and 87 clears the informal rule of thumb of at least 20 per arm, but that rule checks the wrong unit. The pre-registration measured behavior at the level of individual work threads, not draws, and those threads could not be reliably matched back to the draws that assigned them — most draws had six or more candidate matching sessions, not one.
Q. If the join key had been fixed, would the experiment have worked?
No. A minimum detectable effect calculation on a large pool of ordinary usage showed the study's actual arm sizes of 87 and 90 could only reliably detect an effect 6.7 times larger than the real gap being measured. Reaching the sample size the primary metric needed, around 3,953 per arm, would have taken roughly three and a half years at the rate draws were accumulating.
Q. What did the 8 July amendment fix?
It added a task label, a test-draw flag, and a re-roll guard, but none solved the actual problem. The label was free text meant for a person, not a stable key a join could use; the test flag was never triggered; and the re-roll guard misidentified ordinary batch assignment as a re-draw 89 percent of the time it fired.
Q. What is the general lesson from this experiment?
Statistical power is a property of the study's design, not of how well it is instrumented. Before running a comparison, work out the smallest effect that would change a decision and the smallest effect your realistically obtainable sample could distinguish from noise. If the second number is larger than the first, better logging or cleaner joins will not fix it — the honest move is to say the design cannot answer its own question, rather than quietly report an underpowered comparison as a settled result.