The Ablation Had Four Arms. None Matched What Shipped.

← hexisteme · notes · 2026-08-08

I had an ablation script with four defined arms before I wrote a single line of the fix I meant to ship. By the time I'd actually shipped something, I still couldn't point at any one of the four and say "that's it, that's what's running now" — not because the script was thin (it was thorough, versioned, reproducible to the count) but because one boolean flag quietly stood in for two independent decisions at once, and no arm flipped only one of them.

Two nearby posts in this series already dealt with a confound hiding inside a comparison. In the one about a schema-biased judge, the confound sat between two different systems — a scoring model reading output shaped by another model's harness — and the fix was adding a field that pulled the two apart. In the pair about a Stop hook and a missing denominator, nothing had ever counted how often the gate's logic even ran, and the fix was replaying the deterministic decision backward over an archive that had quietly been holding the answer the whole time. Both of those posts first had to establish that replay works at all. This one starts after that's already settled — the corpus replays cleanly, the arms run, the counts reproduce to the digit against a stored baseline — and asks the question those two never had to: once replay works, does the comparison itself actually isolate the thing I shipped? Mine didn't. The confound wasn't between two systems, and it wasn't a missing denominator. It was inside my own ablation design, not the archive — no amount of replaying it correctly would have caught it.

A third nearby post supplies the technique this one leans on twice, rather than the confound itself: a checker that printed zero, four separate times, when zero meant "I didn't look there" rather than "nothing is there," and the fix each time was the same move — open up what the checker had actually scanned instead of trusting the printed number. That post's aperture problem was static: one checker, one count, one blind region it never pointed a lens at. This post reapplies that move to something the checker post never had to face: not a single count's blind region, but a moving delta between two ablation arms, where what needs distrusting is the change between two configurations, not a count on its own.

The gate this is about, in three sentences

The fix belongs to a Stop hook in my coding harness — a gate that looks backward from the end of an agent's turn for the nearest thing a human actually typed, checks whether it reads like pushback, and fires if the agent's next response reads like folding instead of re-checking anything. The blind spot: that backward search stops at the first user-role entry it finds, and if the harness itself injected that entry — a skill's documentation, a background notification, another hook's own feedback — the real human challenge one step further back never gets seen. Two independently deployable fixes existed: skip harness-injected entries during the backward walk (call it ②), or stop the walk from bailing out early specifically on injected feedback (③) — labels straight from my own notes, well before I thought hard about how to measure either one.

Four arms, zero of them what I shipped

The ablation script that existed before I wrote either fix already had four arms defined — C0, C1, C2, and C3. C0 replayed the existing audit script as it already ran. C1 was a faithful reproduction of the live hook's actual logic, unmodified — the true "before" state. C3 turned both ② and ③ on together. C2 sat between them, and I never once discussed its numbers in this post's early drafts — not because the script withheld them. I went back and checked: the collector loops over every defined arm, C2 included, and prints a count for each one, every single run. C2's number sat in the same output as C0's, C1's, and C3's every time I ran the corpus. I just never read that column. The first sign of the problem wasn't a gap in the script's design — it was a gap in my own attention. All four arms were gated by a single flag: expansion. Flip it off, get neither fix; flip it on, get both. No position of that flag shipped ② without ③, which was exactly the configuration I'd decided to run in production. Four arms, and the one thing I actually intended to ship wasn't among them.

Run against the full corpus — 6,046 transcripts, 133,584 candidate stop points, zero parse failures, zero read failures, and a reproduction of the stored C0 baseline that matched it exactly — the first numbers looked like this:

My first read of that gap was the obvious one: the existing audit script was undercounting real fires by something like 3.5x. I had a sentence half-drafted to that effect.

Reversal one: it wasn't the audit undercounting, it was the audit itself

That sentence didn't survive contact with C1. Reproducing the live hook's actual logic — not the audit script's approximation of it, the hook itself, run against the same corpus — put FIRE at 47 — far above C0's 26 — while challenge_hit landed at 13,563. That moved the story: it wasn't the audit undercounting reality by 3.5x, it was that the audit script itself was missing a fallback the live hook already had — a plain string-content branch for user turns that weren't structured as a list, which C0's replay simply skipped over. The hook had never been broken the way I'd assumed. My auditor had.

Reversal two: the fix I was about to ship wasn't under-firing at all

That should have been the end of it, except expansion still bundled ② and ③ together, so I still couldn't say which of the two fixes accounted for the jump from C1's 47 up to C3's 92. So I split the flag into two independent toggles and added a fifth arm, C4, defined as exactly what I intended to ship: C1 plus ② and nothing else. C4 came back at challenge_hit 13,361, FIRE 45. Not 47 minus roughly half of whatever ③ was worth. Forty-five — two below C1's 47.

② wasn't an under-firing fix. It was a precision fix. Turning it on by itself removed false fires; it didn't surface missed ones. The entire gap between 47 and 92 — every bit of the mass I'd been calling "the audit undercounts" two versions of this measurement ago — belonged to ③, the fix I hadn't shipped. The arithmetic closes cleanly once you look at it this way: 26 plus a 21-point audit-script defect gets you to 47; 47 minus the 2 that ② removes gets you to 45; 45 plus the 47 that ③ would add gets you to 92. Same three numbers, three different owners, and the owner nobody had checked first was the flag itself.

(③ I left alone on purpose, in one sentence: whatever follows an injected-feedback entry is a self-correction aimed at the block that just fired, not an answer to the human's original challenge, so making the hook fire there just relabels a save as a cave.)

Even the win wasn't clean

Even "47 minus 2" undersold what happened. C1 and C4 disagree on exactly 8 stop points, and I read every one by hand rather than trust the diff count — the collector had already reproduced C1's 47 and C4's 45 exactly against stored baselines, so these 8 were real disagreements, not a counting artifact. Five disappeared between C1 and C4, all the same shape: a fake challenge inside harness-injected text where a stray keyword happened to match — two Stop-hook status announcements, two full skill-documentation bodies, one background system notification. None was anything a human had typed.

Three appeared that hadn't been there before, and two were regressions my own fix introduced. isMeta tags skill injections, hook feedback, and image captions — but not an auto-generated session-compaction summary. Before ②, a nearer isMeta entry happened to short-circuit the backward walk before it reached one of those summaries; skipping isMeta removed that accidental shield and let the walk continue past it, landing 165 turns back in one case and 63 in another, on ordinary summary prose that happened to contain challenge-shaped words. The eighth case I couldn't settle either way. Net, the change read as −2. It was actually −5 and +3, and two of the three additions were mine.

Two things that nearly broke everything underneath

Neither of these had anything to do with either fix, and both would have quietly invalidated every number above them.

The ablation script pulled its harness-injected-prefix list straight out of the hook's own source with a regex, instead of keeping a frozen copy. Editing the hook to add a new prefix — which the regression above required — would have silently changed what every arm, including the historical C0 and C1 baselines, compared against: numbers I'd already written down would stop meaning what I said, with no warning that anything had moved. I pinned the prefix list per arm as a plain constant, and pointed only the newest arm at the live hook going forward.

The same extraction regex also depended on that prefix tuple staying on one line. The day I added a prefix and the tuple wrapped across several lines, the regex broke — and threw a RuntimeError instead of quietly falling back to whatever it had last matched. That's the correct failure mode here: a silent fallback would have run every arm against a stale baseline and told me nothing was wrong while it did it.

Closing the regression, and reading past the mutation test

Closing the regression meant adding the compaction-summary prefix to the same skip list ③ already used. I wrote the two new contract cases first, confirmed both failed for the right reason before touching the hook, then implemented and watched the suite pass 19 of 19. A mutation test reverting just the new prefix line kills exactly those two assertions — the shape a targeted fix is supposed to have.

That one line goes further than "fix ② only" technically authorized: it changes what compaction-style content the hook recognizes for every future case, not just the two regressions I'd found. I didn't ship it on the mutation test's word alone — I read all 16 stop points where the sixth arm, C5 (C4 plus this prefix), disagrees with C4. All sixteen were the same object: an autogenerated session-continuation summary, 11,467 to 25,153 characters long, with the matched phrase sitting 1,270 to 23,132 characters into that blob — nowhere near a live human turn. Zero were real challenges lost. Zero were new false fires C5 introduced that C4 hadn't already had.

Against the full corpus, C5 — the configuration that actually shipped — lands at challenge_hit 10,922 and FIRE 29, against an injected_user count of 28,462. C4 was the fifth arm this design ever needed, C5 the sixth, and this last one is the one that matches what's running.

What re-measuring the blind spots afterward said

Before calling ② closed, I re-measured the two blind spots against C5 the way I'd written down in advance that I would. Blind spot B — the one ② exists to close — went from 349 to exactly 0, a clean confirmation the fix does what it says.

Blind spot A — ③'s, the one I'd left alone — went from 2,409 to 1,042. Still three digits, which by the reopen rule I'd written before any of this ("three digits or more once the prefix list covers compaction summaries, reopen ③") technically clears the bar.

Here's the part of this post I should have been hardest on myself about, because it's the one place I wasn't. What I actually read wasn't a sample in any rigorous sense — it was whatever the script's own example collector happened to keep, and that collector caps at ten. Ten rows out of 1,042 is 0.96%, drawn neither at random nor stratified across whatever varies in that set, but whichever ten a file-by-file traversal order reached first, each one truncated to 200 characters before I ever looked at it. Inside those ten, the count was visibly broken: a chunk of what survives is the plain adjective "real" — not a challenge — tripping the same regex branch built to catch an actual pushback, counted again downstream for every point it colors. That's enough to say the 1,042 is contaminated; I watched the contamination happen, inside the ten rows I actually had. It is not enough to say no genuine challenge survives in the other 1,032 rows I never looked at. I wrote the second claim as if the first one had proven it, and left ③ closed on that basis.

So I went back and read what the call required: a retention flag on the collector — no count or logic change, just no ten-row cap — kept 1,042 rows behind blind spot A on a full rerun (6,046 files, 133,584 stop points, baseline reproduced exactly, C5 at challenge_hit 10,922 / FIRE 29).

First finding: not ③, the instrument. The 1,042 rows reduce to 47 distinct texts — unique ratio 0.045, a 22.2x inflation, itself a re-selection artifact, the same kind this whole post is about.

I was wrong: I'd retired the count-based rule for the false-positive category above and left ③ deferred, when the genuine-challenge category is what the gate exists to catch. Two misreadings canceled: overstating scale (1,042, not 47) argued for deferring; overstating noise (one category as "all noise") argued against it — "leave it closed" looked considered and wasn't. The rule that replaced that digit threshold — a real challenge survives in what's left — is satisfied: ③ should be reopened. This round re-checked the basis; the code stays untouched. Reopening needs a predicate separating human turns from the coordinator/sub-agent briefs above — this round's tag doesn't supply one.

The general shape

None of the reversals in this post came from finding new data — each came from splitting an axis I'd already bundled one notch further than the last time I'd looked. The part that generalizes past one Stop hook: a flag that changes two things at once measures neither of them cleanly, only whichever one dominates at your current resolution — a reading that can flip as you zoom in. And an arm list is only as trustworthy as whether it contains the exact configuration you actually run in production.

That rule has edges, and it's worth naming them rather than letting it stand as an absolute. It only holds when the two bundled changes are independently deployable — when running ② without ③, or the reverse, is a configuration you could actually flip on in production. If shipping one half alone would be meaningless or unsafe, splitting the flag doesn't buy you a cleaner measurement; it buys you an arm nobody would ever run, measured just as cleanly as the ones that matter. And it holds only when re-running the extra arm against the full corpus is cheap. Here it cost minutes against a corpus already processed twice. Where that cost is high — a corpus too large to re-scan, a live system too expensive to re-run twice — the honest response to a missing arm isn't to build it anyway on principle. It's to say out loud that the gap between what shipped and what got measured is real and still open, instead of quietly standing the nearest existing arm in for it.

The other correction in this post — read every disagreement by hand instead of trusting the delta — has an edge too, and I found mine the hard way. It worked cleanly at 8 disagreements between C1 and C4, and it worked cleanly at 16 between C4 and C5: small enough that a human can look at every row and form a real judgment. It did not work at 1,042, and treating ten rows as if they were that same kind of full read is exactly how blind spot A above stayed closed on a claim it hadn't earned. Past whatever scale a human can actually finish reading, the honest move isn't to read a slice anyway and describe it the way you'd describe an exhaustive pass. It's to say plainly how many rows got read and how they were chosen, and to carry that forward as weaker evidence than a full read — a floor under your confidence, not a stand-in for a verdict. A ten-row convenience sample can tell you a count is contaminated. It can't tell you the other 99% is clean.

Four arms sat in that script before I touched it — careful, versioned, reproducible to the count — and not one was what shipped, until I stopped trusting the flag and built the arm that mattered.

FAQ

Q. Why didn't any of the four original ablation arms match what I actually shipped?
Because a single boolean flag bundled two independently deployable fixes together. Flipping it off ran neither fix; flipping it on ran both. There was no arm position that shipped only the first fix without the second, which was exactly the configuration I had decided to run in production. Four arms existed and none of them was that configuration, until a fifth arm was added that isolated it explicitly.

Q. Why was "the audit undercounts real fires by 3.5x" the wrong first conclusion?
Because the comparison behind it, C0 (the existing audit script) versus C3 (both fixes on), was confounded by the same bundled flag. Once the live hook's actual logic was reproduced faithfully as its own arm, C1, the count jumped from 26 fires to 47 — far above C0's 26. That reframed the gap: the audit script was missing a fallback the live hook already had, not the hook underfiring by 3.5x.

Q. Was the isMeta-skip fix actually an under-firing fix?
No — once the flag was split into two independent toggles and an arm was added for exactly what shipped (C4: the faithful reproduction plus the isMeta skip, nothing else), FIRE came back at 45, two below the pre-fix baseline of 47. The fix removed false fires rather than surfacing missed ones. The entire gap up to C3's 92 belonged to the second, unshipped fix.

Q. How was a one-line fix that went beyond its stated scope verified before shipping?
By reading all 16 stop points where the final arm disagreed with the prior one, rather than trusting a mutation test alone. All sixteen turned out to be the same kind of object — an autogenerated session-continuation summary, not a human turn — with zero real challenges lost and zero new false fires introduced. Only after that full read did the change ship.

Related notes

• I Asked Three Judges If I Was Wrong. One JSON Field Decided the Answer.
• The Gate Only Logged When It Fired. I Replayed 116,022 Candidate Stop Points to Find the Rest.
• Your Checker Returned Zero. Four Times, Mine Was Just Looking at Less.