This is the third note I've written about the same measurement pipeline turning up a bootstrap confidence interval of exactly [0, 0] that doesn't mean what it looks like. What's actually new this time: the pipeline has caught up with the first two notes. It now names this exact failure mode in its own documentation — and its most recent weekly snapshot still shipped thirty of these intervals anyway.
The pipeline is PAMSL, the same one behind Your Agent Telemetry Ranks Your Routing Policy, Not Your Models and Sub-Agent Metrics Are Not Comparable to Main-Thread Metrics. The snapshot here is dated 2026-08-31 — seven model epochs crossed with two roles, association-only per its own header, graded no higher than "C (exploratory)." Nothing below ranks a model. It's about what the report's Bootstrap 95% CI table still does, next to what the report's own footnotes now say about it.
The Bootstrap 95% CI table compares model-epoch × role cells pairwise, restricted to pairs where both sides clear n=20, resampled 2,000 times from a fixed seed (42), no p-values anywhere by design. A pair filter compares only within the same role — main against main, sidechain against sidechain — and skips any pair where both sides are already flat: median 0, interquartile range 0 to 0.
I counted the rows that survive it. Ninety-five, across six metrics. Thirty of them — just under a third — report a 95% CI of exactly [0, 0]: 12 of 22 tool_error_rate rows, 6 of 10 validation_run_count rows, 12 of 22 error_recovery_seq_count rows. None of same_file_reedit_rate, output_tokens_total, or completion_proxy land on a point.
Three concrete ones:
tool_error_rate, claude-haiku-4-5-20251001/base×sidechain vs claude-opus-4-8/base×sidechain: Δ = 0, CI = [0, 0]. Haiku sidechain: n=267, median 0, IQR 0–0.0526. Opus-4-8 sidechain: n=2781, median 0, IQR 0–0. The intuitive case — both medians sit at zero, and one side's IQR is a literal point.validation_run_count, claude-opus-4-8/base×main vs claude-sonnet-5/base×main: Δ = 0, CI = [0, 0]. The first group: n=211, median 0, IQR 0–4, mean 3.9573. The second: n=61, median 0, IQR 0–0, mean 0.3607. One group runs validation an average of almost four times per thread, with real spread up to its third quartile — and the interval on the difference of medians still reads as a single point.error_recovery_seq_count, claude-sonnet-4-6/base×sidechain vs claude-sonnet-5/base×sidechain: Δ = 0, CI = [0, 0]. The second group: n=3478, median 0, IQR 0–1, mean 0.4963 — real dispersion, not a constant — and the paired CI is still a point.Look at the last two examples again. In each pair, one side is exactly flat — claude-sonnet-5/base×main has median 0 and IQR 0–0; claude-sonnet-4-6/base×sidechain has median 0 and IQR 0–0 — and the other side is not, with IQR reaching 4 and 1 respectively and means that are clearly nonzero. The filter's skip rule needs both sides flat to fire. One flat side isn't enough, so neither pair gets skipped. They still collapse to [0, 0], because the non-flat side's median is 0 too: at least half of its own values are exactly zero, which is a narrower condition than "the whole IQR is a point," and it's the one the bootstrap actually runs on.
I've derived this mechanism in detail twice before, on this same pipeline. In July I named it directly: a [0, 0] interval is "tie-degeneracy: so many identical zeros that every resample returns zero". Ten days later, working through this same metric — validation_run_count, means then ranging 0.60 to 4.11 across model pairs, much like the 0.36-to-3.96 pair above — I found the pipeline's existing guard sailing a pair through and degenerating anyway, and called the guard "necessary and insufficient". That's not a new finding here. It's the same mechanism, five weeks later, still doing the same thing to the same metric. An interval that cannot move isn't precise. It's just not moving.
Between the second of those two notes and this snapshot, the pipeline changed — but not by fixing the thing I flagged.
The report's Coverage & Limitations section now carries item F14: for the two binary proxy metrics, completion_proxy and abandonment_proxy, a median-difference CI is explicitly forbidden and replaced with a proportion-difference CI, on the arithmetic grounds that a 0/1 metric's median difference can only ever be -1, 0, or 1. (That it's not much of an interval to build a statistic on is my gloss, not the report's wording.) That is a real fix — narrower than what I'd called for in July, which was checking whether the resampled statistic itself is constant, for any metric. F14 patches the binary metrics specifically. It doesn't reach tool_error_rate, validation_run_count, or error_recovery_seq_count, which is exactly where all thirty zero-width rows above come from.
It also carries item F15, translated from the original: "many cells are nearly constant (mostly 0), so the percentile bootstrap CI collapses to [0, 0] — this is not evidence of precision, but a coverage limitation of the median bootstrap under heavy ties." That is not a fix. It is a sentence in a Limitations section, naming a failure mode, attached to no specific row. The same document that carries it still prints all thirty zero-width rows above with nothing in the table itself distinguishing "genuinely nothing here" from "tie-degenerate, see the footnote."
The self-measurement problem has the same shape. The second of the two July notes found the pipeline measuring its own build-and-audit sessions and promised "the next iteration gets an explicit exclusion stratum." Five weeks later, this snapshot's own Limitations section still marks that stratum as planned for M5 — flagged, not built.
None of this is a complaint that the report is dishonest. It is unusually candid: this section alone also names same_file_reedit_rate conflating iterative editing with rework, a completion_proxy heuristic broken by the exact harness-logging convention I unpacked in the first of these two notes, 37 main and 790 sidechain threads dropped from every comparison for missing model attribution, and historical dispatch-policy versions folded into an explicit UNKNOWN stratum rather than guessed at. Candor at the bottom of a document and a clean number in the middle of a table are two different deliverables. A reader only ever sees one of them without going looking for the other.
Don't read a zero-width CI as "checked, no difference." Check the same cell's IQR and mean first — they sit one column over in the Core Metrics table and cost nothing to look up. If the IQR is a point on both sides, the interval is telling the truth. If the IQR or mean shows real spread on either side while the CI still reads [0, 0] — the validation_run_count case above, median 0 next to a mean of 3.9573 — the interval isn't confirming anything. It's tie-degenerate, and this report, at least, will tell you so if you scroll to the bottom and go looking.
That last part is the part worth fixing, if you own the pipeline rather than just reading its output. A limitation that lives only in prose at the end of the document protects a reader who already knows to look for it, and nobody else. The distance between F15 existing and F15 doing something is the distance between a footnote and a flag on the row: a tied-value share printed next to every median-based interval, or a suppressed cell instead of a manufactured zero — the same gap between citing a number correctly and reading it right, just moved from a model's output to a pipeline's own table. Naming a failure mode is a start. It is not the same work as making the specific row that failed impossible to misread.
It does not mean the true difference is precisely zero with high confidence. It means the percentile bootstrap of the median collapsed to a single point because a majority of the underlying values in one or both groups were tied at zero. That is tie-degeneracy, not precision -- the data is nearly constant and the resampling has nothing to move.
In the 2026-08-31 snapshot, 30 of the 95 rows in the Bootstrap 95% CI table -- just under a third -- report a 95% CI of exactly [0, 0]. They cluster in three metrics: tool_error_rate (12 of 22 rows), validation_run_count (6 of 10 rows), and error_recovery_seq_count (12 of 22 rows). None of same_file_reedit_rate, output_tokens_total, or completion_proxy land on a point.
The tie-degeneracy mechanism itself was written up twice before, in July 2026, on this same pipeline. What's new in this snapshot is that the pipeline's own documentation has since caught up and named the failure mode explicitly, and still shipped 30 unflagged zero-width rows in the same weekly run. Documenting a limitation and fixing it turned out to be two different things.
Partly, and narrowly. For the two binary 0/1 proxy metrics, the report now forbids a median-difference CI and uses a proportion-difference CI instead, because a median difference on a 0/1 metric can only take the values -1, 0, or 1. That fix does not extend to the continuous rate and count metrics, which is exactly where the 30 zero-width rows come from.
Check the same cell's interquartile range and mean before trusting the interval. If the IQR is a single point on both sides, the data really is that constant. If the IQR or mean shows real spread while the CI is still [0, 0], the interval is not confirming a null result -- it means the median stopped being an informative statistic for that cell, whether or not the report has a footnote saying so.