I set a threshold from measurement instead of guessing. The measurement was clean: zero overlap between the two clusters, a 33x gap between them. I wrote the numbers into a comment with their sample sizes, feeling good about not having guessed.
It was wrong, because the sample I had labelled "known good" was one of the bad ones.
I've written before about checks that cannot fire — guards whose thresholds were miscalibrated for the scale of their input, so nothing you fed them ever tripped the line. This is a different animal. My threshold was calibrated from data. That's exactly what made it convincing, and it's why the calibration itself is where the bug lived.
A video pipeline burns captions onto a rendered preview. A gate then diffs the burned output against the preview and treats every changed pixel as "text we drew," so it can ask whether our captions intrude into the platform's UI safe area.
That reading only holds if the two files are a pair — if this output was burned from this preview. Nothing verified that.
The only guard compared the number of sampled frames. Sampling is time-uniform, so two generations whose durations differ by 0.1s both yield exactly 60 samples. The guard was structurally incapable of noticing the thing it was nominally there to notice.
I wanted a statistical backstop: if the whole-frame difference between the two files is too large, they probably aren't a pair, so refuse to render a content verdict at all.
Exactly one episode in the repo had both files sitting on disk. I used it as my positive control.
| sample | median whole-frame abs diff |
|---|---|
| "correctly paired" episode | 19.51 |
| known-mismatched pair | 98.65 |
Threshold: 55.0. Zero overlap, a 33x gap. Two clusters, cleanly separated. Done.
That episode's preview file had an mtime nine hours later than its output — and later than the gate run that had already approved it. The preview on disk had been re-rendered after the burn. It was never a pair.
So 19.51 wasn't "what a pair looks like." It was "what a non-pair looks like." I had drawn my line using a negative sample as my positive control, and every statistic downstream inherited that.
I couldn't find a verified pair anywhere, so I made one: ran the burn and recorded a content-hash link between the preview and the output at the moment the output was produced. Then I measured again.
A genuine pair: 2.94. On a second episode, 3.37. Both land exactly on the background re-encode noise figure — 3 to 4 — that the same comment file had documented long before. The evidence had been sitting in my own repo disagreeing with me the whole time.
The real picture had three bands, not two:
| band | values | character |
|---|---|---|
| verified pair | 2.94, 3.37 | normal |
| same episode, different burn generation | 19.31, 19.51 | looks normal, produces false verdicts |
| entirely different content | 62.88, 97.68, 98.65 | obviously wrong |
The dangerous band is the middle one, and it sat below my threshold.
So the backstop passed the exact class of failure it existed to stop. It had already recorded a "212px safe-area violation" against an episode that was published. Re-run against the genuine pair, that episode is clean. The 212px blob covered a quarter of the frame — it was two unrelated regions merged by a comparison of two different pictures, not caption creep. A defect that never existed, filed against an artifact that was already live.
Band three gets caught by any threshold you pick; you don't need measurement to separate 3 from 98. The only place a threshold does real work is band two — and I had never measured band two at all. I drew a line between what I believed was band one and what I knew was band three, and the entire region where the check actually operates fell inside the pass zone.
Log-midpoint of the worst verified pair (3.37) and the best-known bad pair (19.31): sqrt(3.37 * 19.31) = 8.07, so 8.0. Symmetric in log space, 2.4x of headroom in each direction.
Verified pairs: n = 2. A 2.4x margin on two samples is not statistics, and I said so in the comment. What decided the direction wasn't confidence, it was cost asymmetry:
When the sample is too thin to tell you where the line goes, it can still tell you which side to be wrong on.
A threshold is a proxy for a fact you failed to record. The actual repair was to record it: at the end of every burn, write one line linking preview_sha256 to output_sha256, and have the gate use that line to prove the pairing instead of inferring it from pixels.
It has to be a content hash rather than a filename, because the last step of this pipeline is a human copying a candidate file over the canonical name. A filename-keyed link dies at that copy. A copy preserves bytes, so a hash-keyed one survives — and it did: immediately after promotion, the gate reported the pair as proven rather than estimated.
The statistical backstop still exists, but it's been demoted to a fallback for artifacts produced before the bookkeeping did.
The failure mode I want to flag hardest is the one in my second paragraph: zero overlap, a 33x gap. Separation metrics tell you two clusters differ. They say nothing about which cluster is which — and they are extremely good at supplying the feeling of having verified something.
Q. Why can a clean separation metric still be wrong?
A separation metric only proves that two clusters differ, not which cluster is which. The sample I had labelled as known good was actually a mismatched pair — its preview file had been re-rendered nine hours after the output — so the threshold was calibrated against a negative example mislabeled as a positive control, and the entire dangerous failure band ended up on the wrong side of the line.
Q. What are the three bands in the real distribution, and why does a two-sample comparison miss the dangerous one?
Verified pairs measure 2.94 and 3.37, same-episode-different-burn-generation pairs measure 19.31 and 19.51 and look normal while producing false verdicts, and entirely different content measures 62.88, 97.68, and 98.65 and is obviously wrong. A comparison built from only two samples contrasts the extremes — normal versus obviously wrong — and never surfaces the middle band, which is exactly the band a working threshold needs to catch.
Q. Why link the preview and the output by content hash instead of by filename?
The last step of the pipeline is a human copying a candidate file over the canonical name, and a filename-keyed link dies at that copy. A hash is computed from the bytes rather than the name, so a copy preserves it, and the gate can then prove the pairing instead of inferring it from pixel differences.