A check can be entirely correct and still miss what it was built to catch, if it's answering a narrower question than the one you think you're asking. I ran into this three times, on the same check, on the same video-rendering pipeline, before I stopped trusting a pass and started asking what population it had actually looked at.
The check runs before a clip ships: it scans finished frames for text overlapping other text and reports how many timestamps collide. Three times, across two clips, it reported zero while text was visibly, unreadably overlapping on screen. The check never lied. The closest name I already had for this shape, verification tools don't report their blind spots, undersells it: the tool printed an accurate number that was still the wrong thing to trust.
The first report came back on a freshly rendered clip: zero overlapping timestamps. The frames told a different story — a dollar-value label sat exactly on top of a caption burned into the video, unreadable. Three figures in the clip were hidden the same way — $4.404 billion, $4.147 billion, $23.769 billion — each under the same caption band the moment it appeared.
The check compares text objects the rendering scene itself draws. The caption isn't one of those: it gets burned in during a separate pass, after the scene renders, and never exists in the scene's own coordinate space. Zero was true — nothing the scene drew overlapped anything else the scene drew. Rendering the scene alone hides this too — the caption band is empty in isolation — and the defect only exists once both pieces are assembled.
I didn't estimate where the caption sat. I pixel-diffed matching frames from the finished video against the pre-caption render, at three timestamps: a one-line caption occupies a band from about y = -4.178 to y = -3.585. The old placement for the value label sat almost exactly in the middle of it.
The fix moved the label into a gap between two other fixed elements — the top of a bar and a tick label above it — space nothing else could structurally reach. I hardcoded the boundary as a named constant, BURNED_CAPTION_TOP_Y, commented as measured rather than assumed, and asserted against it. Nothing here replaced opening a rendered frame and looking.
Same check, same clip, right after that fix — zero again, and found only after I'd already called the first one done.
Frames near the midpoint showed two adjacent year labels ghosting into each other — one year's numbers still visible while the next faded in, both readable at once, so neither was. Every transition point did the same, adding up to roughly a fifth of the clip's runtime — about 9.4 of 47.4 seconds.
This time the blind spot was a pairing, not a missing layer. The check's definition of "overlap" was a white pixel cluster intersecting an amber cluster — two named categories. The ghosting was white text fading into white text: the same category transitioning into itself, a combination the definition never included. Worse, during the fade the two texts sit at the exact same coordinates, and a bounding-box check doesn't see two boxes in an identical position as an intersection — it sees one box. A slight misalignment would have registered; a perfect, total overlap did not.
My first replacement instrument was wrong, caught only because I widened the positive control before trusting it. Ghosting is text rendered semi-transparent, so I measured the fraction of "mid-tone" pixels. An old, ghosting frame came back 88.8% mid-tone; a hard-swapped frame came back 29.0% — a 3x gap. Widening the "known normal" sample broke it: frames unambiguously fine by eye scored anywhere from 29.4% to 90.0%. The metric was measuring color, not transparency — red lettering elsewhere in the clip has a luminance around 112, dead center of the 60–195 range I'd called "mid-tone," so any red text scored as ghosting.
The working instrument measured the definition instead of a proxy: ghosting is a region that changes gradually across more than one frame; a legitimate swap is a one-frame event. Independent of color, I measured the length of continuous frame-to-frame change:
| version | change events | longest run | largest single-frame jump |
|---|---|---|---|
| old (crossfade) | 34 | 19 frames | 6.8 |
| new (hard swap) | 18 | 1 frame | 31.3 |
19 and 1 land on opposite sides of any threshold. The largest jump is lower on the broken clip, backwards from expectation, because a crossfade spreads change thinly instead of concentrating it.
The fix: text never crossfades, only bars animate. All 7 crossfade calls became an outright swap — remove old text, add new — with freed time absorbed by an equal-length wait so clip length held: 47.36s to 47.33s, against a narration track needing 45.99s, with 1.34s to spare.
Later the same day, I pointed the frame-change-length instrument from case two at a different clip I believed already repaired. It came back with a longest run of 23 frames — worse than the number that first flagged a problem. A repair can't make a measurement worse, so either the fix hadn't landed, or the instrument was wrong.
It was the instrument. The 23 frames were two legitimate entrance animations — a panel and a callout fading onto the screen, which changes a text region gradually for a different, correct reason. A discriminator for that — content at both ends of a change, versus coming up from nothing — brought the number to 7. Opening those frames too found a second legitimate cause: a background panel's opacity ramping up behind static text.
Three discriminators, three failures to separate the classes:
| discriminator | on the real defect | on the false positive | separates them? |
|---|---|---|---|
| lowest ink level during the change | 0.94 | 0.96 / 1.00 | no — ranges overlap |
| largest single frame-to-frame jump | low | low | no — inverted, same surprise as case two |
| mid-tone pixel fraction | — | — | no — the same color-proxy that already failed once, reused |
The third failure repeated a mistake I'd already made and written down earlier in the same investigation. Three discriminators landing on the same overlap is evidence the channel lacks the information, not that a fourth would work.
So I changed layers and read the source that generated the frames. One line explained everything the pixels had been ambiguous about: a call whose entire purpose is to superimpose two text objects at the same position while one fades out and the other in. Ambiguous in pixels, unambiguous in one line of source.
It wasn't isolated — it was documented as the correct way to change an on-screen number, in the header comment of three separate scene files, for a real reason: the objects it replaced default to a class that shells out to a LaTeX binary and throws a hard error without it. But "there's no LaTeX" and "so cross-fade the text" got written as one instruction and copied as if the second followed from the first. It doesn't — swapping the old object out and the new one in, no fade, satisfies the same constraint without ever putting two objects on screen at once.
The impact wasn't theoretical: a clip that had already shipped had the identical pattern, unreadable for about half a second, confirmed by opening the frame.
I fixed all 12 call sites carrying the convention and re-rendered. Then, instead of trusting the improved number, I looked at the same segment again. The numbers no longer overlapped each other. One label was still sitting on a caption underneath it — nothing to do with any crossfade.
The second cause was separate: a coordinate named HERO_Y that part of the scene converges toward was exactly where an unrelated label stood. An earlier change had made that label persist for the clip's full length instead of disappearing, conflicting with everything sharing its coordinate. I'd already patched that conflict once, in one beat; a second beat using the same coordinate had never been touched. The scene was already calling the overlap-assertion helpers in eight other places — and not in this one. That is the more dangerous shape: a list of call sites long enough that scanning it reads as "this scene checks for overlap," so nobody counts the gaps.
Measuring the same segment across all three states made the fix legible:
| version | longest change-run | clip length |
|---|---|---|
| originally shipped | 19 frames | 1461 frames |
| crossfades fixed | 8 frames | 1461 frames |
| + coordinate collision fixed | 4 frames | 1461 frames |
Clip length never moved, because the swap consumed no animation time and the collision fix layered a fade onto an existing animation instead of extending it.
One more layer sat underneath, and it was mine. The swap helper I'd written did remove(old) then add(new). One caller passes a group, not a single text object — two labels added individually, later bundled for convenience. Rather than reason about the framework's semantics I asked it, in four lines:
add(a); add(b); remove(Group(a, b)) → both are still on screen
Removing a group doesn't remove members added individually. The new text would have been drawn on top of the old one, at the same coordinate — the helper written to eliminate the overlap reproducing it exactly, at five call sites across four files. I caught it before rendering by counting which callers pass a group and testing the assumption instead of trusting it. Code that fixes things is still code, and rarely gets the checks the code it fixes does.
None of these three zeroes were false. Each was a true statement about a smaller world than the one that mattered, and the only way to find the gap was to stop reading the number and go open the frame.
Q. Why did the same overlap check report zero problems three separate times?
Each time it was telling the truth about a smaller population than the one that mattered. It only compared objects the rendering scene itself drew, so a caption added afterward was never in scope. Then it only compared two specific color clusters against each other, so same-color ghosting was never in scope either. Once both were fixed, a pixel-only instrument ran out of information, and the real defect turned out to be one line of source, not a pattern in pixels.
Q. Why is a check that only compares two named categories dangerous?
Because damage that happens entirely inside one category is invisible to it by construction. In this case, a bounding-box overlap check also couldn't tell two identically-positioned objects from a single object, so the worst version of the defect — a perfect, total overlap — was the least visible one to the check.
Q. What should you do when a detector won't converge no matter how you tune it?
Stop tuning it and change which layer you're looking at. After three separate pixel-based discriminators failed to separate a real defect from legitimate animations, reading the source that generated the frames explained the ambiguity in one line — a text crossfade call that pixels alone couldn't distinguish from other causes of gradual change.