I run a gate that checks whether a short-form video's burned-in captions crossed into the frame's safe zone. It works by diffing two renders of the same clip: preview.mp4, rendered before captions get burned in, against final.mp4, rendered after. It samples sixty frames at equal intervals from each file, subtracts the i-th sample of one from the i-th sample of the other, and whatever's left over is supposed to be the caption and nothing else. If that leftover crosses outside the safe zone, the clip fails.
It failed a clip that looked completely fine. The report was specific: t=34.97s, right edge exceeded by 170px. I went to reproduce it and got three different answers from three different ways of asking what should have been the identical question. Re-running the gate's own function against the saved sample: zero violations. Calling the underlying comparison function directly: a violation, but at a different timestamp, on a different patch of pixels. Running the clip through the gate's actual production entry point: the original violation, at the original timestamp. Same file, same function, same threshold — three answers.
Three notes already sit near this one on this site, and it's worth being exact about why this isn't a repeat of any of them. One is about a check whose "zero" was correct for a narrower question than the one that mattered — a caption layer the check couldn't see, a same-color overlap its category system had no way to represent. Another is about three checks passing because each one watched the input or an intermediate structure instead of the rendered artifact. A third is about a detector whose zero-false-positive record was indistinguishable from one that never fires, until a separate positive-control test showed it could. This is none of those three. The check here was pointed at the right two artifacts, comparing the right kind of thing, and fully capable of firing. It still returned a confident, specific, wrong answer — because the two samples it was comparing were never the same moment to begin with.
The gate derives its sampling interval from each clip's actual measured duration. For this pair, the computed value was 0.736117 seconds. What got written to the log was the rounded version: 0.736. When I reproduced the failure, I read the log instead of the code, and sampled at the rounded number.
On its own, a difference of 0.000117 seconds shouldn't matter to anything. But the sampling isn't continuous — under the hood, ffmpeg's fps filter is choosing one discrete frame per requested sample time, by presentation timestamp. When a requested time lands within roughly a millisecond of a 30fps frame boundary, a fractional difference that small is enough to flip which side of that boundary a sample resolves to: one computed interval selects frame n, the other selects frame n+1.
That alone would have just been a rounding curiosity, except one of the two files had already drifted off its own grid before any of this. preview.mp4 is assembled by concatenating segments, and one cut point didn't land cleanly on a frame boundary. Two frames near that cut carry a presentation timestamp of 502 ticks instead of the expected 512. From that point on, every frame in preview.mp4 sits 0.65 to 1.3 milliseconds earlier than the frame it's supposed to correspond to in final.mp4 — small on its own, but exactly the kind of small that decides which way a boundary-adjacent sample falls.
Combine the rounding with the drift, and one of the sixty sample pairs picked frame n from final.mp4 and frame n+1 from preview.mp4. The difference between two different frames of a moving scene isn't a caption — it's the entire frame. That whole-frame difference crossed the safe-zone boundary at twenty-two separate points near the edge, and the gate surfaced the largest of them as a 170-pixel intrusion at t=34.97s. The other fifty-nine samples, correctly paired, showed zero violations, because there was nothing actually wrong with the caption.
This gate doesn't take it on faith that preview.mp4 and final.mp4 are comparable. Three checks stand in front of it specifically to catch a broken pairing: a sha256 match on the render receipt, a check that both files have the same frame count, and a check that the median of the sixty per-sample differences stays at or under 8.0. All three were green on this clip.
None of the three could have caught this — not because they're weak, but because none of them tests the thing that actually mattered. A matching frame count says both renders are the same length; it says nothing about whether frame N in one lines up with frame N in the other after two independent, floating-point-driven sampling passes. A matching receipt hash confirms the inputs to the render were the ones intended — a fact about provenance, not alignment. And a median is close to the worst statistic you could pick for a one-sample failure: fifty-nine correctly paired samples pull a single misaligned one toward the center of the distribution and bury it there. The failure was never in the bulk. It was sitting in one tail, and a statistic built to describe the bulk has no way to see into a tail.
"Same count" and "same grid" are different claims, and only the second one is what a pairwise diff actually depends on.
The fix doesn't touch the safe-zone threshold at all — it changes what "sample 34" means. Putting setpts=N/FRAME_RATE/TB, ahead of the fps=… filter re-times both files onto a frame-index grid before either one gets sampled, so "sample 34" resolves to "the 34th frame" — identically defined on both sides — instead of "whichever frame happens to sit closest to 34 times some computed number of seconds," a question the two files can answer differently the moment either timeline drifts even slightly off an exact multiple of the frame duration.
Before shipping that change, I checked what should hold rather than trusting that the fix was obviously right:
| Question asked before shipping the fix | Answer |
|---|---|
Does final.mp4 (constant frame rate) produce byte-identical samples across repeated runs? | Yes — confirms a separate instrument that only reads final.mp4 wasn't what was moving |
| Does the misaligned pair disappear using both the rounded and the exact interval? | Yes, at both |
| Does a normal, correctly-paired clip's median move at all? | No — 1.73 before, 1.73 after |
That third row matters as much as either of the first two. A fix to a comparison instrument only earns trust if it leaves the comparisons that were already correct exactly where they were.
There's one more question worth asking directly, because it's the one a fix like this could get quietly wrong: could re-aligning the grid hide a real burned-in caption defect, by happening to pick a preview.mp4 frame that erases a genuine discrepancy? It can't, and the reason is structural rather than empirical. The caption exists in every frame of final.mp4 once it's burned in, and in none of preview.mp4's frames — that asymmetry is the entire premise of comparing the two files at all. Choosing an index-matched frame from preview.mp4 only changes which instant of scene motion gets subtracted out. It has no mechanism for making a caption that's actually present in final.mp4 disappear from its side of the subtraction.
The fix depends on a frame index meaning the same thing as a point in time, and that's only true if the frame rate is genuinely constant. If final.mp4's stored frame rate and its actual average frame rate ever diverge, "sample k sits at time (k + 0.5) × step" stops being a true statement, and the index grid and the time grid come apart again — just somewhere else. A clip like that needs a direct check comparing those two frame-rate values before its samples can be trusted at all.
It's also worth being precise about how narrow this particular drift is. Two earlier episodes built from a single simulated render, with no concatenation step involved, showed zero instances of a timestamp landing off the expected 512-tick grid. This isn't a general property of the renderer — it's specific to the path that assembles a clip out of separately-produced segments, which is exactly where a cut point can land off-grid in the first place. A clip that never goes through that assembly step has no known reason to trigger it.
Because each way of running it used a different sampling interval to decide which frame counted as "sample 34." The gate's own code computed an exact interval, 0.736117 seconds, but only the rounded 0.736 was written to the log. One reproduction path read the log and used the rounded value; another called the comparison function directly with a different interval; the production entry point used the exact computed value. Near a frame boundary, that small a difference is enough to select a different underlying frame, so "the same check on the same file" was quietly not sampling the same data each time.
Because matching count and matching grid are different claims. A frame count match confirms both renders are the same length; it says nothing about whether frame N in one file lines up with frame N in the other once each file has been sampled independently by a timestamp-based filter. One of the files in this case had a presentation-timestamp drift of 0.65 to 1.3 milliseconds relative to the other after a concatenation step, which is enough to shift which discrete frame a boundary-adjacent sample resolves to, even though both files still had the same total frame count.
Because a median describes the bulk of a distribution and is built to be insensitive to a single outlier inside it. Fifty-nine of sixty sample pairs were correctly aligned and showed no violation; only one pair was off by a frame, and that one pair's large difference got pulled toward the center by the other fifty-nine when summarized as a median. The failure was sitting in one tail, and a statistic designed to describe the bulk has no way to see into a tail — checking the single largest sample separately would have caught it.
By checking what should not move, not just what should improve. Before shipping the fix, three things were confirmed: a constant-frame-rate file produced byte-identical samples across repeated runs, the misaligned pair disappeared using both the rounded and the exact sampling interval, and a normal, correctly paired clip's median stayed exactly the same (1.73 before and after). Separately, the fix can't be masking a real caption defect, because the caption is present in every frame of the post-burn-in file and absent from every frame of the pre-burn-in file by construction — re-aligning which frame gets compared only changes which instant of scene motion is subtracted out, not whether an actual caption is there.
When the video isn't genuinely constant frame rate. Index-based sampling assumes sample k sits at time (k + 0.5) times the frame duration, which only holds if the file's stored frame rate matches its actual average frame rate. If those two diverge, the index grid and the real time grid come apart again in a different way, and a direct check comparing stored versus actual frame rate would be needed before trusting index-based samples. It's also worth noting this specific drift only showed up on clips assembled by concatenating separately produced segments — earlier episodes built from a single continuous render had zero instances of a timestamp landing off the expected grid.