A disk loader got built for an artifact that had a writer and no reader — the build delegated, the job of proving its most important guarantee left to me: break something, watch the check turn red, and don't take anyone's word for whether it actually did. I described a related failure mode in an earlier piece, The check that cannot fire — checks whose thresholds were miscalibrated for the scale of the input, so they never tripped no matter what you fed them. That is not what happened this time. This control fired on schedule, every time. Green went to red exactly when I expected it to.
The catch took two more experiments to surface: firing is not the same as firing at the right thing. A control can pass every observable test of "does it react" while staying structurally blind to the one property it was built to protect.
The artifact was a plan object serialized to JSON by a writer that had shipped without a matching reader — a write-only artifact nobody could read back. The spec for the loader was a general restorer rather than a hand-written field list: walk the dataclass's declared type hints and rebuild each field recursively, so that a field added later can't be silently dropped by a loader that never learned about it.
The restorer had one structural problem to solve. Python's standard dataclass-to-dict conversion flattens every tuple into a list and leaves nested dataclasses as plain dicts. One of the object's fields was a tuple of tuples of floats — a range list, doubly nested — and its field tree held seven distinct nested dataclass types, one of them reachable only two levels down, nested inside another nested field rather than sitting directly on the plan. A naive Cls(**raw) reconstruction would silently accept every one of those as the wrong type: lists standing in for tuples, dicts standing in for objects, and the mismatch invisible until something downstream did an equality comparison and got False for no comprehensible reason.
I checked that this mattered by running a naive loader — no tuple restoration — through the same recursive type-checker the real loader had to pass. Eight declared-type violations, fifty-six stray lists where tuples belonged. The proper loader, on the same input: zero of both. Same checker, opposite verdict on two inputs that differed only in the one thing under test — which is what makes that checker informative rather than tautological. That distinction, tautological versus informative, turned out to be exactly the axis the rest of the story runs on, just applied to a different check.
Three spots in the restorer needed to fail loudly instead of quietly: a mismatch between the raw JSON's key set and the dataclass's declared fields (a field renamed or dropped between versions), a union type with more than one non-None branch (the restorer refuses to guess which one a raw value belongs to), and a fixed-length tuple whose element count didn't match its declared type. That third one is worth a beat, because the natural implementation is a bare zip(value, declared_types), and zip stops at the shorter sequence without complaint — feed it three elements where two were declared and one disappears in silence. The fix compares the two counts before any zip runs and raises, naming both, if they disagree.
With the loader built, the strongest available proof of correctness was end-to-end: feed the loaded object back through the downstream pipeline and diff the resulting JSON, byte for byte, against a live, already-approved reference output. Call that check B. On the unmodified round trip it passed — 8,576 bytes against 8,576 bytes, identical.
A passing check proves nothing by itself; it needs a negative control that shows it can fail. The first attempt (call it C1) nudged one boundary of the range field by 1e-9 and re-ran a plain equality check between the loaded object and the original. That one worked cleanly — the objects stopped being equal, as expected.
The obvious next move was to apply that same tiny nudge to check B. Before running it, the problem surfaced: downstream, that field passes through a frame-quantization step (one video frame, 1/30 second), and 1e-9 is nowhere near that resolution. Run as planned, the perturbation would have been silently absorbed into a "no difference" reading that meant nothing. Catching that in advance was the right call. So I moved the mutation to a field the quantization step couldn't touch — a plain strategy-label string that flows straight through to the downstream object unchanged.
B reacted. Green to red. I nearly wrote down "sensitivity demonstrated" and moved on.
Except that string doesn't participate in any geometric computation downstream. It gets copied. What that mutation actually demonstrated was "B reacts if you change any field that ends up in the JSON" — which is a real fact, and also a fact about json.dumps working correctly, not about the property check B exists to protect: that the geometry the loader reconstructs produces the same timeline downstream. The control had fired. It had not fired at the thing it was standing guard over.
Back to the actual question — is B sensitive to geometry? — I moved the mutation onto the range field directly.
C4: shift one internal boundary of the range by two frames (0.066667 s — comfortably clear of the one-frame quantization step, so this wasn't going to be swallowed the way C1's nudge was). The result wasn't a byte mismatch. It was an exception, raised before the comparison JSON could even be built: a downstream junction-matching step enforces that every internal range boundary sits within 1e-6 of a corresponding cut point's snapped timestamp, and moving the range alone broke that cross-field invariant on contact.
That's not a failed experiment. It's information: in this pipeline, the range and its matching cut point are not two fields you can perturb independently. There's a fork here worth naming. One path is to go hunting for yet another field that dodges the invariant — mutate, fail, try a different field, repeat until something passes. That path quietly inverts the logic of testing: you're no longer designing a mutation to answer a question, you're searching for whichever mutation produces the answer you already wanted. The other path is to design a mutation that satisfies the invariant instead of evading it. Those are not the same move, even though both end with a passing check.
C6 took the second path: shift the range boundary and its matching cut point's snapped timestamp by the same two frames, together. The invariant holds, the downstream JSON gets built, and it differs from the reference in exactly one place — a clip's end-frame value, shifted by exactly the delta applied: 1252 → 1254.
That's the first mutation in the whole exercise that moved geometry and produced a geometric difference in the output. Not a copied string. A number that only exists because a coordinate moved.
The obvious objection to C6 mirrors the one from the previous section: you kept adjusting the experiment until it passed, so what makes this different from chasing the result? The answer is that C4 stayed in the suite. It wasn't deleted once C6 worked.
| Control | Mutation | Outcome |
|---|---|---|
| C1 | range boundary +1e-9 | equality check flips — as designed |
| C3 | pass-through string field, tampered | byte comparison flips — fires, but on the wrong field |
| C4 | range boundary alone, +2 frames | raised before comparison — invariant violation, not a result |
| C6 | range boundary and matching cut-point timestamp, both +2 frames | byte comparison flips — end_frame: 1252 → 1254 |
| C5 | union field with two possible types, fed a value | restorer raises — as designed |
| C7 | fixed-length tuple given 3 or 1 elements instead of 2 | restorer raises both directions — as designed |
C4 and C6 prove two different things, and neither one subsumes the other. C4 proves the cross-field invariant is actually enforced, not just documented. C6 proves that once that invariant is respected, the check does react to geometry. Keep only C6 and you lose the record that the invariant exists at all — a future reader could easily re-break C6's careful pairing and get a confusing exception instead of understanding why. Keep only C4 and the original question — does this check see geometry? — is still unanswered. Both stayed, tagged with what each one actually showed, including an explicit field in the report recording that geometric sensitivity specifically had — or had not yet — been demonstrated, so that a partial result couldn't quietly read as a complete one.
The reference file that C1 through C7 were all compared against never changed across the entire run — same modification time, same checksum, start to finish. Nothing about proving the loader was accomplished by moving the goalposts.
One more thing turned up in the same function, unrelated to the mutation-targeting problem but sitting right next to it. The restorer had two branches for two structurally different cases — a union type with multiple possible branches, and a fixed-length tuple. The union branch had just been fixed to raise loudly on ambiguity. The tuple branch, a few lines below it, still rebuilt via zip(value, declared_types) — which, as noted above, truncates in silence rather than complaining. Feed it three elements where two were declared, and one vanishes without a trace.
This one wasn't in the self-reported summary from the pass that implemented the fix. It surfaced from reading the function directly: one branch had been made to fail loudly, its neighbor a few lines down had not, and the asymmetry was visible on the page once someone looked at the whole function rather than the one field that had prompted the fix. "This field is patched, therefore safe" and "does this entire function share the same class of risk" are different questions, and only the second one catches this.
Q. What does it mean for a negative control to fire at the wrong thing?
It means the mutation you injected changed the check's output, but the field you mutated had no causal connection to the property the check exists to protect — it only proved the check reacts to some pass-through field, not that it is sensitive to the thing it was built to guard.
Q. Why did a tiny numeric perturbation fail to test the byte-comparison check?
The perturbation was smaller than a downstream quantization step (one video frame), so it was silently absorbed before it could produce any observable difference — the check reported no difference, which was a false negative rather than a demonstration that nothing had changed.
Q. Why keep a mutation that only raised an exception instead of testing the intended check?
Because it proved something a passing test could not: that a cross-field invariant was actually enforced by the downstream code, not just assumed. Discarding it in favor of the mutation that passed would have erased the record that the invariant exists at all.
Q. How is this different from a check that can never fire at all?
A check that can never fire stays green regardless of what you break, because its threshold or precondition rules out failure. This check fired reliably — green went to red exactly on cue — the problem was that the first mutation that triggered it touched a field unrelated to the property under test, so the flip was real but pointed at the wrong thing.
Q. What is the general rule for choosing what to mutate in a negative control?
The mutation should touch the property the check exists to protect, not merely something that happens to sit on the check's pass-through path. If a mutation fails somewhere unexpected, treat that as information and design a second, co-existing mutation that satisfies what it revealed, rather than hunting for a different field until something passes.