must_not_have governs the screen — and against real contracts it had fired eight times and been wrong all eight. Twenty tests went red and got fixed as bookkeeping. Four tests stayed green, and all four had quietly stopped asserting anything: their fixtures no longer triggered the rule they were named after, so the checks had collapsed into comparisons like [] == [].I deleted a validation rule from a video pipeline that had never once been right. Twenty tests turned red immediately, and fixing them was mechanical — one assertion at a time. Then I went looking for the tests that hadn't turned red, and found four that had quietly stopped testing anything at all.
The twenty red tests were the safe ones. Failing loudly, they told me exactly what they were coupled to. The four quiet ones told me nothing — they just kept passing, for reasons that had stopped having anything to do with the feature they were named after.
The pipeline builds video shots. Each shot carries a contract: a must_have list, a must_not_have list, and serves_line — the sentence of narration the shot exists to depict. One validation rule checked a shot's contract against itself, and it had two clauses.
The first was lexical: if the narration and a forbidden item share a content word — the line says "three doors," the contract forbids "doors" — nothing can legally be drawn, so it's a hard failure. Reasonable.
The second banned numeral class: if a forbidden item rules out numeric notation (number, digit, numeral) and the narration states a number, that also counted as a conflict.
The second clause was a category error. serves_line is narration — it goes in your ears. must_not_have governs what's on screen — it goes in your eyes. A shot can say "twenty thousand games" out loud while drawing no digits at all; that isn't a contradiction, it's just house style. Run against real contracts, this clause fired eight times. It was wrong eight times.
So I deleted it. That part really was easy.
The twenty failures were exactly what you'd expect from removing a rule: assertions that a numeral-only input produces a conflict, now producing none. Delete the assertion, rewrite it against the clause that survived, move on. Bookkeeping.
But a deleted rule doesn't only break the tests that assert on it directly. It rewrites the meaning of every fixture that was chosen because it happened to trigger that rule — without touching the fixture's type, shape, or name. Nothing in the toolchain reports this. The test still runs. It still passes. It has simply stopped measuring anything.
I had a module-level constant holding a real narration line whose only conflict ran through the deleted clause. Four tests were built on it:
# "declaring metaphor exempts the conflicts"
conflicts = find_conflicts(LINE, FORBIDDEN, relation="metaphor")
assert conflicts == []That test passed before the deletion and passed after it, and the two green checkmarks meant opposite things. Before, it proved the metaphor exemption actually suppressed a real conflict. After, there was no conflict left to exempt — the assertion had quietly become [] == []. It was green because the input had gone inert, not because the exemption still worked.
The same rot had reached three more:
relation argument behaves like literal — it was comparing two empty lists.metaphor — it was empty for every relation, metaphor included, so the boundary the test was named for no longer existed.literal contract still hard-fails. This one did go red, with DID NOT RAISE — and it's the only reason I opened this group of tests at all.Three of the four would have sat there passing indefinitely, reading as coverage, in a file that no longer tested the thing its name promised.
The audit itself is mechanical, and took about five minutes once I knew to run it:
The second step is the one that takes deliberate effort, because the test runner has already told you those files are fine. You're not chasing a failure. You're second-guessing a pass.
Check the spec before you check reality. I'd falsified the numeral clause empirically — prototypes, sweeps across real contracts, a false-positive tally that came out eight for eight. Only afterward did I read the domain glossary, which had defined the hard-failure rule the whole time as "must_not_have forbids a core noun of serves_line." A core noun. Numerals were never in the specification. The clause was an expansion nobody had asked for, and the cheapest possible refutation had been sitting in a glossary file the entire time.
Measure the replacement before you ship it. My own earlier notes had proposed where the true positives really lived: contradictions between must_have and must_not_have on the same shot. It sounded right, so I prototyped it and ran it across the real corpus. Four firings, all four false positives. The two lists routinely describe the same subject from opposite sides by design — must_have: "the door remaining closed" and must_not_have: "door fully opening" share the word door while fully agreeing with each other. Plain word overlap can't tell contradiction from paraphrase. So I deleted a rule and shipped no replacement for it, and wrote down why, so the next person doesn't rebuild it.
Deleting code silently rewrites the semantics of the data your tests feed it. A test suite only measures what its inputs manage to provoke, and removing a rule can leave an input provoking nothing at all, while every signature — the fixture's type, shape, and name — stays exactly as valid-looking as before.
So the completion criterion for a deletion isn't "the suite is green." It's narrower:
For every test that touched the deleted rule, does its input still reach live code?
A red test is a question the suite is asking you. A test that went quiet is one that stopped asking — and silence reads exactly like success.
This isn't the only way a green suite has lied to me, and it's worth being precise about which failure this is:
Twenty red tests were never the risk. The four quiet ones were — and the only reason to go looking for them is that nothing in the toolchain will ever ask you to.
The test's fixture was chosen specifically because it triggered the rule you just removed. Once the rule is gone, that input triggers nothing, so an assertion like "conflicts == []" is still syntactically true — it just compares two empty results instead of proving an exemption or a conflict actually fired. The test keeps passing for a reason that has nothing to do with the behavior its name describes.
Grep every use of any fixture the deleted rule depended on, then open all of them — the ones that just went red included, not only the ones still passing. For each one, ask a single question: does this input still trigger anything? If the answer is no, the test is asserting a tautology and needs a fixture that exercises a rule that's still alive. The audit is mechanical and takes minutes; the effort is in deciding to run it on files the runner already called fine.
A red test tells you, immediately and specifically, what it's coupled to — you cannot ship past it without looking at it. A test that goes quiet gives you no signal at all: it reports the same green it always has, so nothing in the toolchain distinguishes "still verifying the feature" from "comparing two empty lists." The failure mode isn't the assertion being wrong; it's the assertion becoming irrelevant while looking unchanged.
A wiring gap means the tested code path is unreachable from production — the function works in isolation but nothing live ever calls it. A vacuous-green test is reachable and does run; the code under test executes every time. What's gone is the ability of its input to provoke the behavior being asserted, because a rule the input relied on was deleted out from under it. One is a reachability problem, the other is a semantic one.
Empirical falsification works but can cost real effort — prototypes, sweeps across real data, a false-positive tally. In this case that whole process converged on exactly the answer a domain glossary had stated the entire time: the hard-failure rule only covers a core noun, and numerals were never part of the specification. Reading the spec first would have found the same defect for the cost of opening one file.