I've written before about checks that pass for reasons that have nothing to do with what they claim to verify: a dead-code detector that flagged six constants that turned out to be unenforced specifications, not dead code, a detector whose zero false positives were indistinguishable from a stub's until a positive control showed it could actually fire, which it did eight times, a rule that evaluated cleanly because the field it read had no producer anywhere in the codebase and was always null. This one is a different shape from all three. The check itself was correct on the day I wrote it — it told articles apart from non-articles without any ambiguity. What broke wasn't the logic. It was the definition of the population the logic ran over, and it broke the same day.
I was building a check for a publishing pipeline: does every article on a hub site actually carry a conversion surface, or did one slip through empty? The check needed to walk every page and decide which ones counted as "articles" versus which ones were something else — an index page, a lead magnet, a policy page — that shouldn't be held to the same rule. My first pass did what most first passes do: a denylist of the slugs I already knew weren't articles.
_EXCLUDED_PAGE_SLUGS = frozenset({"privacy", "terms", "disclosure"})
...
return not (_EXCLUDED_PAGE_SLUGS & set(relative.parts)) # exact matchThree reasonable-looking strings. It was also already wrong, and the reason has nothing to do with which three strings they were — it would have been just as wrong with thirty.
Later that same day I published one more static page: privacy-policy. The set comparison is an exact match, and privacy-policy is not privacy. The page sailed straight through the denylist, got counted as an article, and the check reported [PARTIAL] 13/14 — a false positive claiming a real article was missing a conversion surface it never needed, because it wasn't an article in the first place.
That's the more forgiving of the two directions a stale exclusion list can fail in. Under-exclusion produces false positives: a legitimate page gets permanently flagged, the check turns into background noise, and the next time it fires about something real, nobody's watching anymore because it already cried wolf. The other direction is worse. Over-exclusion produces false negatives — something that should have been checked quietly stops being checked, and nothing in the output says so. A list that's too narrow announces itself with noise. A list that's too broad doesn't announce anything at all.
The part that made me stop treating this as a one-off typo: the identical defect already existed, independently, in a deploy script written in a different language against a different file:
case "$slug" in lead-magnets|privacy) continue ;; esac # counts privacy-policy as an article, deploy haltsNobody copied this bug from the Python check into the shell script, or the other way around. Two separate implementations of "which pages don't count as articles" reached for the same shape of answer: write down the names you currently know about. Both were correct on the day they were written. Both were wrong within that same day, for the same reason — a name list has no way to know that a name it hasn't seen yet belongs to the same category as the names it does know.
The fix wasn't a longer list, or a fuzzy match that would eventually hit the same wall with a different string. It was to stop defining the population by name at all. The site's renderer already leaves a mark on every real article and on nothing else: an application/ld+json structured-data block. I checked this empirically before trusting it — every one of the 13 real articles carried the block, and every one of the 9 non-articles (the index page, the lead magnets, both policy pages) didn't. Thirteen for thirteen, nine for nine. That split is the population definition. It doesn't need to know that privacy-policy exists as a string; it only needs to know whether a given page rendered the block, which is a fact about the page rather than a fact about whoever last updated the list.
Once the check reads the marker instead of the name, adding a tenth non-article page the next morning requires touching neither the Python file nor the shell script. The page either renders the block or it doesn't, and the check answers correctly either way — for a page that didn't exist when the check was written.
The move is to flip the question. "What should this check skip?" invites a list, and every list has a day it stops matching the world. "What does a real member of this population look like, from the object's own output?" invites a marker, and a marker doesn't go stale just because the set of instances grows.
Four things make that move stick instead of just sounding nicer:
This isn't a blanket argument against ever writing a list. It fails as advice in a domain where a marker can't attach reliably to every instance — where some real members of the population simply don't carry any distinguishing signal. There, switching to inclusion-by-marker just trades a visible false-positive problem for an invisible false-negative one, which is the worse trade. In that situation the right move is to keep the list and still hold onto the third rule above: expose the excluded count, so staleness is at least visible as a number instead of a silent gap.
It's also not worth the trouble when the excluded population is small and provably fixed — five or fewer items that can't grow because of something structural about the domain, not just "I don't expect more right now." A short, closed list is more readable than a marker nobody else on the team knows to look for. The argument here is about lists that grow by nature — page types, file categories, route names — not about enumerations of a genuinely finite, closed set.
The same day produced a second version of this failure in a completely different outfit. A regression test pinned only the sha256 hash of a build artifact — no copy of the artifact itself, just its checksum. The artifact it was supposed to compare against had quietly stopped existing, through three unrelated paths at once: the build directory was gitignored, the deployment process used an orphan commit plus a force push, and the live copy got overwritten on every redeploy. Each of those is individually a reasonable choice. Together they meant that the one time I actually needed to ask "is this difference the one I intended, or a regression," there was nothing left to diff against — only a hash with no file behind it.
It isn't the same mechanism as the exclusion list — nothing here is a name versus a marker — but it's the same family of mistake: a check that quietly stopped verifying anything, because the thing it depended on could disappear without the check noticing. The fix was to keep the actual golden file in the repository, so a failure produces a real, readable diff, and to keep the hash only as a secondary anchor for noticing if the golden file itself gets silently edited.
The claim is specific enough to falsify. If a codebase's candidate markers turn out not to attach stably to every instance of the intended population — if genuine members sometimes render without the marker for legitimate reasons — then converting a name list into a marker check trades a visible failure mode for an invisible one, and that's a worse trade, not a better one. And if the excluded population genuinely can't grow past a handful of fixed names, the marker is solving a problem that was never going to occur. The 13/13 and 9/9 split is what made the marker trustworthy in this case; without a split that clean, the right fix is a list with its excluded count printed on every run, not a marker adopted on faith.
Because a name list only recognizes strings it was already given. In this case privacy and privacy-policy are different strings under an exact match, so a new page describing itself with the second name sails straight through a list built for the first. The list isn't wrong about the three names it holds — it's wrong about every name it doesn't, including ones that are one word longer than a name it already has.
A structural marker is a signal the object itself produces — in this case a JSON-LD block the site's renderer only writes onto real articles — rather than a label a maintainer has to remember to add. Extending a list still requires someone to notice a new page exists and update the list the same day; reading a marker requires the page to answer a yes/no question about itself, so a page that didn't exist when the check was written still gets classified correctly.
A false positive is loud: a legitimate page gets flagged, and eventually someone notices the check is wrong, though by then it may already be ignored as noise. A false negative is silent: something that should have been checked simply stops being checked, and nothing in the output changes to signal that a page dropped out of coverage. The loud failure is annoying; the quiet one is the one that lets a real defect through unnoticed.
Two cases: when no marker attaches reliably to every real member of the population, which would just swap a visible false-positive problem for an invisible false-negative one; and when the excluded population is small and provably fixed — five or fewer items that structurally cannot grow — where a short list is more readable than a marker nobody else knows to check.
Print the count of excluded items on every run, next to the total population size. Say a list used to exclude 3 items out of a population of 10 and now excludes 3 out of 14 — that kind of shift is a visible signal that something changed underneath the check, even before you know whether that something was a false positive or a false negative. The same logging discipline applies after switching to a structural marker, since its own failure mode — an item whose marker silently fails to render — is exactly as invisible as an over-broad list.