I have a script that turns an evaluation run into a markdown report. Numbers come from f-strings. Sentences come from string literals sitting in the same function. That split is so ordinary I never looked at it — and it is the whole defect.
A literal exists independently of the data. When the data changes, the literal stays. It compiles. It renders. Nothing complains. The number two lines above it moves, and the sentence does not.
An adversarial review of my own report found six prose claims that its own data contradicted. What surprised me was that they had not failed the same way:
The common structure is not rot. It is that a literal has no dependency on the thing it describes. Drift is one way that hurts you. Being wrong on day one is another. Neither is detectable by reading the sentence, because the sentence always reads fine.
My first instinct was: replace every literal with an expression that reads from the source. That is the right instinct and it does not fully work.
Of the six, three became derived expressions — three of the six now read from the same data structure that produces the number printed next to them, instead of restating it by hand.
The other three could not be derived at all. There is no stored counterfactual for "would this have happened without that fix." There is no artifact for a value observed in a previous session. Forcing derivation there would have meant inventing a source.
So the rule I actually landed on is a pair:
The second half matters more than it looks. "This is a session note, not reproducible from this run's artifacts" is eleven words that convert an unverifiable claim into a verifiable statement about verifiability. Leave it as a bare literal and it rots on the next round. Mark it and the next reader — including future you — stops asking whether it should have been derived.
If you write this down as "we replaced all the literals with derived values," you have just written a seventh literal. I know because that is the subheading I wrote in the correction log, with a table two lines below listing the two counterexamples I had personally just written. The review caught it. That sentence is now struck through in the document, above the table that refutes it.
Because this class kept recurring, a previous round added a checker: for a registry of specific claims, pull the supporting evidence out of the JSON, and if that evidence set is empty, refuse to let the corresponding paragraph state a confident conclusion. Conclusion vocabulary near an empty evidence set is a failure.
The obvious false-positive problem is hedged language. A paragraph can legitimately say "this did not fire" right next to a sentence about something that did. So the first version exempted a conclusion word if a hedge marker appeared within roughly a hundred characters.
Character distance is not clause membership.
Real sentences contrast. "A did not fire, but B did" puts the hedge and the conclusion a few words apart and pointing at different things. According to the change log — the old rule's code is gone from the repo, so I can't re-run it myself — the checker exempted the second clause because of a marker belonging to the first, and it passed exactly the sentence shape it existed to catch.
Worse: according to that same change log, the checker's own built-in tamper test said it worked. That test reverted the whole paragraph to its pre-fix text and confirmed a failure was raised. But the failure came from an unrelated sentence elsewhere in that paragraph, not from the headline under test. The test passed by coincidence, and the coincidence read as proof.
Two fixes:
Here is the part I did not expect.
Having fixed the rule, I extended coverage to two table rows that had gone unchecked — including one that had contained a live misattribution. Then I put the original false claim back into the real file and ran the checker.
It passed.
The reason had nothing to do with clauses. That registry entry uses a different axis: when evidence is non-empty, the check is simply "is each evidence item mentioned somewhere in the unit?" The unit was the whole table row — and that row happened to name the missing item again, several clauses later, for a completely unrelated reason. Substring containment was satisfied by an accident.
So the entry I had just added to strengthen the guard was a guard that could not fail. Fixing the checker had produced, one level up, precisely the thing the checker exists to prevent: something that looks like verification and verifies nothing.
The fix was to let a registry entry narrow its check to the clause making the claim, rather than the whole row. But the durable lesson is the ordering: I only found it because I injected the defect after the change, into the real artifact, and watched. Nothing in the passing run would have told me.
That case is now a permanent mutation test. It prints, on every run, what the result would have been without the narrowing — "no missing items, i.e. a false negative" — so the reason the narrowing exists cannot quietly detach from the code.
The checker now catches the shapes I have actually seen. It prints, on every run, what its own result would have been without the clause-narrowing step — a standing false-negative warning — and its registry documents that eight of its nine entries still lack that narrowing. Those limits are not a disclaimer. They are the part of the report that is still a literal.
Q. What's the core problem this piece describes?
In a script that turns an evaluation run into a report, the numbers come from f-strings computed from live data, but the explanatory sentences are string literals sitting in the same function. A literal has no dependency on the data it describes: when the data changes, the literal stays, so the report can print a correct number right next to a sentence that is now false.
Q. How did the six failing sentences break down?
An adversarial review found six prose claims contradicted by the report's own data, and they failed in four different ways: two were drift, where a count changed and the sentence didn't; two were wrong from the first commit, including one that described a state machine backwards; and two were unsupported assertions, attributing an outcome to a cause with no stored evidence to establish it.
Q. Why didn't deriving every sentence from source data fully fix it?
Of the six, three became expressions that now read from the same data structure that produces the number printed next to them, instead of restating it by hand. The other three couldn't be derived at all — there was no stored counterfactual and no artifact for a value from a previous session. For those, the fix was to write the reason no source exists directly into the sentence, turning an unverifiable claim into a verifiable statement about its own unverifiability.
Q. What went wrong with the checker built to catch this class of bug?
According to the change log — the old rule's code is gone from the repo, so it can't be re-run directly — the first version exempted a conclusion word from scrutiny if a hedge marker appeared within about a hundred characters of it, treating character distance as a stand-in for clause membership. In a real contrasting sentence like 'A did not fire, but B did,' the hedge and the conclusion sit a few words apart but belong to different clauses, so the checker exempted the wrong clause and passed exactly the sentence shape it was built to catch. Its own tamper test had reverted a whole paragraph rather than one sentence, so a pass there proved nothing about the specific claim under test.
Q. What happened after the checker was fixed?
One level up, a newly added check on two previously uncovered table rows passed even with the original false claim reinserted, because that check only tested whether an evidence item was mentioned anywhere in the whole row — and the missing item happened to be named again elsewhere in the same row for an unrelated reason. The fix was to narrow the check to the clause making the claim rather than the whole row, and that case is now a permanent mutation test: on every run it prints what its own result would have been without the clause-narrowing step, and its registry documents that eight of its nine entries still lack that narrowing.