I run a renderer that turns geographic data into short map-animation episodes — country polygons filled in over a basemap, driven by a script that lists, per scene, which countries to focus on, which are neighbors, and which get highlighted. On a single day, it produced two defects, and both had the exact same shape: the renderer silently dropped part of what the script had ordered it to draw. Both had passed everything I had running at the time.
The code that draws a country's polygon was only using the outer boundary:
exterior = polygon[0] # simplification: ignore interior holesSouth Africa's polygon carries a real interior ring at Lesotho's location — an actual hole, because Lesotho sits entirely inside South African territory. Drop that ring and South Africa paints as one solid shape. Countries draw in alphabetical order, so Lesotho — drawn first — got painted, and then South Africa, drawn afterward with no hole cut into it, covered it over. The episode this happened in is titled "A COUNTRY INSIDE A COUNTRY." The subject of the episode had disappeared from its own frame.
The code that decides which countries get a filled polygon at all was built from two of the scene's three country lists:
target_codes = set([*scene.focus, *scene.neighbors]) # no highlightThe glow effect around a highlighted country is generated from the separate highlight field, so a country that's only in highlight — not in focus or neighbors — never enters target_codes. It gets the glow. It gets no polygon. One episode's label was "Three, side by side"; only one of the three countries actually had a filled shape, the other two were a blur of red light around nothing. In a different episode, the same gap meant all five highlighted countries came out that way.
At the time, this renderer had three separate verification passes, and all three were green on both episodes.
| Check | What it actually asserts |
|---|---|
| Schema validation | The input is well-formed — country codes exist, fields are populated |
| Compliance gate | Policy was followed — sourcing, AI disclosure, label conventions |
| 501 unit tests | The renderer's internal behavior matches expectations — patch counts, the color at a given coordinate, label-placement rules |
Not one of the three asks what the rendered artifact is actually saying. All three either look at the input or at an intermediate structure the renderer builds on the way to a frame. So "the label claims three countries and the picture shows one" got produced, cleanly, underneath three green signals — because none of the three signals was ever pointed at the thing that was wrong.
This is worth stating plainly, because it isn't a case of the checks being lazy or thin — it's structural. In both bugs, the input was exactly correct. The script ordered the right countries; scene.focus, scene.neighbors, and scene.highlight all had the right values; the country codes existed; the fields validated. What failed was delivery: code that ran precisely as written and, by design, dropped something on the way to the frame. exterior = polygon[0] is not a defect a schema can see, because the schema was never told a polygon might carry more than one ring. A discard bug like this is invisible to an input assertion by construction — there is no wrong input to catch, because the input was never wrong.
I've written before that you should verify the surface the consumer actually sees instead of the payload you built — that's the general principle, and this is a sharp case of exactly why the substitution fails: the input check wasn't weaker evidence, it was evidence about a different fact. A separate note covers a renderer measured against the wrong source geometry entirely, upstream of rendering — a different failure from a mid-render discard, where the source was right and the render dropped part of it anyway. And another one is about an agent's inability to perceive its own render at all; this isn't a perception problem, it's a check aimed at the wrong target from the start.
I opened the frame and saw a black shape sitting inside South Africa's outline — easy to look at and conclude "there's the hole" and move on.
Back-projecting those pixels through the camera transform put them at longitude 30.8–32.2, latitude −27.4 to −25.6: Eswatini, not Lesotho. Eswatini borders Mozambique, so it was never an enclosed country to begin with — whatever put a dark shape there was unrelated to the ring-dropping bug. Looking at the frame produced a confident answer. It just wasn't the right one.
The first attempt to check this programmatically was also wrong. matplotlib's Path.contains_point returns True for a point inside an interior ring, same as for a point inside the fill — true as of matplotlib 3.11, and make_compound_path doesn't change that. Ask the geometry object whether a point is "inside the polygon," and a hole doesn't register as an exception to that; it's still inside the outer boundary.
What actually answered the question was connectivity, not color and not the eye: flood-fill from the frame's outer border, then count the pixels that fill can never reach.
Before the fix: 0 enclosed pixels
After the fix: 3,900 px, x644-731 / y878-955 — matches the back-projected location of Lesotho"Look at the output" is not a verification method. "Ask the output the same question the claim is making" is.
When an artifact is something a person has to interpret — an image, a PDF, a chart, a rendered document, an audio track — a test that inspects intermediate structure cannot, by construction, catch that the artifact stopped saying what it was supposed to say. Those tests are all answering "did the pipeline run the way I coded it." The question that actually matters is "does the artifact say what the claim says," and the two questions do not overlap. A pipeline can run exactly as coded and still discard part of the order, because "ran as coded" was never a promise that nothing gets dropped.
Three rules came out of fixing this:
1. Write one question per claim, and take the question from the claim, not the code. "Is Lesotho a hole?" becomes: flood-fill from the frame border, count the enclosed components. "Are three countries highlighted?" becomes: count the connected components in the highlight color. The question has to come from what the episode is claiming, because the code will happily tell you it's doing exactly what it's doing. 2. Answer the question in the same medium as the artifact. An image's claim gets asked of pixels. A sound's claim gets asked of the waveform. Ask a data structure instead and you get the data structure's answer — contains_point will tell you a hole is "inside," because that's a true fact about the geometry object and a false one about the picture. 3. Don't use what you saw with your eyes as evidence. A human eye is reliable up to "something is there." It is not reliable for "that is what it is" — mine confirmed a dark shape and had nothing to say about whether it was the hole the bug produced or something else entirely. It only becomes evidence once you can back-project it to coordinates or count it into a named, located component.
Once a claim becomes a precise, mechanical question, it can be run against everything you've already shipped, not just the thing in front of you. The second defect's condition writes exactly as highlight ⊄ focus ∪ neighbors — a highlighted country the focus/neighbor set doesn't already cover. Running that condition against the ten episodes already published took about a minute and confirmed none of them triggered it — the condition happened not to be satisfied anywhere in what was already out the door, so nothing already published needed a fix. Turning a claim into a question about the artifact doesn't just catch the next instance of the bug. It tells you, mechanically, how far the last one reached.
Because both checked the input or the renderer's internal behavior, not what the rendered artifact was actually saying. Schema validation confirmed the country codes existed and the fields were populated — true both times. The unit tests confirmed the renderer's internal mechanics (patch counts, the color at specific coordinates, label-placement rules) matched expectations — also true both times, because the code executed exactly as written. The defect was in what the code was written to do, not in whether it did it.
It didn't, and it actively misled. A black shape inside South Africa's outline looked like a hole, but back-projecting its pixel coordinates through the camera transform put it at Eswatini, not Lesotho — a country that borders Mozambique and was never enclosed to begin with. The eye confirmed something was there; it couldn't say what.
Path.contains_point returns True for a point inside an interior ring, the same as for a point inside the fill — true as of matplotlib 3.11, and make_compound_path doesn't change that. What actually distinguished a hole was connectivity: flood-filling from the frame's outer border and counting the pixels that fill could never reach.
For each sentence a claim makes, name one question that can only be answered by inspecting the artifact itself, in its own medium — pixels for an image, the waveform for audio. A discard bug, where the renderer runs exactly as coded but quietly ships less than it was told to, is invisible by construction to any check that asks the question of the input or an intermediate structure instead.
Its trigger condition can be written exactly as highlight ⊄ focus ∪ neighbors. Running that condition against the ten previously published episodes took about a minute and confirmed none of them triggered it, so nothing already shipped needed a fix.