I built a gate to catch a specific kind of lie. One narrative device in a pipeline that renders short map-explainer videos places country B next to country A "at the same scale," so a viewer can compare their sizes at a glance. Get the scaling wrong and the device becomes the lie it exists to prevent. An earlier note on this site covered a related failure — a caption-width guard that passed every test because it measured text the renderer never actually drew. This is the same mistake in different clothes — a validator checking the wrong artifact — except here the artifact is geometry, and the gate didn't just miss a lie. It accused an already-published, honest video of telling one.
The renderer draws on an equirectangular (plate carrée) projection, where a degree of longitude compresses by cos(latitude) — move a country to a different latitude and its apparent width changes even though the country didn't. The renderer already corrects for this: lon_scale = cos(source_lat) / cos(dest_lat) inside translate_polygons_to_centroid in src/shorts_factory/map_scenes.py redraws the moved country as if it had always lived at the destination latitude, so it takes the same 1/cos φ inflation as the target and the ratio between them stays honest.
But that's one scale factor from one centroid latitude, applied to a whole polygon. It cannot unshear a country that spans 40° of latitude — the top and bottom inflate by different amounts, and one factor can't know that.
So I added a gate at script-ingest time, before any rendering happens:
residual = | on-screen area ratio / true area ratio − 1 | reject if residual > 0.05
On-screen area is the planar shoelace area of the laid-out polygons; true area is a spherical integral using Earth's mean radius, 6371.0088 km — pure coordinate math, no frame required. The threshold is OVERLAY_AREA_RATIO_RESIDUAL_MAX = 0.05 in src/shorts_factory/script_schema.py.
To specify it, I hand-computed four golden vectors and told the implementing worker to reproduce them. It matched all four to four decimal places. I was confident. I shouldn't have been.
I didn't stop at fixtures. I ran the finished gate against an episode that had already shipped: scene 0 of mongolia-empty places France next to Mongolia (MNG←FRA) for a size comparison. The gate's verdict and what a viewer actually sees disagreed by an order of magnitude:
| gate verdict | what's actually on screen | |
|---|---|---|
MNG←FRA, mongolia-empty scene 0 | residual 14.4% — reject | residual 1.0% — honest |
That video is public. That scene is honest. The gate was wrong, and wrong in the least comfortable direction a validator can be: confidently, specifically, about something real that had already shipped.
The codebase has two ways to turn the same GeoJSON file into polygons: a raw parse that reads the file exactly as it ships, and load_world_geojson(), which parses and then calls _filter_far_flung_polygons() to drop pieces sitting far from a country's mainland — built for camera framing, so a bounding box doesn't get dragged across an ocean by a speck of overseas territory. The renderer calls load_world_geojson().
The gate called the raw parse. My reasoning felt airtight: to measure a country's true area you need its whole geometry, not a filtered subset. That's exactly backwards for the two countries where it mattered most.
Russia's raw polygon set crosses the antimeridian — Chukotka sits on the far side of the date line, so the coordinates span the full −180° to 180° range. Once that happens, both the shoelace-area formula and the centroid stop meaning anything; the shape isn't the shape anyone intends.
France fails more quietly. French Guiana, in South America at about 4°N, drags the combined centroid to (−10.7°E, 35.4°N) — the middle of the Atlantic, nowhere near France. Anchor the translation math to that point and the scaling is wrong before the comparison starts.
The renderer never draws either piece. The mongolia-empty script had include_overseas_territories: false, so the France on screen was metropolitan France, full stop. I had measured the honesty of a picture using geometry the picture never contained.
I'd computed all four golden vectors by hand, from the same raw geometry the gate used. The worker implementing the gate reproduced them to four decimal places — precision that's exactly what makes the failure worth writing down. Reproducing my numbers meant reproducing my misunderstanding. A golden vector proves an implementation matches its spec; it cannot prove the spec is correct. When the vectors and the spec share an author and an assumption, they don't cross-check each other — they agree with each other, and the agreement feels like confirmation.
What actually caught the bug wasn't a vector. It was pointing the finished gate at an artifact that already existed and was already known to be correct, and watching it fail anyway.
The fix is one substitution: measure with load_world_geojson() plus that script's own include_overseas_territories flag — the exact geometry the renderer will draw for that episode, no more and no less.
I didn't stop at the one failing case. I re-derived the whole thing exhaustively (report/scripts/recount_honest_pairs_render_geometry.py) across all 177 countries and every pair whose true area ratio falls between 0.5 and 2.0 — 6,478 pairs, the band where a size comparison is actually a visual claim. Under the corrected geometry:
| metric | value |
|---|---|
| pairs passing the 5% threshold | 6,353 / 6,478 (98.1%) |
| median residual | 0.3% |
| 90th-percentile residual | 1.4% |
| maximum residual | 79% (CHN←ATA) |
The worst pairs are all Antarctica — CHN←ATA 79%, BRA←ATA 76%, AUS←ATA 75% — and the rest of the 1.9% that fail concentrate in Antarctica and Russia, the countries with the most extreme latitude span. A few representative pairs show the shape of the correction:
| target ← moved | Δ centroid latitude | residual (corrected) | residual (uncorrected) |
|---|---|---|---|
| India ← Sweden | 38.8° | 0.8% | 101.0% |
| Kazakhstan ← Mongolia | 0.1° | 2.5% | 2.6% |
| Mongolia ← France | 0.6° | 1.0% | 0.5% |
| Canada ← Russia | 4.1° | 6.0% (fail) | 12.5% |
| Brazil ← Russia | 70.8° | 7.6% (fail) | 106.6% |
Read the second and third columns together: Kazakhstan←Mongolia moves centroids a mere 0.1° apart and still lands at 2.5% residual, worse than India←Sweden's 38.8°-apart 0.8%. Centroid distance doesn't predict the residual — the cos correction already removes that axis. What's left is each country's own latitude span: Russia, running 41°N to 81°N, fails both pairs above regardless of how close the other centroid sits.
Before the fix, my own report on this device had concluded it was honest "only in very limited cases" — not a finding about the world, but an artifact of the instrument: a broken measurement doesn't only let bad things through, it also invents constraints you don't have. The same blind spot showed up again when a review proposed a simpler guard instead of the residual math — just keep the two centroid latitudes within ±5° of each other, a rule that sounds conservative because it's tight and cheap to explain. Measured against the same 6,478 pairs, the ±5° band passes only 828, against 6,353 (98.1%) for the residual gate, and it rejects 5,527 pairs that are honest by the actual area math while still letting 2 dishonest ones through. It isn't conservative on one side of a tradeoff; it's wrong in both directions at once, and the only way to see that was computing both rules over the same data instead of trusting which one sounded safer.
The obvious regression test is "call load_world_geojson, not the raw parser." That pin breaks the moment either function gets renamed, and it doesn't say what actually needs to stay true. What has to hold is a relationship, not a function name: the gate's verdict has to track the render setting.
@pytest.mark.parametrize("include_overseas, should_raise", [(False, False), (True, True)])
def test_overlay_area_ratio_follows_include_overseas_territories(...):
# same MNG←FRA pair: False -> residual 1.0%, passes / True -> Guiana returns, 14.4%, rejectedSame pair, same script, one boolean flipped, and the verdict flips with it — a smaller, more durable promise than "always call the right function," and the one the parametrized test in tests/test_script_schema.py actually pins down.
parse_x next to load_x.This reasoning breaks if the geometry the renderer drops is routinely part of the claim being made. If narration says "all of Canada" while the screen shows the 84.2% that remains once the Arctic archipelago is filtered out, a screen-based gate happily passes that lie — it was never built to catch it, because for this device the screen was the right thing to measure. That failure mode doesn't need a stricter threshold; it needs a second, independent check asking whether the screen represents the claim, a different question from whether the screen is internally consistent. For an overlay size comparison, measuring the screen is the right call. The day narration starts making absolute-size claims instead of relative ones, this judgment needs revisiting, not reuse.
Q. My validator passes every golden vector but rejects data I know is good — what does that mean?
It means your golden vectors and your validator's logic were built on the same assumption, so matching them proves internal consistency, not correctness. Golden vectors show an implementation matches its spec; they cannot show the spec itself is right. Run the validator against an artifact you already know is correct — that cross-check is what a self-consistent vector suite can never provide, and it is what actually surfaces the bug.
Q. Why would raw, unfiltered input be less trustworthy than filtered input for a correctness check?
Because unfiltered is not the same as correct. In this case the raw GeoJSON was numerically broken for the two countries that mattered most: Russia's polygon set crosses the antimeridian, making its area and centroid meaningless, and France's raw set includes French Guiana, dragging its centroid into the middle of the Atlantic Ocean. The renderer never drew either piece, so validating against the raw file measured geometry the screen never showed.
Q. How do I write a regression test for a bug like this without pinning to internal function names?
Pin the invariant, not the implementation. Instead of asserting that a specific loader function gets called, assert on the observable relationship that has to hold: here, that the gate's pass/fail verdict for a given pair follows the script's own render setting (include_overseas_territories) rather than staying fixed. A parametrized test that flips that one setting and checks that the verdict flips with it survives any future refactor of the loading code.
Q. Should a validator always trust rendered output over the original source data?
For checking whether what's on screen is honest, yes — measure the screen, not the source. But that is not universal: if the renderer drops geometry that the accompanying narration still claims as part of the picture, a screen-based gate will pass that specific lie, because it was never built to check whether the screen represents the full claim. That failure needs a second, separate check; it does not mean the screen-based measurement was the wrong choice for the property it was built to guard.