res.ok, so a rate-limited response parsed as a legitimately empty table and was memoized — the board served zero rows for every date long after the upstream recovered, with HTTP 200, a completed walk, a fresh memo and silent logs the whole time. Two lessons generalize past this stack: success is a verdict on the body schema, not on the status code, and the cache write has to sit downstream of that verdict, because a cache in front of validation is poisoned by construction. A revision token in the key is what evicted the entries afterwards.My travel app's backend walks Korea's government open-data portal, data.go.kr, for airport departure boards. One evening a board walk came back empty, memoized that, and kept serving zero rows for every date long after the upstream had recovered. Nothing reported a failure, because by the only definition the code was checking, nothing had failed: the response was HTTP 200.
The portal's gateway wraps every response in an envelope carrying its own status. Success is the header code 00. Throttling is reported through that same envelope — a 200, a well-formed body, a non-00 code in the header, totalCount: 0, and an empty item list.
{
"response": {
"header": { "resultCode": "<anything but 00>", "resultMsg": "<why it failed>" },
"body": { "totalCount": 0, "items": [] }
}
}The consumer is a paginated walk: fetch page 1, read totalCount, derive the page count, fan out the rest, concatenate, memoize the assembled board per operation. The guard before parsing was the one every fetch wrapper I have ever written has:
const res = await fetch(url);
if (!res.ok) return null;
const body = await res.json();
const items = body?.response?.body?.items ?? [];
return items; // throttled -> [] -> memoized as "the board"Under throttling, res.ok is true. The JSON parses. The item list is empty. The walk reports success with zero rows, and the memo stores that as the board. The visible symptom: the international board for Gimpo (GMP) had returned 32 rows earlier that day, and after one evening walk it returned zero — for every date, for as long as the memo lived. HTTP 200. Walk completed. Memo fresh. Logs silent.
The design deliberately treats a missing flight as a fallthrough rather than an error: absence in one feed is not evidence of nonexistence — a board can transiently miss a flight through a gap or a codeshare edge — so a lookup the free feed cannot answer falls through to the metered commercial provider instead of returning a 404. That is the right call, and it is also why an empty board is a completely legitimate value here, indistinguishable at every layer above the walker from "the upstream refused to talk to us."
Generalize it: any system with a legal nothing here — an empty search result, a zero balance, an empty entitlement list, a null price — will silently absorb an upstream error that decays into that value. Once the error is laundered into the domain's legal empty, no consumer downstream can tell the difference, and no amount of defensive coding at those layers will help. The check has to happen at the boundary where the error is still distinguishable from the value.
Every one of these calls stacks two protocols. HTTP transports the exchange; the gateway's envelope reports the application's outcome. res.ok — or raise_for_status(), or if err != nil — adjudicates the transport only. Whether the upstream did the thing is a separate claim, made in the body. This is not an exotic quirk of one government portal: GraphQL conventionally answers 200 with an errors array, JSON-RPC transports failures as an error member in a 200 body, SOAP puts faults in the envelope, and plenty of enterprise and public-sector gateways stamp 200 on anything that reached the application at all. Wherever a body carries its own status, the status code is a routing detail, not a verdict.
So the rule I now apply is that success is a schema predicate: a response is a success when its body parses into the shape success has — envelope status is the documented OK value, and the fields the caller needs are present and typed. Everything else, including a beautifully formed error body under a 200, is a failure and returns nothing.
const res = await fetch(url);
if (!res.ok) return null;
const body = await res.json();
if (body?.response?.header?.resultCode !== "00") return null; // envelope gate
const items = body?.response?.body?.items ?? [];
return items;The broken ordering was: fetch → transport check → parse → memoize → return. Any judgement about what the payload meant happened, if at all, downstream of the write, so the moment a bad value cleared the transport check it was durable. The fixed ordering is: fetch → transport check → parse → adjudicate → memoize. Only an adjudicated success is cacheable. The walker returns null on an error envelope; null is not a board, so the caller never writes the memo and the next request retries. That single reordering changes the failure's lifetime from the cache's TTL to the upstream's outage — which is the whole game, because the throttling passes and the memo does not.
Two corollaries came with it. Partial successes are not successes: each operation's board is memoized atomically, all-or-nothing, so a truncated walk is never kept — a half-populated cache entry is the same bug wearing a friendlier face. The layer above has to agree: the route layer marks an incomplete board cacheable: false, so a partial result cannot be re-poisoned into a downstream cache. Fixing validation in one layer is worthless if the layer above saved the bad value anyway.
Nothing about this failure looked like a failure. The honest answer to "how was it caught" is that a staged proof re-run at a different hour returned different data for the same date — that was the whole signal, and chasing it took three instrumented rounds, tail logging plus a raw re-probe, because every layer reported health. What I would actually trust as a detector, in rough order of cost:
Fixing the walker does not fix the entries the broken walker already wrote: the deploy ships correct code that goes on reading a wrong value it saved earlier. You have repaired the producer and left the poison in the store, and a remediation plan amounting to "wait out the TTL" just makes the incident's length a property of a config constant. The fix that generalizes is a revision token in the cache key:
const memoKey = `${WALK_REV}:${op}:${day}:${pageCap}`;Bumping it to 2 in the same deploy as the envelope gate orphaned every entry the old walker had written — instantly, without enumerating keys, without a flush endpoint, and without needing to know which entries were bad. New code reads new keys; the orphans expire unread. That property is what makes it the right tool rather than a clever one: this memo lives per worker isolate, on a deployment where the edge cache itself was inert — no key list, no admin flush, no way to address the store from outside. A revision component is the only invalidation primitive that works on a store you cannot enumerate or reach, and it costs one string concatenation.
Its sibling rule fell out of the same proofs. The international walk had a page cap of 6, sized off Gimpo's 4-page board. Busan's international board is 22 pages, so the cap truncated it — and the surviving rows were all outside their effective date window, so the board rendered stale-empty: the same wrong answer by a different route. The fix was per-operation caps plus putting the cap in the memo key, so raising a cap cannot keep serving the board the old cap truncated. Generalized: every input that changes the value belongs in the key — operation, parameters, caps, and the revision of the code that built it.
Under active throttling the board now serves the domestic rows alone and retries the international operation next request, rather than pinning an empty board. That is a real cost, stated plainly: during an upstream outage a complete-looking board is quietly missing a section. It was judged the lesser harm for suggestion-grade data that never blocks manual entry — a partial suggestion list recovers on the next request, a memoized empty one does not. An earlier draft coupled the established board's availability to the brand-new operation's uptime, which inverted the priority; review reversed it. Other residuals I would rather name than let a reader assume away:
After the fix the boards converged and stayed converged, with warm calls around 0.16s and no client change at all. Three rules survive the specifics. Success is a body-schema verdict, not a status code — if the payload carries its own status, check it, on every page, before anything else looks at the data. The cache write goes after the verdict, never before it — a cache in front of validation converts every upstream hiccup into a durable lie whose lifetime is your TTL rather than their outage. Keys carry the revision — anything that changes what a cached value means belongs in the key, because that is the only invalidation you can perform on a store you cannot enumerate.
Q. Why does an API error arrive with an HTTP 200 status code?
Because many gateways report the application-level outcome inside the response body rather than in the status line. The government portal in this case answers throttling with a 200, a well-formed JSON body, an envelope header code other than the documented success value, a total count of zero, and an empty item list. GraphQL, JSON-RPC and SOAP all do versions of the same thing. The status code told me the exchange completed; only the body says whether the upstream did the work.
Q. Is response.ok enough to check whether an API call succeeded?
No, not when the payload carries its own status. response.ok, raise_for_status and their equivalents adjudicate the transport only: a response arrived with a 2xx code. Success has to be a schema predicate — the envelope status equals the documented success value and the fields the caller needs are present and typed — so everything else, including a well-formed error body under a 200, is a failure. Run that check on every page of a paginated walk, not just the first, because pages two onward are exactly where a rate limiter starts saying no.
Q. Why does a cached error keep being served after the upstream recovers?
Because the cache write happened before anything adjudicated the response. When the pipeline is fetch, transport check, parse, write, any value that clears the status-code check becomes durable, and the failure then lasts as long as the cache entry rather than as long as the outage. Moving the write after the success verdict — returning null on an error envelope so the caller never writes the entry and the next request retries — makes the failure's lifetime the upstream outage again.
Q. How do I invalidate cache entries that a bug already poisoned?
Put a revision token in the cache key and bump it in the same deploy as the fix. That orphans every entry written by the old code path at once, with no key enumeration and no flush endpoint, which is what makes it work on a store you cannot address from outside — a per-isolate memo has no key list and no admin flush. Waiting out the TTL is not a remediation plan; it makes the incident's length a property of a config constant.
Q. Why did nothing downstream notice that the cached board was empty?
Because empty was a legal answer. The design treats a flight missing from one feed as a fallthrough to another provider rather than an error, so an empty board is a legitimate value that no consumer above the walker can distinguish from an upstream refusal. Any system with a legal nothing-here value — an empty search result, a zero balance, an empty entitlement list, a null price — will silently absorb an upstream error that decays into that value, so the check has to happen at the boundary where the error is still distinguishable from the value.