Prompt caching is the largest single token lever for any long-lived LLM session — a cached token is served at roughly a tenth of its normal price. It is also one of the easiest things to break by accident, because when it breaks there is no error. One timestamp in the system prompt, one unsorted JSON key, and your cache-read rate quietly goes to zero while the bill goes up tenfold. This is how the failure works, and how to make it loud.
datetime.now() in the system header, a per-request UUID, a dict serialized without sorted keys, a tool list that reorders — silently defeats caching for the entire request. The cost is real (a miss is ~10× a hit on the cached span) and invisible (no exception, no warning). The only signal is one field in the usage object, and the only durable fix is to assert on it.The mental model that prevents every bug below is one sentence: the cache key is the bytes of the rendered prompt, matched as a prefix. The provider renders your request in a fixed order — tools, then the system prompt, then the messages — hashes the bytes up to each cache breakpoint, and looks for a prior entry with the same prefix. If it finds one, the shared span is a cache read; the rest is processed fresh. If the very first byte differs, nothing matches and the whole thing is processed at full price.
Three properties follow, and all three are load-bearing:
The economics are what make this worth caring about. A cache read costs about 0.1× the base input price; a cache write costs about 1.25× for the 5-minute TTL. So the break-even is fast — two requests sharing a prefix already come out ahead — and the downside of a silent miss is steep: every token you could have read at one-tenth price is instead reprocessed at full price. That is the 10× in the title, and on a long agent loop that resends a large fixed preamble every turn, it is most of the bill.
cache_read stays zero and nothing tells youHere is what makes this class of bug nasty. A broken cache is not an error. Your requests succeed, your outputs are correct, your latency is a little worse and your cost is a lot worse, and nothing in the response says "you just paid full price for 40,000 tokens you had cached thirty seconds ago." The system is behaving exactly as designed; it simply never found a matching prefix.
The only signal is the usage object. Every response reports three numbers you have to actually look at:
| Field | Meaning |
|---|---|
cache_read_input_tokens | Served from cache — you paid ~0.1× |
cache_creation_input_tokens | Written to cache — you paid the ~1.25× write premium |
input_tokens | Reprocessed at full price — not cached |
cache_read_input_tokens is zero across repeated requests that should share a prefix, a silent invalidator is at work. That single field is the whole diagnosis. When it's zero and you expected a hit, the move is mechanical: capture the fully rendered prompt bytes from two consecutive requests and diff them. The byte that changed is your invalidator, and it is almost always one of a small, known set.Every one of these changes the prefix bytes without changing the logical prompt — which is exactly why they're easy to write and hard to notice.
f"Current time: {datetime.now()}" or a per-request trace ID interpolated into the system header changes the prefix on every single request, so the cache is never read even once. Anything that must be dynamic belongs after the last breakpoint, in the message list — a fact injected at turn 5 invalidates nothing before turn 5.json.dumps(d) without sort_keys=True, or by iterating a set, can emit different bytes run to run for identical content. Same logical prompt, different key order, cache miss. Sort your keys and order your collections.if flag: system += "..." means every combination of flags is a different prefix. Each variant caches separately and shares nothing.The cruel part is that the failure concentrates in precisely the workloads caching is for. A one-shot classification call barely cares. A long-running agent — a fleet of workers, a multi-step DAG, an orchestration loop that resends a large fixed system prompt and tool set on every one of hundreds of turns — is entirely built on the assumption that the big fixed prefix is nearly free after the first write. When a worker reassembles its system prompt on each call and one field in that assembly is nondeterministic, that assumption silently inverts: the largest, most-reused span of the prompt becomes the most-repaid. The bigger and more disciplined your prompt architecture, the more a single stray byte costs you.
Because the failure is silent, the only real defense is to stop trusting and start measuring. The audit is short:
cache_control markers you sprinkle downstream.usage.cache_read_input_tokens > 0 on the second — and fail loudly if it isn't. A silent cache miss caught by an assertion in development is a non-event; the same miss shipped to a long-running fleet is a standing tax you won't see until you read the bill.None of this is exotic. It's the same discipline as any other cache: know your key, keep it stable, and verify your hit rate instead of assuming it. The only twist with prompt caching is that the key is the entire rendered prefix, so "keep it stable" reaches all the way back into how you build the system prompt — and the penalty for a stray byte is measured in tens of percent of your token bill, paid quietly, forever, until someone looks at the one field that would have told them.
Q. Why does a timestamp in the system prompt make prompt caching stop working?
Because a prompt cache is a prefix match: the key is the exact bytes of the rendered prompt up to a breakpoint, and any byte that changes anywhere in the prefix invalidates everything after it. A timestamp, UUID, or per-request ID near the front of the system prompt changes on every request, so nothing behind it is ever reused. Freeze the system prompt and move volatile values to the end of the message list, after the last cache breakpoint.
Q. How much does a prompt-cache miss actually cost?
On Anthropic's API a cache read costs about 0.1× the base input price and a cache write about 1.25× for the default 5-minute TTL. A token you could have read from cache at one-tenth price but instead reprocess at full price costs roughly ten times what it should. Across a long session that reuses a large prefix every turn, that gap is usually the single biggest token lever you have.
Q. How do I tell if my prompt cache is actually being hit?
Read the usage object. cache_read_input_tokens is what was served from cache, cache_creation_input_tokens is what was written, and input_tokens is what was reprocessed at full price. If cache_read_input_tokens stays zero across repeated requests that should share a prefix, a silent invalidator is at work — diff the rendered prompt bytes between two requests to find it.
Q. Does non-deterministic JSON serialization break prompt caching?
Yes. If any part of the cached prefix is built by serializing a dict/map without a stable key order, or by iterating a set, the bytes can differ run to run even when the content is identical, and the cache misses. Serialize with sorted keys and iterate ordered collections so the same logical prompt always produces the same bytes.
Q. Why does adding or reordering a tool invalidate the whole prompt cache?
Tools render at the very front of the prompt, before the system prompt and messages. Because caching is a prefix match, any change to the tool set changes position zero and invalidates the entire cache below it. Keep the tool list frozen and deterministically ordered per session; for dynamic tools, use a discovery mechanism that appends schemas rather than swapping the base set, and don't switch models mid-session, since caches are model-scoped.