503 auth_unavailable, never reaching the vendor. The tell is in the access log's latency column: the first 503 in an episode takes hundreds of milliseconds because it actually reached the vendor, and every 503 after it comes back in a few milliseconds because the gateway rejected it locally. The fix is disable-cooling: true on that one provider when it holds a single key, or adding a second key so the cooldown goes back to being a real rotation.For about three months, I had a working theory about one leg of my model roster: the free tier was flaky. Every so often, a request routed through an NVIDIA NIM model would come back 503. My client would retry on a one-second backoff, then a three-second backoff — retry_transient: [502, 503, 504] — and both would also fail. Then, roughly a minute later, with no intervention from me, the same request would go through fine. I filed it under "free tier, what do you expect," because a failure that heals itself on its own timetable is what a rate-limited endpoint is supposed to look like.
The theory survived because the evidence fit it every time. What broke it wasn't a new outage — it was a probe result and one column in an access log I'd been reading past for months. The vendor wasn't the unreliable part. A gateway I run myself was — and, as it turned out, just one of four separate causes behind the unstable leg I was chasing that same day, the only one that actually lived inside the gateway.
The gateway is CLIProxyAPI 7.1.58, a multi-credential proxy — the kind of thing grouped with LiteLLM-class routers. It runs locally; everything downstream of it, including an MCP tool server's custom provider and a council persona whose OPENAI_BASE_URL points at it, reaches it on a local port. Its access log writes one line per request, with a latency column I'd been skimming past:
[gin_logger.go:97] 503 | 6ms | 127.0.0.1 | POST /v1/chat/completionsOnce I actually read that number across a run of failures, the pattern was obvious. The first 503 in an episode carried a latency of 316 ms — a real round trip, long enough to actually reach NVIDIA and come back with a failure. Every 503 that followed in the same episode came back in 6 ms, and in older episodes as fast as 1–2 ms — the gateway answering from memory before the request ever left the building. A direct probe to the vendor, at the same moment one of these episodes was running, came back 200. A status code alone doesn't say which layer failed — the same discipline applies in reverse, where an error envelope inside an HTTP 200 still reads as success if the only check is response.ok (An Error Inside HTTP 200 Poisoned My Cache). Here, a 503 looked exactly like a vendor failure and wasn't one; the only way to tell them apart was to time it.
A gateway holding more than one credential for a provider uses a cooldown to protect itself: if key A errors, set it aside and route the next request through key B. In CLIProxyAPI's code, that logic lives in sdk/cliproxy/auth/conductor.go — case 408, 500, 502, 503, 504: sets NextRetryAfter = now.Add(1 * time.Minute) unless disable-cooling is set on that provider block (429 runs through a separate quota path). The per-provider switch lives at internal/config/config.go:658.
That logic assumes a key B. My NVIDIA NIM provider block had exactly one API key configured — one api-key-entries entry. So the same cooldown that's a sensible rotation with two keys instead put the entire provider to sleep for that minute: zero available auth, for the only credential it had. For the rest of that window the gateway answered every request itself, immediately, with 503 auth_unavailable / "no auth available," without ever contacting NVIDIA. That's the 6 ms.
The arithmetic downstream was already decided before the cooldown even started. My retry policy backs off at 1 s and then 3 s, both well inside the 60-second window the cooldown holds open — not because the vendor failed again, but because a minute-long timer doesn't care how fast a client retries. The seed for a given episode was ordinary on its own — NVIDIA NIM's free tier returning "Service temporarily overloaded," or once a 302-second hang ending in a 504 — but the blackout that followed had nothing more to do with how bad that seed event was.
200 at the same moment the gateway refused the request. The outage never left my machine.disable-cooling on globally — wrong for any provider that actually holds two or more keys, where the cooldown is doing its real job. The setting belongs per provider, sized to how many keys that provider has.410 means a model has genuinely reached end of life — permanent, needs its alias redirected, not retried. A 429 runs through the gateway's own quota path. Folding either into "the leg is flaky" erases what tells you what to do next.For a provider with one key — the default case — the fix is disable-cooling: true on that provider's block only:
openai-compatibility:
- name: "nvidia-nim"
base-url: "https://integrate.api.nvidia.com/v1"
disable-cooling: true
api-key-entries:
- api-key: "<redacted — injected from env>"CLIProxyAPI hot-reloads its config file via a watcher — the log records "config successfully reloaded" — no restart, no gap for other consumers. I applied this at 2026-09-05 20:41:58 and reran the probe: PASS 3/3 — nano in 3972 ms, super in 558 ms, and the 49b model correctly back with 410, its expected end-of-life answer, not a new failure.
For a provider where I can hold two or more keys, the fix runs the other way: add another API key, let the cooldown become a real rotation, and leave it on. Custody matters here for the same reason it mattered when a metered API key, exported globally in a shell profile, turned out to be inherited by unrelated tools that had nothing to do with the one originally suspected (The Key's Owner Wasn't the Spender) — each key gets injected per consumer from the environment, never printed in the clear.
Either way, retries get exactly one owner. Mine live in the consumer's roster — retry_transient plus its backoff — and nowhere else. Stack a retry loop on top of a gateway-side cooldown and you have two layers with two different budgets deciding the same failure — and here the smaller budget was always going to lose.
The diagnostic itself is just the latency column, grepped and tailed:
grep -E "gin_logger.go:[0-9]+\] 503 " ~/path/to/gateway/launchd.out | awk -F'|' '{print $1 "|" $2}' | tail -20Run against the history, the same signature — one slow 503, then a run of 1–2 ms 503s — showed up on 2026-06-25, 2026-07-01, and 2026-08-26, and again on 2026-08-27 and 2026-09-02, the two episodes I'd gone looking for. The access log held 37 lines like that; most of them carried the signature. It had been there the whole time I was blaming the free tier.
This sits in the same family as a tail -20 that couldn't emit until EOF, with the pipe's write-end held open by an inherited descriptor — the tool was fine there too, and an observation pipeline manufactured the failure (For Weeks I Logged a CLI as Flaky). The roles are flipped here: the vendor was fine, and my own gateway layer manufactured the outage. Same instinct to blame the far end of the wire first; the defect was one layer closer to home both times.
503's latency in the log reads in the hundreds of milliseconds or seconds, that request went upstream and came back with a real failure — a vendor problem again, not a local blackout.None of the individual facts here were hidden. The code was right there to read, the log line carried a latency column the whole time, and the vendor answered honestly when I asked it directly. The signature sat in that log for about three months before I read it as a local blackout instead of vendor instability — not for lack of evidence, but because I hadn't yet thought to distrust the layer I'd installed to make the fleet steadier in the first place.
The gateway's cooldown logic treats any 408, 500, 502, 503, or 504 from a provider as a reason to rest that key for one minute before trying it again. If the provider only has one API key configured, that minute leaves the provider with zero available auth, so the gateway answers every request itself with a 503 auth_unavailable error instead of contacting the vendor. Once the minute passes, the key is available again and requests go through normally.
Check the latency recorded in the gateway's access log for that request. A 503 that took hundreds of milliseconds to a few seconds actually made a round trip to the vendor and got a real failure back. A 503 that came back in only a few milliseconds never left the gateway; it was rejected locally before any network call was made. A direct probe to the vendor at the same moment can confirm this: if it returns 200, the vendor was fine.
Only for a provider that has exactly one API key, where a cooldown can't behave like a rotation because there's no second key to switch to. For a provider that holds two or more keys, the cooldown is doing its intended job, moving traffic off a key that just failed, so turning it off there removes a legitimate protection instead of a false one. The setting should be applied per provider, based on how many keys that provider actually has.
The gateway's cooldown window is a fixed 60 seconds, and a typical client retry backoff is measured in single-digit seconds, so every retry lands inside that window by construction. The retries aren't failing because the vendor is unstable; they're failing because the client gave up and tried again long before the local cooldown timer had any chance to expire.
In this setup a 503 from the gateway can mean either a real upstream failure or the gateway's own cooldown rejecting the request locally, and the access log's latency column is what tells them apart. A 429 runs through a separate quota-limit code path and isn't related to the cooldown at all. A 410 means the specific model has permanently reached end of life; it isn't a transient failure and won't recover with a retry, it needs its alias redirected to a different model.