UNMEASURABLE verdict for exit 3, moved the one outlier gate from 1 to 3, and kept exit 2, a real finding, as a regression. With no exit code in the log, a cache side effect and the list of what runs the gate were what pointed to a run that never measured.On October 23 the daily drift report for my gate suite contained this line (translated from Korean):
check_offer_subject_alignment.py: PASS → REGRESSION ← a fix you'd made broke again.That note is the loudest thing the drift watcher can say. It is reserved for the case where something I had already fixed has stopped working. The same gate had printed it on October 18, gone back to PASS on October 21, and printed it again on the 23rd. Before that day's fix, the drift log already held ten verdict changes for this one gate, going back to September 7.
When I opened it, the live data the gate checks was nowhere near the failure it guards against. What had failed was the gate's ability to look. The suite had no verdict for that, so it used the loudest one it had.
The two Stop hooks behind this note are on GitHub under MIT: hexisteme/hard-gate-hooks. They ship with their tests and a read-only scanner that prints what they did on your machine, not mine — including the case where it tells you they aren't worth wiring up yet. No email, no signup.
The gate is one of 22 scripts in a suite I run over my publishing pipeline. This one checks, against live data, that a link I put in some dev.to posts sits on the posts it belongs on (the ones tagged agents) rather than drifting onto unrelated ones. It reads articles/me/published, then makes one comments?a_id= call per commented article: about 100 live API calls per run. A launchd job runs the suite once a day; on October 23 the run landed at 00:19 UTC.
The gate ends with one of three exit codes:
0 ALIGNED: the link sits where it should2 REVERSED: the link has drifted onto the wrong posts, which is a real finding1 UNMEASURABLE: it could not read the live dataInside the gate those three meanings are distinct. The runner only saw two of them.
The suite runner turns each gate's exit code into a verdict. Before the fix, the decision was this (comments mine):
def judge_gate_outcome(exit_code, declared_reason, output="", expect=None):
if exit_code == 0:
return STALE_BLOCK if declared_reason else PASS
if exit_code == 124:
return CRASHED # the runner's own timeout
if not declared_reason:
return REGRESSION # any other undeclared non-zero lands here
if expect and expect not in output:
return REGRESSION
return BLOCKEDdeclared_reason comes from a file, gate_human_blocks.json, where I can declare that a gate is red because it is waiting on a person. A declared gate that fails reads BLOCKED instead of REGRESSION, and a declared gate that passes reads STALE_BLOCK, so exemptions can't outlive their reason. This gate had no declaration. Exit 1 and exit 2 went down the same branch: "I couldn't read dev.to at 00:19" and "the link has drifted onto the wrong posts" both came out as REGRESSION, and the watcher put its broken-fix note on both.
The drift log stores verdicts only. It keeps neither the exit code nor the stderr of each scheduled run, so I could not look up what happened at 00:19. I had to reason from a side effect.
The gate caches its last successful live measurement and reuses it for 24 hours. It writes the cache only right after a successful fetch. If loading the API key or the fetch raises an error it expects, such as a network error or a timeout, it prints UNMEASURABLE to stderr and exits 1 before the write. Any other exception in that span also exits 1, without that line, and also before the write.
At 13:32 UTC I ran the gate by hand, without the flag that forces a refetch. It reported its data as 0.0 hours old: it had found no cache young enough to reuse and had fetched everything live. As far as my session logs and the scheduler list show, nothing else had run the gate in between. The daily launchd job is the only scheduler that calls it, and no session of mine had run it since the previous day's scheduled run. That leaves two ways the 00:19 run could have produced a verdict from data:
Neither fits. What fits is a run that failed somewhere before the cache write, before it had any data. The fetch from dev.to is the only network step on that path, so the likeliest reading is that the gate could not read dev.to at 00:19. The logs can't tell me which error it was, or whether the gate caught it.
Here is what the afternoon run measured:
agents posts carry the link, against 32.0% (32 of 100) of the other postsThat is the alignment the gate exists to protect, by a wide margin. At 00:19 the gate never got to look, and the suite reported that as the gate finding a problem.
The obvious patch is one line in the runner: read exit 1 as unmeasurable. That would have broken other gates. Exit codes are a protocol between the gates and the runner, and in this suite 1 already meant something else. check_reader_turn.py, for one, returns EXIT_MISSING_TURN = 1 when it finds a real defect. Remapping 1 suite-wide would have quietly downgraded that gate's real failures. Python also exits 1 on an uncaught exception, so 1 is what a crashing gate produces by default.
The suite already had a convention. It just wasn't written down where the runner could enforce it. Seven other gates declared EXIT_UNMEASURABLE = 3. This gate declared the same constant with a different value, 1, and it was the only one that did.
Three small changes:
EXIT_UNMEASURABLE = 3 as the suite-wide convention and returns a new verdict, UNMEASURABLE, for exit 3. A human-block declaration does not turn it into BLOCKED. A measurement gap is not a failure waiting on a person, and letting a declaration absorb it would hide a dark gate behind a reason written for something else. Exit 2 still returns REGRESSION, because a gate that found a bad state should still alarm.EXIT_UNMEASURABLE from 1 to 3, joining the other seven.The runner now reads:
EXIT_UNMEASURABLE = 3 # suite-wide: "I could not measure", not "I found a problem"
def judge_gate_outcome(exit_code, declared_reason, output="", expect=None):
if exit_code == 0:
return STALE_BLOCK if declared_reason else PASS
if exit_code == 124:
return CRASHED
if exit_code == EXIT_UNMEASURABLE:
return UNMEASURABLE # a declared human block does not convert this
if not declared_reason:
return REGRESSION
if expect and expect not in output:
return REGRESSION
return BLOCKEDThree tests keep it from drifting back:
EXIT_UNMEASURABLE with a value other than 3, which is the exact drift that caused thisThe guard test is short:
def test_every_gate_declaring_unmeasurable_uses_the_shared_exit_code():
offenders = []
for gate in discover_gate_scripts(REPO_TOOLS):
text = gate.read_text(encoding="utf-8")
m = re.search(r"^EXIT_UNMEASURABLE\s*=\s*(\d+)", text, re.MULTILINE)
if m and m.group(1) != "3":
offenders.append(f"{gate.name}={m.group(1)}")
assert not offendersThe tools test suite passed, 113 tests. I re-ran the drift watcher. It recorded REGRESSION → PASS for the gate and the false alarm cleared.
A check can look and find everything fine, look and find a problem, or fail to look at all. A runner that only has words for the first two folds the third into one of them.
Fold it into the first and a check that cannot run reports something that looks like a result. In A Measurement That Never Ran Became a Verdict, every read failed and the script still printed a normal-looking verdict built entirely out of failed reads; part of that fix was a guard that refuses to turn a failed read into a zero. In The Scheduled Step Failed on Night One. The Heartbeat Stayed Green., a nightly job's step failed on its first scheduled run while the job's own heartbeat kept reporting healthy.
Fold it into the second and you get what I had: a gate that cries regression on every transient outage. An alarm that is wrong often enough gets read as noise, and a real reversal printed on the same line, with the same note, gets read the same way. The exit code is also where Exit 0, Empty stdout: the Quota Died on stderr goes wrong, from the other side. There, a CLI worker hit its usage cap and exited 0 anyway, so a dispatcher that only checks the exit code would log a completed task. In that case the process sent the wrong code. In my suite the gate sent the code it meant, and the runner read it with the wrong dictionary.
Test runners already draw this line. pytest has a separate exit code for "no tests were collected", distinct from the one for failing tests: a run that tested nothing is not reported as a run where tests failed. A gate suite needs the same vocabulary. The measurement gap gets a verdict loud enough to see, because the gate went dark and someone should check live access, and quiet enough not to pass for a regression.
UNMEASURABLE, and the same failure would still read REGRESSION after this fix. Only the errors the gate catches now become UNMEASURABLE.EXIT_UNMEASURABLE constant. A gate that used 3 for a real failure without declaring that constant would now be read as unmeasurable, and the test would not notice. A grep of the current gates found none that does, but nothing stops a future one.The suite runner mapped every non-zero exit other than its own timeout code, 124, to REGRESSION when no human block was declared. The gate exited 1 when it could not read live dev.to data, so the runner reported it the same way as exit 2, which meant the gate had found a real reversal.
Reason from side effects and from what else could have run. This gate writes its cache only right after a successful live fetch and reuses it for 24 hours. A manual run at 13:32 UTC, without the refetch flag, found no reusable cache, so the 00:19 UTC run had not fetched successfully; reusing the previous day's cache would have reproduced that day's PASS, not a REGRESSION. The likeliest reading is that the 00:19 run failed before it had any data.
In this suite 1 already means a real failure for other gates, such as EXIT_MISSING_TURN in check_reader_turn.py, and Python exits 1 on an uncaught exception. Seven other gates already declared EXIT_UNMEASURABLE = 3, so the outlier was aligned to them, and a guard test now fails if any gate declares a different value.
It makes the suite quieter on measurement gaps on purpose, so UNMEASURABLE sorts just below REGRESSION and CRASHED, the suite report still prints the tail of the gate's output, and a human-block declaration cannot convert it. A gate that must hard-fail when it cannot measure should not use exit 3.
The 00:19 diagnosis is an inference, and the transient cause (DNS, a rate limit, a timeout) is not pinned. If that failure was an exception the gate doesn't catch, it exited 1 and would still read REGRESSION after the fix. The drift log keeps no exit codes for its earlier entries, so some of those REGRESSION verdicts may have been real reversals.