I was writing a briefing about whether it was safe to just wait until September. A governance kill-condition in one of my projects is evaluated over non-overlapping 56-day blocks, and the first block closes on 2026-09-27. Before I could recommend waiting, I wanted to check that the series feeding that condition was actually accumulating. A predecessor note covered a related but different failure — gates whose production call site simply doesn't exist, findable once you know to ask a database instead of a shell. This one is not that. Every static check that note would have run against this system comes back clean.
A nightly job at 03:00 mirrors a corpus of session transcripts to an external volume, then runs a sequence of follow-up steps. One of those steps computes the meter that the kill-condition reads. When I queried the table holding that meter's history, it had exactly one row. That row had been written by hand the previous afternoon, in the session where I built the meter — not by the schedule that was supposed to add one every night. The step had been wired into the job that same day, so the WARN line below is its entire scheduled career: one attempt, one failure.
The archive log already had the explanation sitting in it, in plain WARN text:
[archive_corpus] WARN 2026-08-03T18:00:05Z: decision_trace failed (archive succeeded) — ModuleNotFoundError: No module named 'pydantic'
The root cause was ordinary once I looked straight at it. launchd hands a scheduled job PATH=/usr/bin:/bin:/usr/sbin:/sbin and nothing else. A bare python3 inside the script resolves, under that PATH, to Apple's system Python — which ships with no third-party packages. The framework Python on the same machine, the one that answers to python3 in an interactive terminal, is on 3.13 and has pydantic 2.12.5 installed. Every time I'd run the script by hand to sanity-check it, it worked, because my shell's PATH was never the scheduler's PATH. The check I trusted and the environment that mattered were two different machines wearing the same hostname.
None of these were sloppy decisions. Stacked, they made a step that had never once succeeded on schedule look exactly like one that had.
"status": "ok" regardless.Had I not checked, the 56-day block would have closed on 2026-09-27 with a single data point in it, and the kill-condition would have been permanently INDETERMINATE — not failing, not passing, just structurally unable to ever be evaluated, because you can't derive a rate from one measurement. "Wait until September" was, without my knowing it, a plan that produced nothing. The schedule wasn't slow. It wasn't running at all.
I want to be exact about how I found out, because it wasn't a detector. I noticed one night in, on the morning after the only scheduled run that step ever attempted, and only because I'd sat down to answer an unrelated question about whether waiting was safe. There was nothing in the system that was going to raise its hand over the remaining 55 nights. Catching it on night one was luck; the fix below is the part that isn't.
Two parts. The first closes this specific hole; the second is the one I'd reuse anywhere.
Resolve the interpreter by capability, not by path. Pinning a version path just relocates the failure to the next Python upgrade, which will die the same silent way. Instead, probe for an interpreter that can actually import the dependency you need:
resolve_py_with_deps() {
local cand
for cand in /Library/Frameworks/Python.framework/Versions/Current/bin/python3 \
/opt/homebrew/bin/python3 /usr/local/bin/python3 python3; do
if command -v "$cand" >/dev/null 2>&1 && "$cand" -c 'import pydantic' >/dev/null 2>&1; then
command -v "$cand"; return 0
fi
done
return 1
}
If nothing qualifies, that's its own state — nopy — distinct from a computation that ran and failed, and distinct from a step that was deliberately skipped.
Fail-soft is correct design. Fail-silent is not. If a step is allowed to fail without killing its parent job, its state has to land somewhere the parent's own monitor already looks. Don't invent a new channel for it — nothing reads new channels. The heartbeat file already existed, so the state rides along in it:
write_heartbeat "ok" "files=${DST_FILES} new=${NEW_COUNT} shrunk=${SHRUNK_COUNT} ing=${ING_STATE} dt=${DT_STATE}"
Then the monitor has to actually read those keys, and complain when they're absent entirely — because a monitor that can quietly stop monitoring is the identical defect, one level up.
This is the part that generalizes past this one job.
env -i PATH=/usr/bin:/bin:/usr/sbin:/sbin ./the_job.shlaunchctl kickstart -k gui/$(id -u)/com.example.job. In my case the reproduction wrote the second row and the real scheduled run wrote the third, taking the meter's history from 1 row to 3, and the heartbeat came back as files=3773 new=5 shrunk=2 ing=ok dt=ok.SELECT run_id, computed_utc FROM meter_run ORDER BY run_id DESC LIMIT 3; If a job that's supposed to be nightly has one row, and you're the one who wrote it, the schedule is dead — no matter what the code, the crontab entry, or the documentation says.Two earlier notes in this series were about gates whose production call site didn't exist at all — grep found nothing, and a fill-rate query on the underlying table found the rest that grep missed. This one is close to the inverse. Grep finds the call site here; it's right there in the script. The scheduler entry exists. Run the script yourself and it works. Every static thing you could check about this system is correct. The defect lives at runtime, in the gap between an interactive shell's environment and an unattended scheduler's environment, and a fail-soft design is what keeps it from ever surfacing as a failure. It's the only variety in this family where the monitor doesn't just miss the problem — it actively reports green while the problem is happening.
Fixing this pushed me to generalize a check that had been hardcoded to look at a single row of a governance table: does every kill-condition in the table actually have a way to be measured? Turning that single-row check into a full-table sweep immediately surfaced two more instances sitting in the same table, neither related to the launchd issue:
The lesson underneath both: don't try to parse the prose of the condition itself. Condition text mixes real values, object names, and names of things entirely outside the system, so pulling identifiers out of a sentence can't reliably tell those apart — not as an implementation gap, but in principle. The fix instead is a declared meter column, a small controlled vocabulary that a checker validates against instead of inferring from prose:
table:NAME (the table exists and has at least one row) · table?:NAME (the table exists; zero rows is itself a legitimate observation) · guard:NAME · script:PATH · os:... · and n/a: followed by a mandatory reason.
Making that reason mandatory is what removes the silent exemption — you can no longer write n/a and move on without saying why. One aggravating pattern gets flagged automatically: if a condition's declared meter is n/a: while its status cell starts with a number, that's the exact defect above, wearing a different table. Run against the real table, this turned up 13 conditions total, of which 4 had automated meters and 9 were explicitly n/a: with a written reason attached to each. Having that number visible, instead of assumed, was the actual point of doing the sweep. I also wrote seven mutation tests, each one deliberately breaking a single rule — deleting the column, pointing at a table that doesn't exist, an empty reason, a status cell that's numeric where the meter says n/a: — and asserting that the checker fails in each case, plus two controls that assert it still passes on the pristine table and tolerates a legitimately empty one. A checker you've only ever watched pass is itself unverified; so is one you've only ever watched fail.
Q. Why did the job's heartbeat stay green while the step failed?
The failing step was deliberately fail-soft, so a failure in it could not be allowed to kill the parent job or its archive run — that part of the design was correct. But the failure was absorbed into a single WARN log line, and nothing propagated it into the heartbeat file the job already wrote. The heartbeat only ever reported the parent job's own status, which stayed ok because the parent job itself never crashed.
Q. Why does a bare python3 break under launchd but work fine when run by hand?
launchd runs scheduled jobs with PATH set to only /usr/bin:/bin:/usr/sbin:/sbin. A bare python3 inside a script resolves under that PATH to Apple's system Python, which has no third-party packages installed. An interactive terminal has a much larger PATH that resolves python3 to a framework installation with the needed dependency already present, so running the same script by hand always succeeds — which makes that manual test worthless as proof the scheduled version works.
Q. How do you verify a "nightly" job is actually running nightly?
Query the table or file the job is supposed to be writing to and compare its declared cadence against the actual interval between rows: SELECT run_id, computed_utc FROM meter_run ORDER BY run_id DESC LIMIT 3. If a job that is supposed to run nightly has produced one row, and that row was written by hand during a session rather than by the schedule, the schedule is dead regardless of what the crontab entry, the code, or the documentation claims.
Q. How is this different from a dead gate that grep can find?
In a dead gate with no call site, grep finds nothing because the wiring genuinely doesn't exist. Here grep finds the call site, the scheduler entry exists, and running the script by hand succeeds — everything static is correct. The defect only exists at runtime, in the gap between an interactive shell's environment and an unattended scheduler's environment, and it is the only variety in this family where the monitor doesn't just miss the problem but actively reports green while it is happening.
Q. Why not just let checkers parse the condition text to find its meter?
Condition text mixes real values, object names, and names of things entirely outside the system, so extracting identifiers from a sentence can't reliably tell those apart, not as an implementation gap but in principle. The fix is a declared meter column with a small controlled vocabulary — table:NAME, table?:NAME, guard:NAME, script:PATH, os:..., or n/a: followed by a mandatory reason — that a checker validates directly instead of inferring from prose.