Green for four days while nothing shipped: a reader rebutted my monitoring post, and he was right

← hexisteme · notes · July 14, 2026 · 8 min read

I published a post arguing that for a small agent fleet you don't fully control, polling on-disk state beats an event bus. The slogan: "state is truth, events are rumors" — an event only exists if the emitter survived to send it; state is evidence left behind regardless.

A reader rebutted it within hours. He was right, my own code proved him right, and the failure he described had already been running in production. This is the trail.

The line that convicted me

Here is the freshness check from the original post:

status = {
    "alive":   process_is_running(job),             # ps / launchctl — external
    "fresh":   newest_output_mtime(job) > expected,  # the job's OWN output dir
    "healthy": health_check(dependency),            # poll endpoint — external
}

Two of those three lines observe something the job does not own. The middle one globs the job's own output directory. I shipped it in a post about not trusting components to report on themselves.

The rebuttal

@anp2network wrote:

Then he named the failure mode: liveness without progress. A job wedged in a retry loop keeps appending to its own log. The log grows. The mtime advances. The process is alive. Every freshness check stays green. Nothing useful happens for a day. His rule: every job must move something it does not own. Score freshness on the effect instead of the report.

I had spent a whole post arguing that a bus is only as honest as its least-reliable emitter, then shipped a poller only as honest as its least-honest writer. Same disease, different transport.

What I did instead of writing a reply

The instinct when someone rebuts you in public is to reach for the strongest counter-argument. That instinct is a bug. It answers "how do I respond?" when the only question that matters is "is he right?" Those questions have different answers, and only one is worth anything.

So before writing a word of reply I ran the rebuttal past two other model families with an adversarial prompt — judge whether the commenter is correct; do not defend the original post; prefer falsification over rhetoric — plus my own pass reading the code.

Gemini was the only leg that pushed back on the commenter, and it pushed back on the cure rather than the diagnosis:

It also produced a sharper defense of my post than I would have written for myself:

The honest caveat about that table

It would be a tidy story to say "three models, only one caught the counterexample — that's why you fan out." That story is partly an artifact of my own sloppiness.

The two prompts were not identical. The Gemini prompt had a JSON slot named where_the_commenter_overreaches. The nemotron prompt did not — its schema was commenter_correct, strongest_point_for_commenter, strongest_defense_of_original_post, what_the_author_should_concede, confidence. Every field there leans toward agreement. There was no field in which nemotron could have told me the commenter was overreaching, and it duly did not tell me.

So I can't claim Gemini is the smarter model. The claim I can defend is narrower and more useful: the leg with a slot for disconfirmation produced disconfirmation, and the leg without one didn't.

So I went back and measured it rather than asserting it — same model, same question, one field different, repeated runs. The slot moves the verdict, and the run-to-run noise turned out to be large enough that my one-shot comparison of two models was never evidence of anything in the first place. That experiment, and what it means for anyone wiring up an LLM as a judge, is its own post: I Asked Three Judges If I Was Wrong. One JSON Field Decided the Answer.

Which is, I realized with some irritation, the same bug as the monitor.

The bug, in the actual catalog

My fleet dashboard reads a catalog.json telling it, per job, where to find evidence of life. The entry for the dev.to publishing job said:

"com.hexisteme.devto-crosspost": {
  "report_globs": ["~/fleet/crosspost/publish-log.txt"]
}

The publishing job proves it is healthy via the mtime of the log file it writes itself.

And I had already been bitten — I just hadn't recognized the bite. For four consecutive days in July the pipeline published nothing. Here is the log it was judged by:

2026-07-05 09:00:05 KST queue empty — skip
2026-07-06 09:00:05 KST queue empty — skip
2026-07-07 09:00:05 KST queue empty — skip
2026-07-08 09:00:05 KST queue empty — skip

Read those four lines carefully: the whole lesson is in them. The log is honest. It says in plain text, every morning: I did nothing. The drain ran on schedule, found an empty queue (the upstream step that refills it had quietly stopped), wrote a line saying so, and exited zero. The mtime advanced. Freshness went green.

Nobody lied to the monitor. It asked "has this file changed recently?", was truthfully told yes, and that question had no bearing on whether anything was published. I found the outage by looking at the published count — not at the monitor whose job is to tell me about outages.

The fix: score the effect

The dashboard already had health_proc: a pgrep pattern proving process liveness from the process table, which the job does not own. The fix is its twin — a field proving real-world progress by sampling something the job also does not own.

"effect_probe": {
  "cmd": "ls ~/fleet/crosspost/published/*.md 2>/dev/null | wc -l",
  "expect": "increase",
  "label": "cumulative dev.to publishes"
}

The probe samples an integer each poll into a history file. The anomaly check is then deliberately blunt:

if effect_probe is not None:        # NOTE: mtime is not consulted in this branch
    recent = [e for e in history if e["ts"] >= now_ts - 1.5 * period]
    if len(recent) < 2:
        return None                 # cold start / too few samples — stay quiet
    if recent[-1]["value"] > recent[0]["value"]:
        return None                 # it moved — really moved
    return f"stalled: {label} flat for {hours}h (process is running)"

When a probe exists, mtime is not consulted at all — not weighted lower, not read. The job's own artifacts are inadmissible as evidence about that job; any blend would let the self-report leak back in.

Respecting the counterexample

Gemini's objection changed the design. The tempting move after a correction is to over-apply it — force an effect_probe onto every job and declare victory. That would break exactly the jobs it named — a cache warmer, a log rotator, a terminal sink whose whole purpose is to leave something on local disk. They have no downstream artifact to move, so a strict "every job must move something it does not own" makes them unmonitorable rather than merely hard to monitor.

So effect_probe is optional, and jobs without one keep the mtime path — but not quietly:

freshness_source = "effect" if entry.get("effect_probe") else "self-report"

The card renders a small self-reported badge, so the monitor tells me per job how much its own green light is worth. A weak signal labeled weak is a different object from a weak signal labeled strong — the second one cost me four days.

The probe I refused to write

I wired three probes: the dev.to publisher (count of published files), the citation ledger (rows in an append-only ledger), and the blog publisher — where the obvious probe would have been a bug. "Count the local HTML files it generated" is a file the job writes, in a directory the job owns: precisely the bug I was fixing, wearing a different hat. It would have reproduced the self-report failure inside the fix for it.

So the probe instead reads a marker whose write is gated behind a remote check — the publisher fetches the public blog's front page back over HTTP, confirms the post's title is really there (_verify_published_on_blog), and only then writes the marker. The effect scored is "a remote server I do not own serves a page containing this title", anchored in something no local process can fake by merely running.

What does this file's existence actually prove? is the whole discipline — and it is invisible in the file's name and path. You have to know who writes it, and when.

Does it work?

Replaying the real code against the scenario that fooled it — effect flat at 13 publishes for four days, log mtime one hour old (deck strings are Korean; translated here):

[A] effect flat 4d + log mtime 1h ago (fresh):
    old code (mtime only) -> None
    new code (effect)     -> 'stalled: cumulative dev.to publishes flat for 96h (process is running)'

[B] effect increased 9 -> 13 over the window (real publishes):
    new code (effect)     -> None

[C] cold start (probe deployed 30 minutes ago):
    new code (effect)     -> None

[A] is the four days, reproduced: the old code returns None — green — for the exact scenario that hid the outage. [B] confirms it stays quiet when work is genuinely happening.

[C] is a false alarm I did not predict: a freshly deployed probe has no history spanning its staleness window, so "nothing increased in 30 minutes" is not yet a meaningful verdict — without a guard it reads STALE. I found it by running the poller against the live catalog rather than reasoning from a fixture. The same lesson in a third costume: I saw it in the effect, not in my model of the effect.

What survives

I still run pull, for the reason Gemini articulated better than I did. But that is an argument for external observation; pull is merely the cheap transport for it. "State is truth" is false as written: a file the job wrote is a rumor the job told the filesystem. The slogan holds only when the state was authored by someone other than the job being judged.

I have not retracted the post. I annotated it in place, with the correction and a link to the commenter, because a post that quietly becomes correct teaches nobody anything.

Two rules came out of this, and they turn out to be one rule:

The best comment I have ever received cost me a slogan, and surfaced four days of silent failure I had already paid for without noticing. Cheap at the price.

FAQ

Q. Why does my monitoring dashboard show green when the job did nothing?
Because you are scoring the job's report, not its effect. If freshness comes from the mtime of a file the job itself writes — its log, its output directory — then a job that runs, does nothing, and writes a line saying so still advances that mtime. The signal is truthful and worthless: it answers "did this file change?", not "did anything happen?". My publishing job did exactly this for four days. The log grew every morning and said, in plain text, "queue empty — skip". The monitor stayed green. Fix it by scoring an effect the job cannot produce by merely running: a downstream artifact count, rows appended to a ledger someone else owns, or a remote page fetched back and verified.

Q. What is liveness without progress?
A failure mode where every health signal is green while nothing useful happens. A job wedged in a retry loop keeps appending to its own log; the log grows, the mtime advances, the process is alive, and every freshness check passes. The monitor sees liveness and infers progress, but the two are independent. It is caught by watching a downstream count that has not moved — never by watching the job's own output, which is precisely the thing that keeps moving.

Q. Is polling on-disk state really better than an event bus?
The conclusion survives but the usual reason for it does not. "State is truth, events are rumors" is false as written: a file the job wrote is a rumor the job told the filesystem, and it is only as honest as the writer — the same weakness people attribute to a bus. The real axis is self-report versus external observation. Pull still wins for components you cannot instrument, because file writes and process liveness are side effects of execution rather than voluntary emissions, so a poller can observe a component that never agreed to be observed and a bus cannot force a liar to emit. Pull is simply the cheap transport for external observation, not truth by itself.

Q. Should every job move something it does not own?
As a heuristic it is excellent; as a hard rule it over-generalizes. It would make terminal sinks unmonitorable — a cache warmer, a log rotator, or a self-contained background worker legitimately has no external artifact to move. The rule that survives is: score the effect, and define the effect from the job's purpose rather than from ownership. Where a job genuinely has no external effect, keep the weaker mtime signal but label the card self-reported instead of pretending it means more than it does. A weak signal labeled weak is a different object from a weak signal labeled strong.

Q. How should I check whether a commenter who rebutted my post is right?
Ask "is this person right?" rather than "how do I rebut this?" — they have different answers and only one is worth your time. Then ask it somewhere that can answer no. Run the rebuttal past models from other vendors with an adversarial prompt that forbids defending your original post, and make sure the response schema contains an explicit slot for disconfirmation, such as where_the_commenter_overreaches. A schema whose every field leans toward agreement can only return agreement — the same bug as a monitor that reads a job's own log.

Related notes

• Why I rejected an event bus for my solo agent fleet: state is truth, events are rumors — the post this one corrects
• Do multiple personas on one LLM give real diversity, or do you need different model families?
• Verify the output surface: a pipeline isn't verified until you look at what it actually published