I had weeks of notes saying a vendor CLI "works sometimes." Availability seemed to depend on the hour. I'd started routing around it.
The CLI was fine every time. The bug was in how I was watching it.
Every one of those observations came through some variant of this:
perl -e 'alarm 45; exec @ARGV' agy models 2>&1 | tail -20Timeout on the left, truncate on the right. This idiom hung indefinitely three separate times, in three separate sessions, on two different subcommands.
agy models | tail -20, PIDs 96135→96138. No response.agy --print ... | tail -120, PIDs 87045→87048.Sat there for 81 minutes.
agy models 2>&1 | head -20, PIDs 14195→14198. Hung past 33 minutes.Every one of them, re-run without a pipe, returned in seconds:
agy models > /tmp/agy-check.out 2>&1 # EXIT=0, a few seconds, every timeTake the blocked head/tail and look at what it's reading from:
lsof -p 14198 | grep -i pipe
# fd 0 → 0x4881848dc2e3034fNow find anyone else holding that same pipe address:
ps -ef | grep -i npm # what else spawned around then?
lsof -p 14251 | grep -i pipe
# fd 5, fd 6 → 0x4881848dc2e3034f ← same pipe, write endPID 14251 was npm exec xcodemcp@latest. It had nothing to do with my command. It had spawned in the same second as the head.
The other two matched the same way. In A, the culprit was npm exec xcodemcp@latest (PID 96158, spawned at 10:40:22 — within seconds of the tail). In B it was npm exec @modelcontextprotocol/server-redis (PID 87098), confirmed by matching the pipe address 0xe3e977fcc256b0c7 on both ends.
Three facts, and it falls out:
1. tail -N without -f can't emit anything until EOF. It has to know which lines are the last N. head -N looks safe by comparison — it should exit as soon as it's seen N lines — but if total output is fewer than N lines, it also waits for EOF. Both are vulnerable. (Before case C, my notes only mentioned tail, so the head variant wasn't covered by the rule I'd written. The rule was narrower than the mechanism.)
2. EOF arrives when the pipe's write end reference count hits zero — not when your command exits. A process spawned around the same moment can inherit that write-end file descriptor (a missing close-on-exec, as best I can tell) and hold it open indefinitely.
3. So head/tail blocks forever, waiting for a signal that a completely unrelated process is holding hostage.
The kill switch doesn't save you either. alarm 45 reaches the process exec replaced — the vendor CLI. But the CLI already exited. What's blocking is a descriptor held by a different process, so raising the timeout does nothing. I'd been raising it for weeks.
This is the part that made it hard to see.
If your caller just waits, you observe an infinite hang. If your caller has its own timeout, you observe empty output — and empty output reads as "that leg is dead / the vendor is down." Same bug, two entirely different verdicts, and the second one is the one that got written into my notes as evidence about a third party.
Case B was worse still. When I killed the process holding the write end, tail finished within two seconds — and what came out wasn't a successful response. It was Error: timeout waiting for response, 36 bytes.
The original command had failed, immediately, and said so. That 36-byte failure message then spent 81 minutes stuck in a pipe.
That's worth stating precisely, because it's not the usual failure mode. The bug did not turn a failure into a success. It blocked a failure signal from arriving at all. A wrong answer and no answer are different problems, and the second one is the one that trains you to distrust a tool that was telling you the truth on time.
Don't attach a truncator to a live pipe. Land the output in a file, then cut it.
# Vulnerable — head and tail alike (head waits for EOF when output < N lines)
perl -e 'alarm 45; exec @ARGV' agy models 2>&1 | tail -20
perl -e 'alarm 45; exec @ARGV' agy models 2>&1 | head -20
# Safe — receive completely (process exit closes the fds), then cut
perl -e 'alarm 45; exec @ARGV' agy models > /tmp/agy-check.out 2>&1
tail -20 /tmp/agy-check.outStructurally better: skip the shell pipe. A runtime that defaults descriptors to close-on-exec closes the inheritance path entirely. Python has since PEP 446:
import subprocess
r = subprocess.run(["agy", "models"], capture_output=True, text=True, timeout=45)
print(r.stdout[-2000:])This only fires when some other process — an MCP server, an editor helper, a daemon — happens to spawn in the same instant. Most of the time nothing does, and the command returns fine.
Which means "not reproducible" was never evidence of absence here. It was evidence that the race is a race. Weeks of intermittent success is exactly the signature, and I read it as a property of the vendor instead of a property of my own plumbing.
The portable version:
When a tool feels like it "works sometimes," suspect the observation pipeline that produced that judgement before you suspect the tool.
Availability may not be an attribute of the tool at all. It may be a state of your machine at that moment — and if your measurement apparatus is what's flaky, every measurement it produces is about the apparatus.
# 1) read-end pipe address of the blocked head/tail
lsof -p <blocked_pid> | grep -i pipe
# 2) find another process holding the same address as its write end
ps -ef | grep -i npm # check spawn times of candidates
lsof -p <candidate_pid> | grep -i pipe
# matching pipe address on both = confirmedThe close-on-exec diagnosis is an inference, supported by three independent cases showing the same correlation — simultaneous spawn plus matching lsof pipe address — not a root cause I traced to a line of source.
The falsifier is clean: if the same hang reproduces with no pipe at all (cmd > file 2>&1), this diagnosis is wrong and I need to start over. Across all three cases, file redirection returned in seconds, every time. That's where the attempts to break it stand so far.
The pipe stays open as long as any process holds its write end. A process spawned around the same moment can inherit that descriptor, so the reference count never reaches zero and tail waits for an EOF that will not come. Confirm it with lsof: compare the read-end pipe address of the blocked tail against the descriptors of other processes that started at the same time.
No. head -N exits early only once it has actually seen N lines. When the total output is fewer than N lines it also blocks until EOF, which makes it vulnerable to exactly the same inherited-descriptor stall. A rule written only about tail leaves the head variant uncovered.
A wrapper like alarm reaches the process that exec replaced — the command itself — and that command has already exited. What blocks is a descriptor held by an unrelated process, so raising the timeout changes nothing. Redirect to a file and truncate afterwards, or use a subprocess API whose runtime defaults descriptors to close-on-exec.