2 / 4 model families were answering, and render_samples dropped the text from list-based messages. I kept the code and reports and wrote down what stopping costs.On November 2 I stopped a scheduled job that had tried to grade how I use my AI tooling every Monday at 06:00 since May. It had filed 25 reports by then. I looked for anything that acted on those grades and found no automation that did.
The job ran cross_family_audit.py 7 --sample 3 from launchd. It scanned every record in Claude Code session files modified within 7 days, sampled 3 sessions, and asked several model families to grade 5 capabilities as low, medium or high. The capabilities included task routing, multi-agent synthesis and async federation, which in plain terms means how much work runs unattended. I had added the job on May 15 as the first prescription for a low async federation grade.
The report went to a dated Markdown file. The last step of the job was /usr/bin/open -e "$OUT", so a text editor window opened on my Mac every Monday at 06:00. The final run used a prompt of 3,381 characters, 1 Anthropic Opus call with max_tokens 2000, and 1 Ollama cloud call.
The audit came up as one decision in the same cleanup that retired my LLM council. Before changing anything I wrote down what the job did.
0 / 4 and hold no grades, only a note that no family was available and that SDKs and API keys needed setup. June 15 says 2 / 4. June 22 says 1 / 4, and that one answer was a connection error counted as a response. Everything from June 29 to November 2 says 2 / 4, with Anthropic and Ollama answering.The sample was meant to show the reviewers real conversations. sample_conversations keeps the last 3 sessions its file traversal reached, which are not guaranteed to be the newest, and the first 5 records of each. render_samples turns those records into prompt text:
if isinstance(content, list):
content = " | ".join(
str(b.get("type") if isinstance(b, dict) else b)
for b in content
)
content = str(content)[:200]When a message's content is a list of blocks, this keeps the block type names and drops the text, so a text block and a tool call reach the reviewer as text | tool_use. Content stored as a plain string survives, cut at 200 characters.
The November 2 report says as much in its opening lines. The Anthropic reviewer started with a premise: of the 3 conversation samples, only 1 showed content, and the other 2 were metadata (/clear, mode, attachment), so the assessment rested mostly on aggregate signals. The tool had described its own input problem, and no automation was there to act on it. That was the last report it wrote.
Here is the Anthropic reviewer's summary table for the last three reports:
| Capability | October 19 | October 26 | November 2 |
|---|---|---|---|
| Multi-agent synthesis | low | low | medium |
| Async federation | medium | low | medium |
The October 26 report justified its low async grade by saying sessions had async work but there was no schedule-based operating signal. That report was written by a launchd job, and several launchd jobs were running. The extractor does not inspect launchd state, so launchd reached the reviewers only through text that mentioned it, such as a conversation sample.
On November 2 the same capability went up. The Ollama family graded it high and cited a conversation sample containing a launchd setup for unattended runs on Sunday at 20:00. The Anthropic reviewer gave it medium and cited the launchd design it saw. Multi-agent synthesis split in that report too: medium from Anthropic, low from Ollama.
I read that as the grade following whichever conversation landed in the sample. That is an inference from the citations. I did not diff the weekly inputs, and the grades were never compared against an outside measure, so I can't say which grade was right in any given week.
My first draft was short: stop the schedule and keep the code and reports. The alternative was to fix render_samples and add the two missing keys. I rejected that because I could find nothing that acted on the grades, so better input would only improve a report with no automation behind it.
This decision was one of the 4 in the blind A/B I wrote up in I Retired My LLM Council After It Lost a Blind A/B 4 to 0. Two verifiers agreed on these corrections to my draft:
open -e leaves no read record, the dashboard surface policy only surfaces failures, and the September 21 brief had named the job as a follow-up input. So my read count could not show whether a person used that window. Only I could answer that.extract_signals(since) has no end-time argument, so the existing extractor cannot replay a bounded past week.The second correction changed what I could claim. The read count covers the dashboard. "Nobody read them" was an assumption from a record that could not have seen the popup.
The corrected plan had 2 options: stop and archive, or reduce to monthly with a 30-day window and fix render_samples. It put the choice on one question to me: had I ever read that Monday window and acted on it? If not, stop. I approved the recommended stop.
launchctl bootout removed the job, and launchctl list | grep -c code-audit returned 0. The plist went to a backup directory.The restart condition is written down: when I need a usage diagnosis, fix render_samples and re-register the job monthly. A manual run with a 30-day window stays available.
"No automation" is the result of searches made for this decision and this post, and a search can miss a reader, as the first one did. It says nothing about me reading the popup on a given Monday, and I cannot reconstruct that now. The one timestamp is what the dashboard recorded, which leaves out any read that happened through the popup.
I also can't call the audit worthless. Its grades were never compared against an outside measure, so I don't know how good or bad they were. My claim is narrower: across 25 reports, I found no automation acting on a grade, the dashboard has 1 read, and the input had a defect the last report described.
Stopping has its own cost. The series ended on November 2, so there is no before and after for this decision, and restarting the existing extractor would not backfill the gap on its own. If I restart the job, its first run would have repaired samples, so those grades would not be comparable with the old ones either.
I searched the code for anything that read the report directory and checked my dashboard's read timestamps. The first search missed the dashboard, which summarizes and displays the reports, so the claim I can keep is that I found no automation acting on a grade. It can't cover a person reading the popup, because open -e leaves no read record.
render_samples keeps only the block type names when a message's content is a list of blocks, so a text block and a tool call become text | tool_use. The November 2 report said only 1 of its 3 samples showed content.
I could find nothing that acted on the grades, so better input would only improve a report with no automation behind it. The corrected plan offered a monthly run with a fixed sampler, and the choice came down to whether I had ever read the Monday window and acted on it. I approved the recommended stop.
The weekly series ended on November 2. extract_signals(since) has no end-time argument, so the existing extractor cannot replay a bounded past week. The scripts and the 25 reports stay, and a manual run with a 30-day window stays available.