I run a small fleet of Claude-family agents on one Mac. Every session gets logged — the long interactive ones I drive by hand, and the short subagent runs those main sessions spawn. A while ago I built a measurement pipeline (I call it PAMSL) that reads those transcripts into SQLite and attributes every single turn to the model that produced it. A few thousand real agent threads later, I had the thing people keep asking me for: a table comparing the models on actual workload telemetry. Tool error rate. Re-edit rate. Output tokens. A rough "did the task finish" proxy. Five models, two roles.
The honest deliverable was a stack of refusals to conclude. This note is the walkthrough, because every trap in it applies to anyone reading their own AI usage logs.
Here is the seductive row. On my main interactive sessions, median output tokens per thread:
| model (main sessions) | median output tokens |
|---|---|
| Claude Opus 4.8 | 404,080 |
| Claude Sonnet 5 | 27,616 |
That is nearly a 15x gap. Read naively, it says one model burns fifteen times more tokens than the other for the same work. People see a number like that and immediately want to rewrite their whole setup.
It is not a fact about the models. It is a photograph of my dispatch rules.
I deliberately send big, open-ended jobs to Opus main sessions and route quick, mechanical work to cheaper tiers. The models never got assigned to tasks at random. So that 15x is exactly what my own routing policy was built to produce: the expensive tier is where I put the expensive work.
This is what "association-only" means. Every difference in the table is entangled with task difficulty, project, and time-of-week — because those are the very things I use to decide where each job goes. Nothing here is randomized, so every confounder is wide open. The table can describe. It cannot explain. I marked the whole thing Grade C, exploratory, and meant it.
Once you accept that, the job stops being "rank the models" and becomes "list every way this table will lie to you." Here are the ones I had to code around.
Main sessions are long, interactive, and I stay in the loop. Sidechain subagent runs are usually one-shot: fire, produce, exit. Those are two different populations with different length and structure. Cross-comparing them measures the harness, not the model, so I keep the two roles strictly separate and never put them in the same comparison.
The sharpest version of this bit me on the completion proxy — a heuristic that guesses whether a thread "finished" from how its last line looks. One model's sidechains scored terribly: 92 of 99 ended on a tool_result line, which the heuristic read as "stopped mid-work." But that is just a harness logging convention for how subagent transcripts terminate. The work was fine. The metric was measuring log formatting. I threw sidechains out of the completion comparison entirely and kept it to main threads only.
In early July I relaunched one of the models after a change to its setup. The name on the label was identical before and after. The behavior was not. If I pool those turns under one name, I am averaging two different treatments and calling the blend "the model."
So I split every model at its version boundaries into epochs — the pre-relaunch epoch and the post-relaunch epoch are separate rows, never summed. Epoch boundaries are calendar dates, fixed in advance. A model name is marketing. A version epoch is the thing you can actually reason about.
Most cells in the table have a median of 0 and an interquartile range of 0 to 0 — tool errors are rare, re-edits in one-shot runs are rare, so the modal thread scores zero on almost everything. When I ran percentile bootstrap confidence intervals on those cells, they came back as [0, 0].
A [0, 0] interval looks like breathtaking precision. It is the opposite. It is tie-degeneracy: so many identical zeros that every resample returns zero. The interval is not telling you the effect is pinned down to nothing; it is telling you the data is almost constant and the method has nothing to chew on. If you report those intervals as confidence, you are laundering a tie into a result.
After discarding the degenerate cells and the cross-role comparisons, plenty of contrasts still qualified for an interval — but every one of them was a process-or-volume measure: token totals, tool-error rates, re-edit rates, recovery sequences. Gaps there mostly restate my dispatch decisions. Exactly one qualifying contrast measured whether the work actually finished rather than how big or bumpy it was: the completion proxy, main sessions, older flagship versus the relaunched model:
The interval spans zero. The tempting headline — "the newer model finishes more of its tasks" — dies right there, in the one place I actually had the data to test it. That is the whole essay in three numbers: the cleanest comparison I could build still refused to separate the two.
Only output tokens add up. My pipeline stores input-token and cache-read totals per turn. Summing them across a thread looks reasonable and is wrong: every turn re-sends the same growing context, so those totals are the same tokens counted over and over, duplicates and all. Only output tokens are genuinely additive. Comparing "input token totals" across models compares context re-send patterns, not usage. I let only output tokens into the comparison.
I was inside my own corpus. The sessions where I built and audited this very pipeline are logged like any other work — and they land in the project stratum for the pipeline's own project, distorting exactly the numbers I was most tempted to read. Self-measurement contaminates itself. I flagged it, and the next iteration gets a dedicated exclusion stratum so the observer stops standing in the frame.
Two reasons, both real.
First, drift monitoring. Even without causal claims, a stable per-turn baseline tells me when something moves — when error rates or completion proxies shift after a relaunch or a harness change. You do not need identified causation to notice a regression.
Second, and more useful: an association table tells you where randomization would pay. The cells that look interesting but are hopelessly confounded are a shopping list for the experiment worth actually running. I pre-registered a separate randomized dispatch experiment aimed at precisely the cells this table lit up. Observational telemetry is a hypothesis generator, not a verdict. Its job is to point at the question, not to answer it.
That is also why I keep the pipeline honest about being a measurement lab, not a scoreboard.
Five refusals I now apply to any table pulled from usage logs:
None of this ranks the models. That was the point. The most honest thing my measurement pipeline produced was a precise account of what it could not yet know.
Q. Can I rank LLM models from my own usage logs?
Not if you routed the work yourself. Observational usage logs reflect your dispatch policy — which model you sent which tasks to — so any gap is entangled with task difficulty, project, and timing. Without random assignment the comparison is association-only: it can describe differences but cannot attribute them to the models. Use it to generate hypotheses, then run a randomized test on the cells that look interesting.
Q. Why do bootstrap confidence intervals collapse to [0,0] on agent telemetry?
Because most cells are zero-inflated: tool errors and re-edits are rare, so the median and interquartile range are both zero. When almost every value is an identical zero, every bootstrap resample returns zero and the interval collapses to [0, 0]. That is tie-degeneracy, not precision — the data is nearly constant and the method has nothing to resample. Do not report it as a tight result.
Q. Why should model version epochs never be pooled?
Because a model name is not a stable treatment. After a relaunch, config change, or silent update, the same label can describe different behavior. Pooling turns from before and after averages two treatments into a blend you cannot reason about. Split each model at its version boundaries into epochs and compare epochs, not names.
Q. What is an association-only comparison?
It is a comparison where the groups were not randomly assigned, so observed differences are confounded with whatever decided the assignment. In agent telemetry, your routing rules decide which model runs which task, so the model column is tangled with difficulty, project, and time. Association-only tables can describe and monitor for drift, but they cannot establish cause.
Q. How do I compare AI models honestly?
Refuse five things: cross-comparing populations you route differently; pooling a model across version epochs; reading a [0, 0] interval as precision; summing counters that are not additive, like re-sent input tokens; and calling association causal. What survives those refusals is usually a hypothesis worth testing with a randomized experiment, not a ranking.