GROUP BY was an afternoon. Everything after it was a decision about what not to count, because each shortcut produced a confident number that was wrong. This is that list.I run several models across a fleet of agent CLIs, and I wanted a plain question answered: on my own work, which ones re-edit the same file more, which ones burn more validation runs, which ones error on tool calls more often. I already log every turn, so the measurement looked like a GROUP BY.
It wasn't. The query was an afternoon. The honesty was the week. Almost everything I did after the first GROUP BY was a decision about what not to count, because every shortcut I could take would have produced a confident number that was wrong. This is that list.
Each turn in my logs belongs to a role: a main thread, where a model works interactively on a long task, or a sidechain, a short one-shot the harness spins off. These are not two samples of the same thing. A main thread for one model had a median re-edit rate near 0.44; its sidechains sat at 0. Pool them and the model's "re-edit rate" becomes an artifact of how many sidechains happened to run that week — a number that moves when the workload mix moves, not when the model changes.
So the unit of comparison is model × role, never model alone. Every table is stratified by role, and I do not compare a main cell to a sidechain cell even for the same model: the session length and structure differ too much to read the gap as anything but confounding.
The same model name covers more than one thing over time. One of mine ran under a pre-launch configuration, then a relaunch, with a blocked window between. Those are not one population with more data; they are distinct treatments. Averaging across them hides exactly the shift you built the lab to see. I keep them as separate rows — model/pre and model/relaunch never get summed — with epoch boundaries pinned to calendar dates so the split is reproducible and not drawn to flatter a result.
Two thresholds, declared before I looked at the outcomes:
NOT_COMPARABLE, not the value. One cell had exactly one observation. Its median is a real number and a useless one; setting it beside a cell with 2,781 observations invites a comparison the sample cannot support.The point of writing the floors down first is that otherwise you discover them afterward, at whichever value makes the cell you like look solid.
871 of my threads — 57 main, 814 sidechain — could not be attributed to any model. They were synthetic harness activity with zero tool and token behavior. The tempting move is to treat "unknown" as "zero" and fold them in. That single substitution would have shifted every rate and every denominator, because missing is not the same as zero and an unattributed turn is not a well-behaved one.
They appear in exactly one place: a reference line that says excluded, 871, here is where they went. They enter no average. A measurement that silently drops what it can't classify is the one that lies first, so I made the drop loud and countable instead.
Most of these per-turn metrics are zero most of the time. A model does not re-edit on most turns; it does not error on most turns. When I bootstrapped a median difference between two such cells, the interval frequently came back [0, 0].
That looks like a precise result. It is the opposite. When the data is almost all ties at zero, the median of every resample is zero, so the interval collapses to a point — not because the estimate is sharp but because the statistic has nothing to vary over. I label those collapses as a coverage limitation of a median bootstrap under heavy ties, not as a tight finding. A degenerate interval and a precise one print the same two characters; only the sample behind them tells you which you have.
I wanted a total-output comparison, so I summed output tokens per turn. That one is genuinely additive. The trap was next to it: the stored input-token and cache-read counts are the context re-sent on every turn, counted again each time — a running sum with the same tokens in it over and over, not a count of distinct tokens. Summing those to compare "how much each model read" would have rewarded whichever model took more turns. They are logged; they are not a total. Only the output sum answers the question I actually asked.
Two labels in my own schema promised more than they measure, and I wrote the gap down next to each:
same_file_reedit_rate counts a file touched again. It does not separate iterating toward a result from reworking a mistake. The name suggests a quality signal; the measurement is a frequency. If I ever make it a primary metric, that limitation gets pre-registered with it.completion_proxy reads the shape of a thread's last line. It is a heuristic for "finished," not a judgment that the work was done, and I had to censor threads whose snapshot was cut off mid-turn so a truncation wouldn't get scored as an abandonment.The sessions in which I built and audited this pipeline are in the corpus it measures. Those turns land in one project's stratum and distort its numbers — the tool is watching itself work. I can't retroactively remove the observer from the observed, so the limitation is named in the report and the next version adds a stratum that excludes self-measurement. Naming it is not fixing it; it's refusing to let the contamination pass as a finding.
Models were never randomly assigned to tasks. Harder work, different projects, different weeks, changing dispatch policy — all of it is confounded with which model ran. So the whole thing is graded association-only and exploratory. There are no p-values; the comparison answers what co-occurred, not what caused what, and any sentence that drifts toward "model X is better" is one the design cannot support.
The deliverable here was never the ranking table. It was the list above — the specific numbers I declined to print, and the reason each one would have been a lie. A comparison is mostly the rows you refuse to count.
Q. Why not just GROUP BY model and compare the averages?
Because the turns belong to two roles — interactive main threads and one-shot sidechains — that are not samples of the same thing. Pooling them makes a model's rate an artifact of the workload mix, not a property of the model. The unit is model-by-role, and a main cell is never compared to a sidechain cell.
Q. Is a bootstrap interval of [0, 0] a precise result?
No. When a metric is almost all ties at zero, every resample's median is zero, so the interval collapses to a point — a coverage limitation of a median bootstrap under heavy ties, not a sharp estimate.
Q. Does this show which model is better?
No. Assignment was not random, so task difficulty, project, timing, and policy are confounded with the model. It is association-only and exploratory, with no p-values.