I run a small fleet of coding agents on one machine. Every thread ends up in a log, and a measurement pipeline reads those logs into a database, attributing each turn to the model that produced it. After a few thousand threads I had the table people keep asking for: seven model-versions, five behavioural metrics, real workload rather than a benchmark.
Then I printed one cross-tab I had been skipping, and most of that table stopped meaning what I thought it meant. The cross-tab was role by model. A model here is either the main thread I drive by hand, or a sub-agent that a main thread spawns, runs once, and discards. Same model, same weights, two completely different jobs.
Model names are replaced with letters on purpose: the point of this note is that these numbers are not a model ranking, and printing the names invites exactly that misreading. Every figure is real, from one snapshot of one fleet.
Median output tokens per thread, same model, split by role:
| model | main n | main median | sub-agent n | sub-agent median | main / sub |
|---|---|---|---|---|---|
| A | 200 | 478,238 | 2,761 | 6,212 | 77x |
| C | 88 | 427,838 | 156 | 8,784 | 49x |
| D | 11 | 185,534 | 122 | 1,371 | 135x |
| G | 21 | 128,415 | 18 | 11,358 | 11x |
| B | 20 | 16,432 | 1,338 | 14,157 | 1.2x |
| F | 15 | 4,194 | 6 | 51,562 | 0.08x |
The behavioural metrics are worse than lopsided. In the sub-agent stratum the same-file re-edit rate has a median of exactly 0 for six of seven model-versions, and so does the error-recovery count; validation runs are 0 for all seven. In the main stratum those same metrics carry real spread - re-edit medians of 0.40, 0.44, 0.50 and 0.53, recovery medians of 1 and 2.
The four-model qualifier is load-bearing, so let me not hide behind it: the other three model-versions sit at a median of 0 in main too - two genuinely, one because its main cell holds 2 rows and is marked not comparable. Main is where between-model signal can live, not where it always does.
Still, the pattern is not a subtle covariate. A main thread iterates: read, edit, re-edit the same file, hit a failure, recover, run a check. A sub-agent does its one job and exits, so it rarely touches a file twice and rarely has a failure to recover from. The metric is structurally near-zero there. Which means the sub-agent stratum carries no between-model signal at all on those metrics - the median is a constant. All the signal lives in the main-thread stratum, which is 7% of my rows.
| model | main rows | sub-agent rows | main share |
|---|---|---|---|
| E | 2 | 132 | 1.5% |
| B | 20 | 1,338 | 1.5% |
| A | 200 | 2,761 | 6.8% |
| D | 11 | 122 | 8.3% |
| C | 88 | 156 | 36.1% |
| G | 21 | 18 | 53.8% |
| F | 15 | 6 | 71.4% |
Across the attributed corpus, 4,533 of 4,890 rows - 92.7% - are sub-agent rows. So a pooled number is mostly a description of sub-agents. But how mostly ranges from 1.5% to 71.4% main, a 48-fold spread in composition.
The reason is not random sampling: it is the delegation policy. Orchestration rules send mechanical fan-out work to cheaper tiers, so those models accumulate sub-agent rows by the thousand, while models I drive by hand accumulate main rows. Model A spawned 1,788 sub-agents that were also A and 730 that were B; model C spawned 393 that were B. Role is assigned by the same policy that assigns the model, so the confounder is baked into the architecture rather than introduced by chance. Anything with an orchestration layer has this shape - retry tiers, canary versus steady-state traffic, batch versus interactive queues, free versus paid users.
Pooling A and C across roles, weighted by row counts - which is exactly what a GROUP BY model query gives you:
A: (200 x 684,639 + 2,761 x 13,691) / 2,961 = 59,010 C: ( 88 x 565,678 + 156 x 13,913) / 244 = 212,910
Pooled, C burns 3.6x the output tokens of A. Inside each stratum the picture inverts:
| stratum | A mean | C mean | C / A |
|---|---|---|---|
| main | 684,639 | 565,678 | 0.83x |
| sub-agent | 13,691 | 13,913 | 1.02x |
Within each stratum the two models are close. C is lower than A on main threads (0.83x) and 1.6% higher on sub-agent runs - a near-tie that happens to lean the same way as the pooled number, just nowhere near its size. Neither ratio comes close to 3.6x. The 3.6x is manufactured by the weights: main threads carry 40-50x the mean output of a sub-agent run, and 36.1% of C's rows are main threads against A's 6.8%, so C's average is dragged toward the expensive stratum and A's is not.
Worth being precise about the name. This is not textbook Simpson's paradox, which needs every stratum to point one way while the pooled number points the other; here one stratum reverses and the other is a near-tie. Arguably that is the more dangerous shape - there is no meaningful within-stratum effect in either direction, and pooling still produced a 3.6x headline out of nothing. Call it an amalgamation effect: a difference in composition, amplified into an apparent difference in the metric.
The reason to walk it out is that it never announces itself. Nobody sets out to pool - pooling is what the obvious query does. The number arrives looking like a model comparison and is in fact a weighted average with different weights per model. What you measured was your dispatcher.
A metric may not mean the same thing in each stratum. My completion proxy - a heuristic that reads whether a thread finished from the shape of its last logged line - collapsed on sub-agent threads, where one model had 92 of 99 threads ending on a tool-result line purely as a harness logging convention. I unpacked that particular failure in an earlier note; what matters here is what it implies for stratification.
It implies stratification is not a repair. A metric whose value is set by logging convention in one stratum is not a noisier measurement there - it is a different measurement, and averaging it separately does not make it comparable. So the question after asking whether you stratified is whether the metric measures the same construct in every stratum. Construct validity is per stratum, not per metric. Where the answer is no, drop that stratum for that metric and say so in the output.
There is a quieter version of the same problem: the unit can change between metrics. My behavioural metrics are per-turn attribution rows; the completion proxy is thread-level, with a purity filter and a censoring rule (threads whose snapshot was cut right after a user turn count as missing, not as failures). Model A has 200 main attribution rows and 154 usable main threads for the proxy. Two counts named n in the same report, meaning different things.
The role gap is not a constant you can adjust away. It is 77x for A, 135x for D, 11x for G, 1.2x for B, and in two thin cells the sign flips outright. If role were a fixed multiplier you could divide it out. It is not; it interacts with the model. Partly that is because a stratum holding 1.5% of a model's rows is not a random 1.5% - whatever rare circumstance put that model in that role is also selecting the kind of work it did there. Stratification buys you a list of licensed comparisons, not a correction factor.
Validation-run counts, main threads only. Every model's median is 0. The means are not: A 4.11, C 2.38, G 1.67, B 0.60. Percentile bootstrap on all six model pairs returned a median difference of 0 with a 95% interval of [0, 0] every time - a roughly 7x difference in means, reported as exactly zero with zero width.
Both statements are arithmetically correct. The median really is 0 in each cell, and it is 0 in nearly every resample, so the bootstrap distribution is a spike and the percentile interval collapses onto it. That is tie degeneracy, not precision: the interval is not saying the effect is pinned down, it is saying the statistic cannot move. Read as confidence, it launders a tie into a finding, in the most persuasive possible format - a narrow interval.
My pipeline already had a guard: skip a pair if both cells have median 0 and an interquartile range of 0 to 0. Necessary and insufficient. A's range here is 0 to 4 and B's is 0 to 0, so the pair sails through the filter and degenerates anyway. The right condition is not that both inputs look constant but that the statistic is constant across resamples, which you can only check on the resample distribution itself.
Routing was never randomized, so none of this is causal - the model each thread got was chosen by policy, entangled with task difficulty, project and week. That argument has its own note, linked below; everything here is association under a fixed dispatch policy, and the role finding does not rescue it.
Role is also not the only stratum. Split model A's main threads by project and the re-edit median runs 0.34, 0.45, 0.53 and 0.70 across four projects - a 2x spread inside a single model-and-role cell, while the same model's sub-agent value is 0 in every project. The re-edit metric also cannot tell healthy iteration from thrash; it counts both. And the sessions in which I built and audited this pipeline are logged like any other work, so the observer is standing inside the frame; the next iteration gets an explicit exclusion stratum.
This is one snapshot, one operator, one harness, and the constants will not transfer. The procedure does: cross-tab first, threshold in advance, construct validity per stratum, and no faith in narrow intervals over tied data. Before you compare treatments, check whether they are running in the same role, in the same proportion. In an agent fleet the answer is almost always no, and nothing downstream survives that going unasked.
Q. Why can sub-agent metrics not be compared to main-thread metrics?
Because they are different populations doing different jobs. A main thread is a long interactive session that iterates on files and recovers from errors; a sub-agent is a one-shot run that produces output and exits. In my fleet the median output tokens per thread differ by 77x between the two roles for the same model, and behavioural metrics such as same-file re-edit rate have a median of exactly 0 in the sub-agent stratum for six of seven model-versions. A contrast across those roles measures the harness, not the model.
Q. What goes wrong if I pool roles and compare models?
The role mix differs per model, so a pooled average is a weighted average with different weights per treatment. In my snapshot one model had 1.5% main-thread rows and another had 36.1%. Pooling made model C look like it emitted 3.6x the output tokens of model A, while inside each stratum C was 0.83x A on main threads and 1.02x A on sub-agent runs. A 3.6x headline built out of two within-stratum ratios near 1 is an amalgamation effect: the composition is doing the work, not the model. Note this is not textbook Simpson's paradox, which needs every stratum to point the same way; here one stratum reverses and the other is a near-tie.
Q. Is stratifying by role enough to make the comparison valid?
No. Two problems survive stratification. First, a metric can measure a different construct in each stratum: my completion proxy read a thread's last logged line, and in one model's sub-agent threads 92 of 99 ended on a tool-result line because of a harness logging convention, so the metric scored the logger rather than the agent. Second, the role gap is not a constant multiplier you can adjust away - it ranged from 1.2x to 135x and flipped sign in thin cells. Stratification gives you a list of licensed comparisons, not a correction factor.
Q. Why did my bootstrap confidence interval come back as [0, 0]?
Almost certainly tie degeneracy rather than precision. On a near-constant metric the median takes the same value in nearly every resample, so the percentile interval collapses onto a point. In my validation-run counts the medians were 0 for every model while the means ranged from 0.60 to 4.11, and all six pairwise intervals were [0, 0]. A guard that skips a pair only when both cells look constant is not enough - check whether the statistic is constant across resamples, and report the share of tied values next to any median-based interval.
Q. What should I check before comparing treatments in an agent fleet?
Print the stratum by treatment cross-tab before any metric, set a minimum cell size in advance and mark thin cells as not comparable, pre-register cross-stratum comparison as forbidden rather than merely discouraged, ask per stratum whether the metric measures the same construct, refuse to read a zero-width interval as a result, and publish the pooled and stratified numbers together whenever they disagree.