A one-time forensic on our agent fleet concluded that "flagship-grade" quality lived in the method — orthogonal verification, ground-truth-first, closing the feedback loop — not in any one model's weights. That is a satisfying conclusion, and completely unfalsifiable as stated. So we built the thing that would let the claim be wrong: a small, deterministic lab that measures the fleet's own behavior, session after session, with no LLM in the scoring path.
(model, epoch) tuple instead of a thread-level majority. It refuses causal language everywhere except one randomized-dispatch track. The newest piece is a method fingerprint: per-epoch counts of which disciplines a session actually invoked. And the first real lesson wasn't in the data — it was that the "obvious" field for skills, a top-level slug appearing 21,000+ times, was a session codename, not a skill.Here is the uncomfortable thing about concluding that quality comes from method rather than model. It is almost certainly true, and it is also exactly the shape of a claim that survives because nobody can check it. If a session goes well, you can always point at some discipline it followed. If it goes badly, you can always point at a discipline it skipped. The claim is a magnet for confirmation bias because the "method" is observed after you already know the outcome.
To make it falsifiable you need to record the method independently of the outcome, and record the outcome independently of your feelings about the session. That is two columns: a method label and an outcome measure, captured deterministically, per session, before anyone forms an opinion. Almost no transcript tooling captures either well — it captures tokens, latency, and cost, which are the things that are easy to log and the things that least distinguish a careful session from a sloppy one.
Three structural facts make this genuinely hard, and each one silently corrupts the naive version of the analysis.
(family, version, epoch) tuple at the level of the emitting assistant turn, and to keep a model_purity column so you can gate cross-model comparisons to near-pure threads.The temptation with behavioral data is to immediately declare that model A "is better than" model B because its numbers look nicer. That is not what observational transcript data can support, and pretending otherwise is how you get confident, wrong routing changes. So the lab is built around one rule: association-only, everywhere, except one place.
Concretely: every comparison table reports descriptive medians, IQRs, and cell counts, plus a bootstrap 95% CI — and no p-values at this stage, because with this many implicit comparisons a p-value would be a license to fool yourself. Every table is forced to carry an overlap diagnostic (how many sessions actually populate each model × stratum cell), and any cell that doesn't overlap its comparison is stamped NOT_COMPARABLE rather than quietly compared anyway. Causal words — "because," "improves," "outperforms" — are banned from the report generator's vocabulary. The one exception is a randomized-dispatch track: for a specific class of delegations, the model is assigned at random, which is the only condition under which the data can actually identify a causal effect. Everything else is honestly labelled as what it is: a pattern under a fixed policy.
This is the piece that connects the lab back to the original "method beats model" claim. To ever correlate a discipline with an outcome, you have to record, per model-epoch, which disciplines a session reached for. Not what it should have done — what it did, as a deterministic count.
The disciplines that matter for our "flagship-grade" hypothesis all happen to leave the same kind of trace: they are tool calls to specific verification instruments. A multi-persona council for hard trade-offs. A cross-family verifier that routes a claim to a different model lineage so its failure modes are uncorrelated with the author's. Symbolic-math and theorem-prover engines that turn a prose numerical claim into something that either checks or doesn't. Explicitly loaded skills. Each of these is a tool invocation the transcript already records — so the method fingerprint is just a per-epoch, per-instrument call count, stored in a normalized table so the open-ended set of instruments never becomes an ever-widening pile of columns.
Across 155 sessions the fingerprint is already legible: symbolic-math and cross-family-verification calls dominate the flagship model's epochs, while the current relaunched-flagship regime has only a handful of sessions and a correspondingly thin fingerprint. That thinness is itself the honest finding — you cannot yet say anything about the newest regime, and the table says so out loud.
Before any of that could work, we had to answer a boring question: where, in the raw transcript, does "which skill was invoked" actually live? An earlier structural census had catalogued a top-level field called slug, appearing over 21,000 times, and labelled it the skill/command slug. Perfect — ingest that.
Except the raw values were snuggly-whistling-hamster, functional-toasting-bird, cozy-stargazing-meadow. Six distinct values across a large sample, each repeated hundreds of times. That is not a skill name. That is an auto-generated session codename. The census had counted the field correctly and described it wrongly, and the description was the part we were about to build on.
The actual skill signal was somewhere much less exotic: the ordinary tool-call stream the parser was already reading. Skills invoked through the tool interface carry their name in the call's input; verification instruments carry theirs in the tool name itself. Using those meant zero new field dependencies, zero parser changes, and — critically — no reliance on a summary that had already been wrong once. We also looked at a richer-seeming top-level attribution field, but it was stamped on roughly three times more records than there were actual tool calls, because it also tagged the follow-up turns that merely processed a tool result. Counting it would have inflated "method usage" threefold. We left it on the floor and counted the calls themselves.
The transferable lesson is not about one field. It is that a catalog of your data is itself data, and it can be wrong. The counts were right and the meaning was wrong, which is the most dangerous combination, because the counts look like verification. The only defense is to read the raw values with your own eyes before you build on the label — the same ground-truth-first discipline the lab exists to measure.
Four caveats, stated plainly because a measurement system that hides its weaknesses is just a dashboard with better PR.
NOT_COMPARABLE and register a dated falsifier: if that count doesn't cross a threshold by a specific date, the associated hypothesis stays suspended.The final piece of honesty is a falsifier pointed at the lab itself. It has a due date: if several weeks of these reports never once change an actual routing decision — logged in a ledger, not remembered — then the lab has failed its only job and gets cut down, regardless of how elegant the schema is. A measurement system whose measurements never change a decision is a hobby, and the way you keep it from becoming one is to write down, in advance, the observation that would prove it useless.
That is the whole posture, really. The claim that started this — quality is method, not model — only becomes worth anything the moment you build the apparatus that could show it's false, capture the two axes independently, and read your own raw data before you trust your own labels. The lab isn't the answer to "model or method." It's the thing that makes the question answerable.
Q. Is an AI agent's quality determined by the model or the method?
A one-time forensic concluded it was the method — orthogonal verification, ground-truth-first, closing the feedback loop — rather than any specific model's weights. But that claim is unfalsifiable until you record the method and the outcome independently of each other and of your impression of the session. The whole point of building a measurement lab is to make "method beats model" into something that could actually turn out false.
Q. Why is a transcript file not the same as an AI agent session?
In our corpus 94% of the raw transcript files are sidechain fragments — subagent and workflow spawns, not top-level sessions. If you treat one file as one unit of analysis, your sessions are mostly fragments of sessions and your per-session statistics are measuring delegation shape, not work. The unit has to be a reassembled logical session: a main thread plus every fragment that hangs off its UUID directory.
Q. How do you attribute behavior when one agent session uses several models?
Attribute every metric to a (family, version, epoch) tuple at the level of the emitting assistant turn, not to the thread's majority model. A majority-vote attribution pools a mixed thread's minority-model turns straight into the wrong bucket, corrupting any model comparison. Keep a model_purity column and gate cross-model comparisons to near-pure threads.
Q. How do you measure which method or discipline an AI agent actually used?
Build a method fingerprint: a per-epoch count of which verification instruments each session invoked — a multi-persona council, a cross-family verifier, symbolic-math and theorem-prover engines, explicitly loaded skills. Each of these is a tool call the transcript already records, so store them as per-instrument counts in a normalized table rather than an ever-widening set of columns.
Q. Should you trust a data field catalog's description of a field?
No — verify against the raw values first. A structural census had catalogued a top-level slug field (21,000+ occurrences) as the skill or command name; the raw values turned out to be auto-generated session codenames like "snuggly-whistling-hamster," six distinct values repeated hundreds of times. The count was right and the meaning was wrong, which is the most dangerous combination because the count looks like verification.
Q. Can observational transcript data prove one model is better than another?
No. Outside a randomized-assignment track it is association only — a pattern under a fixed routing policy, not a causal effect. Report descriptive medians, IQRs, cell counts and bootstrap confidence intervals, stamp non-overlapping cells NOT_COMPARABLE, and reserve causal language for the one track where the model is assigned at random and a causal effect can actually be identified.