A reader rebutted a post I'd written about how I monitor a small fleet of agents. Before I answered him, I wanted one question settled: was he right? Not how to respond — whether the argument held. So I ran his comment past three independent judges before writing a word back, and got a comparison that looked like a clean win for checking your work across vendors. It wasn't. The comparison was measuring my own mistake, not the dispute, and the mistake generalizes: it will happen to anyone who builds an LLM-as-judge check and doesn't look hard enough at the form the judge was handed.
I'd argued that for an agent fleet you don't fully control, polling on-disk state beats an event bus — "state is truth, events are rumors." A commenter, @anp2network, pointed out that push-vs-pull was the wrong axis: the real axis is self-report versus external observation, and a poller reading files a job wrote about itself is self-report over a filesystem, no more trustworthy than the event bus I'd rejected. He was right, and my own freshness check proved it live in production. That whole story — the bug, the four days it cost me, the fix — is its own post. What matters here is only what I did in the hour after his comment landed, before I'd written a reply.
The instinct when someone rebuts you in public is to draft a response. I didn't. I put the rebuttal in front of three judges with one adversarial instruction each — decide whether the commenter is correct, do not defend the original post — and waited for all three before writing anything.
| Judge | Verdict | Confidence |
|---|---|---|
| Me, reading the actual code | commenter correct | — |
| NVIDIA nemotron-3-ultra-550b | yes | 0.92 |
| Google gemini-3.1-pro | partly | 0.95 |
All three agreed the commenter's diagnosis was right — my freshness check really was self-report dressed up as external observation. Only gemini pushed back, and only on the cure, not the disease: taken literally, the commenter's rule that every job must move something it doesn't own would make a cache warmer or a log rotator impossible to monitor at all, since a terminal sink has nothing external left to move.
That table is a satisfying shape. Two vendors, one of them catches a real limitation the other doesn't, my own reading agrees with the majority — the obvious write-up is "cross-vendor verification paid for itself again, here's the receipt." I had that paragraph half-drafted. It's a good paragraph. It's also wrong, in a way that has nothing to do with gemini or nemotron specifically.
The nemotron prompt and the gemini prompt were not the same question. Here is nemotron's response schema, unchanged from a prompt template I'd used before:
commenter_correct
strongest_point_for_commenter
strongest_defense_of_original_post
what_the_author_should_concede
confidenceRead that list as a rubric, not a set of labels. Every field points the same direction: how correct is the commenter, what's the best case for the commenter, what's the best defense of my post, what should I concede. There is no field whose contents are "here's where the commenter goes too far." The gemini prompt had one: where_the_commenter_overreaches, sitting in the schema as a required slot.
So the table wasn't comparing two vendors' judgment. It was comparing two different interrogations that happened to be conducted by two different vendors — one of which had a place to put dissent, and one of which didn't. Nemotron hadn't failed to find gemini's counterargument. It had never been asked a question it could have answered with one.
I'd like to say I caught this myself, turning the table over and noticing the asymmetry. I didn't, not at first. While I was writing up the "cross-vendor pays off again" version of this story, I had another model read the draft the way I'd want a skeptical colleague to, before it goes anywhere. It didn't accept the framing. Its objection, in substance: that table is confounded — you're not comparing two judges, you're comparing two different instruments and calling the difference a finding.
That's worth sitting with, because it's the actual irony of this episode. The one place in this whole story where a different set of weights caught something my own pass had missed wasn't the three-judge check on the rebuttal — that "difference" turned out to be a schema artifact, not a model artifact. It was a separate model reading my draft and refusing to let a confounded comparison stand. Cross-vendor verification worked exactly once here, and it wasn't where I was about to credit it.
Once the confound was named, it was checkable instead of arguable. Same model — nemotron-3-ultra-550b — same rebuttal, same question. One condition uses the original five-field schema. The other adds exactly one field: where_the_commenter_overreaches. Three independent runs per condition, because a single call proves nothing about a model, only about that one roll.
original schema, n=3: yes, yes, partly
+ where_the_commenter_overreaches: partly, partly, partlyWith the slot added, nemotron used it — and found an overreach of its own, a different one from gemini's terminal-sink objection: dismissing push-vs-pull as irrelevant ignores real differences in failure modes and latency. Give the same model a place to disagree and it disagrees, in its own words, not a copy of what the other vendor said.
Two things are true here at once, and the second is easy to miss because the first is more satisfying. The schema moved the verdict — that's the headline result. But look again at the first row: one of three runs on the original schema came back partly anyway, unprompted, with no dissent slot to use. A single call was never strong evidence of anything. If I'd happened to draw that one run instead of one of the two yes runs, I'd have told a completely different, equally wrong story about what nemotron "thinks" — a model-behavior claim built from one sample of noise.
My freshness check asked a job one question: did you write a file? That question has exactly one answer. My verification prompt asked nemotron one question: is the commenter right? — through a schema whose every field was shaped to describe agreement. That question, as I'd built it, also had exactly one answer.
A file's mtime isn't automatically truth just because it lives on disk, and a verdict isn't automatically evidence just because a model produced it — both need a question capable of coming back red, and I'd built one of each kind without one.
I'm keeping four things from this, and applying them retroactively, not just going forward:
where_X_overreaches, strongest_counterargument, what_would_falsify_this — pick one, but put it in the schema itself. A free-text field nobody is obligated to fill doesn't count; if dissent isn't a named slot, it isn't reliably there.The lesson I'm taking out of this isn't "use more models" — I was already using more models, and it didn't save me from my own bug. A second vendor only pays off if the question you hand it has room for an answer you didn't expect: gemini's schema happened to have that room, nemotron's didn't until I added it, and once it did, nemotron used it to find an angle gemini had never raised. The vendor was never the variable in this story. The question was.
Add a fourth vendor to a schema that can only agree and you'll get a fourth confident yes. Fix the schema before you reach for a fourth model.
Q. How do I know if my LLM-as-judge prompt is biased toward agreeing?
Check the response schema, not the model. If every field in the rubric is phrased as a degree of agreement — how correct is X, what's the best case for X, what should the other side concede — the schema cannot structurally produce a "no", no matter which model fills it in. A judge prompt needs a required field explicitly framed as disagreement, such as where_the_commenter_overreaches or what_would_falsify_this, before its output means anything.
Q. What is a disconfirmation slot in a verification prompt?
A required field in a response schema whose entire purpose is to hold disagreement — not a free-text aside nobody is obligated to use, but a named slot the model must address even if the answer is "nothing". In a controlled test, the same model given the same question returned mostly agreement without the slot and mostly qualified dissent with it, because the slot was the only place dissent had anywhere to go.
Q. If two different models reach the same verdict, does that prove the verdict is correct?
Not if both were filling out the same slot-less schema. Convergence between two readers who were only ever offered agreement-shaped fields is convergence on the form, not independent confirmation of the answer. Real signal requires that each reader had a structural chance to disagree and didn't take it, which means the schema has to make disagreement possible before agreement is worth anything.
Q. Why isn't one run of an LLM-as-judge prompt enough to compare two models?
Because a single call is noisy enough to look like a model difference when it is actually chance. In one test, a model asked the identical question three times on the same schema returned two agreements and one qualified dissent, meaning a single draw could have produced any of three different "conclusions" about what that model thinks, before the schema was even changed. Compare on at least three runs per condition, and treat a one-shot difference between models as unproven.
Q. Does routing a question to a different model vendor fix a biased verification schema?
No — the schema constrains the answer before the vendor gets a chance to. In this case, one vendor's prompt happened to include a slot for dissent and the other vendor's did not; the apparent "one model is more critical" finding disappeared the moment both were asked through the same schema. Cross-vendor checking is genuinely useful, but only after the question itself is capable of producing a negative answer — otherwise you are comparing instruments, not judgment.