For a consequential decision, record what observable evidence would invalidate it and when it will be checked. That makes the decision reviewable; it does not make an LLM's output true.
This article describes an operating protocol. The earlier numerical council-performance table is withdrawn because this page does not supply its original sample, observation period, metric definitions, or a reproducible comparison. It does not establish an effect of model diversity or a measured improvement from the protocol.
A sentence such as "There's a 70% chance this stock moves up" is not a calibrated forecast merely because a model produced it. That is an illustrative unsupported output, not a prediction made here. A useful decision record needs the statistical source, assumptions, and validation if it uses a probability.
The protocol adds a structural requirement: no consequential decision is accepted without explicit falsification conditions and an evidence review. That is a policy choice, not a claim of measured superiority over better prompting.
For the consequential decisions covered by this protocol, the accepted record must answer:
Under what observable conditions would this claim be false?
Without that answer, the claim cannot be acted on. It can be noted, logged as uncertain, or escalated — but not used as the basis for a decision.
The agent cannot produce numerical probabilities in any consequential domain (finance, deployment decisions, technology adoption). When a probability is needed:
# ❌ Blocked output
"BTC has a 70% chance of moving up in the next 2 weeks."
# ✅ Required format instead
Scenario A (upward): requires X AND Y to materialize.
Leading indicators currently showing: [observable signals]
Scenario B (sideways): ...
Scenario C (downward): ...
Note: this system does not produce probabilities.For a time-ordered quantitative model in this protocol, separate training and evaluation windows and record out-of-sample results before deployment. An in-sample backtest alone does not establish future performance. This article provides a review requirement, not a validated trading strategy or deployment recommendation.
Co-occurring events cannot be labeled as causal without explicit causal identification. The agent must label observations as:
| Label | Meaning | Action allowed |
|---|---|---|
FACT | Directly observed, source cited | Use in decisions |
INFERENCE | Derived from facts, assumption stated | Use with explicit caveat |
UNIDENTIFIED | Correlation only, causal mechanism unknown | Log, do not act |
SPURIOUS | Apparent relation shown to be misleading by an appropriate check | Do not use as support |
The record format can be adapted to different decisions. These are the three required slots in this protocol; the library example and its historical date below are illustrative, not an executed evaluation:
falsifier:
condition: [observable event that invalidates the thesis]
threshold: [measurable value, not vague]
evaluation_date: [concrete date to check]
# Example: technology adoption decision
falsifier:
condition: >
Library has no releases for 6 months, OR
Critical CVE published with no patch in 30 days, OR
Primary maintainer announces deprecation
threshold: any of the above
evaluation_date: 2026-09-13A decision lacking these fields is incomplete under this protocol. To implement the rule, the actual acceptance path must reject the incomplete record; a prompt alone does not establish enforcement. A structural pass still requires evidence review and a later outcome check.
Multi-agent systems introduce a specific failure mode: empty consensus. Multiple agents agree on a claim — but the claim isn't actually verifiable.
Agreement among models is not an independent observation of the world. However, the earlier table on this page cannot support a measured failure rate or the conclusion that model diversity does not help. The necessary public definitions, denominator, original records, and controlled comparison are not available here. The table and that effect claim are therefore withdrawn rather than replaced with another unsupported number.
To test the effect of this protocol, specify the task sample, model composition, baseline, scoring rubric, observation period, and handling of unresolved outcomes before comparing results. Until such an evaluation is supplied, the recommendation is to make decisions checkable, not to claim a particular reduction in error.
For any council conclusion that is load-bearing (consequential decision, money, deployment) and contains mathematical or logical claims, we route through a verification bridge before accepting the conclusion:
Record the verification result and eventual outcome in the decision ledger, for example through a report_outcome path where that API is actually installed. A stored result gives reviewers something to inspect; it does not demonstrate automatic model learning or improved calibration.
The minimal implementation:
The ledger is the most important part. Without it, falsifiers are just good intentions — stated conditions that never get checked.
The scikit-learn Brier score documentation evaluates predicted probabilities against observed outcomes. A probability-free scenario is not a probability forecast. Falsifier-trigger records can support review of decisions, but they do not by themselves form a Brier-score dataset. This article supplies neither a calibration dataset nor a measured improvement.
It checks whether a decision record states an observable invalidating condition, a threshold, and an evaluation date. Passing that structural check does not establish truth or eliminate hallucinations. The evidence and the eventual outcome still need review.
Do not use an LLM-generated percentage as a calibrated forecast without a statistical source and validation. Conditional scenarios can state what evidence would support or invalidate each branch, but those branches are not numerical probability estimates.
For an illustrative library-adoption decision, a team could reconsider adoption if a critical vulnerability has no patch within 30 days or the maintainer announces deprecation. The team must choose an evaluation date and record what it actually observed. This is an example policy, not a reported experiment.
This article provides no controlled experiment that answers that question. Empty consensus here names agreement without a checkable evidential basis. Model agreement is not itself verification; the relevant mathematical, logical, or empirical claim needs its own appropriate check.
Brier score evaluates probability forecasts against observed outcomes. Use it only when explicit forecast probabilities and corresponding outcomes have been recorded. A conditional scenario without probabilities, or a checklist of falsifier triggers, is not by itself a Brier-score dataset. This article reports no calibration score or measured improvement.
← RDU: Reusable Decision Units · → Stop Hooks as Behavioral Gates