In a public ML benchmark competition, we ran a submission policy: don't spend a submission — a scarce, rate-limited resource — unless a candidate change beats the current best by at least 2 percentage points on a fixed, 40-task local holdout set. It read like ordinary scientific discipline: don't act on noise, require a minimum effect size before you spend something scarce. Over a 125-day eligibility window, at one submission per day, exactly one submission went out — under an explicit calibration exemption written into the rule itself, not because anything cleared the +2pp bar. In 125 days, the threshold itself was cleared zero times.
The policy had three conditions. Two were fine: reproduce across multiple random augmentation seeds, and a full-length dry run finishing without timing out. This is about the third — beat the local holdout by at least 2 points — because in 125 days nothing ever actually cleared it. The one submission that did go out was explicitly exempted from it by the rule's own calibration clause, not passed by it.
At first the near-silence looked like it might just be the truth: maybe nothing attempted in that window genuinely worked. That's possible — and it is also exactly what you would see if the gate itself could not tell a real improvement apart from sampling noise. Those two very different realities produce an identical pass log. The only way to tell them apart is to stop staring at the log and start measuring the instrument that produced it.
Before arguing about whether 2 percentage points was the right bar, we measured something nobody had measured yet: how noisy is the ruler itself, compared to the thing we actually cared about?
| Instrument | Standard deviation | Basis | Role |
|---|---|---|---|
| 40-task local holdout | ±4.79pp | Simulated, from a measured task-churn rate of 11/120 | The gate — decided whether you could submit |
| Larger 120-task local eval set | ±2.77pp | Simulated, from the same measured task-churn rate | Available, but never used as the gate |
| Actual scored result (the real target) | ≈0.2pp | Observed directly — 4 submissions of the same configuration, made by different teams | What you were actually trying to predict — those 4 submissions landed inside a 0.42-point band |
Worth flagging that "Basis" column: the two gate-side numbers are not repeated measurements, they are a simulation seeded by one observed churn rate; only the target-side number is a direct, repeated observation. An essay arguing you should measure your own instrument's noise before trusting it does not get to skip that disclosure about its own numbers.
The gate was carrying close to 24 times more noise than the thing it existed to protect access to. And it was not just imprecise in some abstract sense — a companion analysis measured the false-positive rate of the "beat the holdout by 2 points" bar directly and got 39.4%. More than a third of the time, pure sampling noise, with zero real improvement behind it, would clear that bar on its own. The gate was not a strict filter guarding a scarce resource. It was closer to a coin flip wearing a lab coat.
This is a completely general shape, not a quirk of one competition. It is the same comparison as a performance budget enforced on noisy, shared CI runners gating a deploy whose real production regression rate is tiny and well characterized, or a flaky pre-merge test suite blocking merges into a service whose actual incident rate barely moves release over release. In every case there is one diagnostic question that settles the argument before you even get to threshold values: whose standard deviation is bigger, the gate's or the target's?
It is tempting to read "the threshold was cleared zero times in 125 days" as proof the gate worked — screening out everything that was not good enough, letting nothing through that had not earned it. That reading does not survive contact with the numbers above. When a gate's own noise floor is several times larger than the effect size it is supposed to detect, a zero-pass log is exactly what you would see whether the true rate of real improvements underneath was high or low. The gate cannot distinguish those two worlds, so its output does not carry that information either, no matter how strict it looks from the outside.
The general version: a team that points to "our gate rejects 95% of what comes through it" as evidence of rigor is citing a statistic about the gate, not about what it screened — unless it has separately confirmed the gate's noise floor sits below the effect size that matters. Otherwise a zero-pass (or near-zero-pass) log and a well-calibrated strict gate look identical from the outside, and only one of them is doing anything useful.
Once a gate looks untrustworthy, the natural move is to make the gate itself more precise: a bigger holdout, more seeds, a tighter statistical test. That instinct is only correct if the resource being protected is the noisier side of the comparison. Here it was the reverse — the thing behind the gate, the actual scored result, was already dramatically more precise than the gate itself. Spending effort shrinking the local holdout's noise further would have meant polishing the wrong ruler, like recalibrating a bathroom scale to match a lab balance you already have standing right next to it, instead of just stepping on the lab balance.
The general rule: only invest in tightening a gate when the resource on the other side — the expensive or scarce one — is itself the noisier side of the pair. If the target signal is already more precise than your proxy, refining the proxy further changes nothing; you still have a cheap, noisy instrument standing in front of an expensive, precise one. The real question is not how to make the proxy better. It is when you are allowed to just consult the precise instrument.
The other tempting move runs the opposite direction: since the gate clearly does not work, delete it — submit whenever you want. That is also wrong, for a reason that has nothing to do with noise. The gate existed to block a specific, real failure mode: spending a scarce resource by trying many variants and only reporting the one that happened to score best. That failure mode — call it fishing, or informal multiple testing — is just as real with a perfectly precise proxy as with a noisy one. Delete the gate and it comes straight back.
The old rule had quietly conflated two different jobs under one number: a statistical bar on a noisy proxy metric, and a discipline requirement against fishing. Only the first one was broken. So we deleted the first and kept, then sharpened, the second.
In place of the deleted statistical bar, we added a process requirement: state, in one sentence, the specific hypothesis this attempt tests, and explain why your cheap local measurement is structurally unable to resolve it — not "it's noisy this time," but incapable by design. Then write down, for every range the real result could plausibly land in, what you will conclude and do next, before you see the result. If any range's answer is "we'll figure it out when we see it," the attempt does not qualify.
The anti-fishing rule stayed completely untouched: no resubmitting an unchanged configuration hoping for a better roll. One configuration, one measurement, and remeasurement only after a real change. That is what separates the two things the old rule had lumped together — selecting on noise, illegitimate and non-transferable, from measuring a hypothesis your cheap instrument structurally cannot answer, legitimate and transferable — while still blocking the failure mode the gate was originally built for.
Translated to ordinary software practice: you do not scrap "only one canary deployment in flight at a time." You might well scrap "must beat this noisy synthetic load test by X%" and replace it with "state which regression this canary is checking for, and write down what you will do for every value the canary metric could show, before you ship it."
Everything above — the rewritten rule, and a full table mapping every possible real-world outcome to a conclusion — was written down before the actual result existed. That ordering is the entire point. Rewrite a rule after you have already seen the number, and you are not correcting a bad gate. You are picking whichever justification would have let you do what you already wanted to do.
The pre-registered outcome table looked roughly like this (scores on this competition's normal 0–100-ish scale, baseline at 28.47):
| Real score lands in... | Conclusion |
|---|---|
| ≥ 30.5 | Local read had the direction wrong; every recent local-only verdict this week needs re-checking |
| 28.9 – 30.4 | Real effect exists but is small; local got sign or size wrong |
| 28.05 – 28.89 | Effect is zero within noise; local verdict stands |
| 27.5 – 28.04 | Effect is real and negative, matching local, just smaller |
| < 27.5 | Can't separate a real negative effect from an unrelated execution failure; inconclusive |
The candidate under test had searched a wider space per task — about 31% more candidate answers generated, roughly 22% more compute spent — and the real result came back at 28.19. That falls inside the 28.05–28.89 band. The conclusion followed mechanically, exactly as written down in advance: the extra search bought an effect indistinguishable from zero at this instrument's noise level. No re-litigating, no "well, technically." The table had already decided.
There was a second, unplanned payoff. An earlier calibration point had shown the local proxy's absolute level predicted the real score reasonably well — the real result had landed at about 97.6% of the local number. But one point can only validate level, never ranking; checking whether a noisy proxy preserves order takes at least two independent readings moving in a known direction. This measurement supplied the second point: the ratio-adjusted local prediction was 27.93, the actual result was 28.19, a gap of 0.26 — about the size of the real instrument's own noise floor. A gate can be too noisy to threshold on directly and still be trustworthy for ranking candidates against each other, but that is a separate property from absolute-level accuracy, and it takes a second measurement to check, not one.
Three honest limits, stated on the record rather than left implicit:
The general habit worth keeping, independent of any of this: before arguing about where to set a gate's threshold, measure the gate instrument's own noise and put it next to the noise of the thing it exists to protect. If the gate turns out to be the noisier side, the fix is not a bigger or smaller number. It is moving what you are gating on entirely — from a statistical bar you cannot trust to a discipline you can. And whichever fix you pick, write down why, and what every outcome will mean, before the result exists to tempt you.
Q. What does it mean for a quality gate to be noisier than what it guards?
It means the instrument enforcing the gate has more measurement variance than the real signal it exists to protect access to, so the gate's pass or fail outcome carries less information than the number it is gating.
Q. Why doesn't a near-zero pass rate prove a gate is working?
Because a gate whose own noise dwarfs the effect size it is supposed to detect produces a near-empty pass log whether the true rate of good changes underneath is high or low. The pass log is a statistic about the gate, not about what it screened, unless you have separately confirmed the gate's noise sits below the effect size that matters.
Q. If a gate is too noisy to trust, why not just remove it?
Because a noisy threshold and an anti-fishing discipline are usually two different rules wearing one number. Removing the whole gate brings back the failure mode the discipline half was blocking, which is trying many variants and only reporting the best one, even though that failure mode has nothing to do with the noise problem.
Q. When is it actually correct to make a gate itself more precise?
Only when the resource on the other side of the gate is the noisier side of the comparison. If the target signal is already more precise than your proxy, refining the proxy further does not fix anything. The fix is deciding when you are allowed to consult the precise signal directly.
Q. How do you check whether a noisy proxy still preserves ranking?
You need at least two independent measurements moving in a known direction. One point can confirm a proxy's absolute level, but only a second point, compared against the same target, can show whether it also preserves the order between candidates.