We built a randomized experiment to earn the right to say one word — because. The random assignment was cryptographically fair; we verified it. Weeks in, an audit found that the experiment could not be analyzed at all, and never could have been. The coin was flawless. The join from the coin to the thing it decided did not exist in the data.
Most of what you can measure about a running system is observational — patterns under whatever policy happens to be in force. Observational data can generate hypotheses, but it cannot, by construction, tell you that a change caused an outcome, because the thing you changed is tangled up with everything else that moved at the same time. There is exactly one clean escape: assign the treatment at random. Randomization is what severs the treatment from all the confounders and lets you say the forbidden word — because — about the difference between arms.
So for one specific class of decisions we did the disciplined thing. Instead of always routing a task to the default handler, a dispatcher flipped a fair coin and sent it to arm A or arm B. That single randomized track was meant to be the one place in the whole system where a causal claim would be defensible. Everything hinged on it being done right.
The randomizer used a cryptographically seeded source (SystemRandom().choice, backed by the OS entropy pool), so the draw was an honest, independent p=0.5 flip. When the observed split came in lopsided — roughly four to one — the instinct was to suspect a broken coin. That instinct was wrong: with that few draws, a two-sided binomial test puts a 4:1 split at P ≈ 0.375, nowhere near anything you would call biased. The coin was fine.
Here is the trap. Verifying the coin feels like verifying the experiment. It is vivid, it is mathematical, it produces a satisfying green check. And it is maybe ten percent of what makes a randomized experiment work. The other ninety percent is bookkeeping so boring nobody writes a test for it — and that is exactly where this one had already failed.
To analyze a randomized experiment you compute an intention-to-treat comparison: for every unit, what arm was it assigned, and what outcome did that unit get. That requires a join key — a stable identifier stamped on the unit at the moment of assignment and carried through to wherever the outcome is recorded. It is the spine of the entire analysis.
Our dispatcher was a function that took no arguments, returned an arm, and appended the draw to a log. The log recorded that a draw happened and which arm won. It did not record which task the draw was for. Nothing downstream ever read the dispatch log to tag the resulting work. In other words, the assignment and the outcome lived in two separate piles with no key joining them.
The missing join was the headline, but the same audit found three smaller failures, each of which independently prevents an experiment from being trustworthy even by an honest operator.
is_test column.Suppose we fix all of the above tomorrow. There is a separate, earlier failure hiding underneath: the experiment was never powered. No minimum detectable effect was computed before it started. When we projected the effective sample size forward to the pre-registered deadline, it came to roughly 14–19 usable units against a target of 40 — under half. The experiment was on track to end inconclusive no matter how clean the bookkeeping was.
This is the quietest and most common way experiments waste themselves. Without a pre-planned MDE, an inconclusive result is ambiguous: you cannot distinguish "the two arms are genuinely equivalent" from "we never had enough units to see a difference." Those are opposite conclusions that produce identical-looking null results. The power calculation is what forces you to find out, before you spend the sample, whether the question is even answerable at the scale you can afford.
The transferable lesson is that randomization is the part that comes free from a library, and everything that makes the randomization usable is manual work you have to design in from the first line. Concretely, before the first real draw:
unit_id, the assigned arm, an is_test flag, and a timestamp. The unit_id is non-negotiable — it is the key the entire analysis hangs from.unit_id. If the outcome side doesn't know the key, the assignment side stamping it was pointless.is_test flag, set at draw time, is the only thing that reliably keeps development noise out of your effect estimate.Randomizing an assignment is a one-line call to a good RNG. Turning that into an experiment you can actually analyze is a join, a guard, a flag, and an arithmetic check you do up front. We had the one-liner and skipped the four boring things, and the result was a beautifully fair coin deciding an experiment that, on the day we looked, could not answer its own question. The coin was never the hard part.
Q. Is a fair random assignment enough to make an experiment analyzable?
No. A cryptographically fair coin guarantees the assignment is unbiased, but analyzability requires a join: a record linking each draw to the specific unit it was applied to and that unit's outcome. If the randomizer logs the draw but nothing stamps a stable key connecting draw → task → result, no code can ever compute the intention-to-treat comparison, no matter how fair the coin was.
Q. What is the minimum you must record at assignment time in an A/B test?
At the moment of the draw, stamp four things onto one joined record: a stable unit id, the assigned arm, an is_test flag separating development draws from production draws, and a timestamp. The unit id must then be carried through to wherever the outcome is measured. Anything you fail to capture at draw time generally cannot be reconstructed, because the linkage only exists at that instant.
Q. Why can't you backfill the link between assignment and outcome later?
Because the correlation you need is between a specific draw and the specific task that draw dispatched, and that pairing is only knowable at the moment the dispatcher hands the task off. If the draw was a stateless call that returned an arm and logged nothing task-specific, then afterward you have a pile of draws and a pile of outcomes with no key joining them. You can count each pile; you cannot pair them.
Q. Does an uneven split like 4:1 mean the randomizer is biased?
Not necessarily. With a fair p=0.5 coin and few draws, a 4:1 split is well within chance — a two-sided binomial test gives P ≈ 0.375, nowhere near significant. Treating a small-sample imbalance as a broken coin is a common error. The real problem with a small draw count is not bias, it is power: too few units to detect the effect you care about.
Q. What is a minimum detectable effect and why compute it before starting?
The minimum detectable effect (MDE) is the smallest true difference your experiment can realistically detect given sample size, variance, and significance level. Computing it up front tells you whether the experiment can answer its question at all. Skip it and an inconclusive result becomes ambiguous — you can't tell "the arms are genuinely equivalent" from "we never had the power to see a difference."
Q. Why is a falsifier useless if you can't compute it?
A pre-registered stopping rule only has force if the quantity it references can be measured. A kill condition like "declare design failure if assignment/outcome mismatch exceeds 20%" is unevaluable when nothing joins assignment to outcome — the mismatch rate is uncomputable, so the falsifier can never fire. A stopping rule you cannot compute is not a stopping rule.