On 2026-08-14, a screen-metric experiment in my YouTube Shorts pipeline ended undeterminable. Not because the metric misbehaved, but because a threshold I had pre-registered depended on a number I had never pinned down. Once I fixed that, a second, quieter version of the same mistake was already waiting one layer downstream — inside the rule I wrote to fix the first one.
The pre-registration set a single bar: motion median has to be at least 50% of the original for a clip to pass. It said nothing about how many frames that median gets computed from.
Measured across 8 frames, the clip's motion median came out at 47.5% — below the bar, falsifier fires, clip fails. Measured across 24, it came out at 56.5% — above the bar, falsifier doesn't fire, clip passes. Same clip, same threshold, same metric definition — only a sampling parameter the pre-registration never fixed, and that alone flipped the verdict's sign.
A threshold with an unpinned measurement isn't wrong, it's undefined: pass on one run, fail on the next, both honestly computed. (This isn't the underpowered-sample problem from Perfect Instrumentation Won't Save an Underpowered Experiment — that piece has a real, fixed measurement and not enough of it; this one never had a fixed measurement at all.)
I wrote the fix into that experiment's retro (report/simreveal-axisB-official-render-20260814.md, §4-3), on the spot. (The passages quoted in this section are translated from the Korean original.)
Rule starting with the next pre-registration: a rate prediction has to pin its
measurement parameters — sample count, window, sampling rule — alongside its
threshold. A threshold without a pinned measurement can't adjudicate anything.Same commit, next pre-registration document (report/prereg-simreveal-axisB-replication-montecarlo-20260815.md). Its §2 followed the rule exactly: window fixed to [0, 5.5s], sample count committed to report both 8 and 24, sampling rule spelled out. It said so directly:
The reason axis B ended undeterminable on motion median on 08-14 was that the prediction fixed only the threshold (50%) and never fixed the sample count (§4-3). Enacting that lesson here for the first time.
Less than a day after I wrote the rule, the person who wrote it followed it. So far, a pipeline working exactly as designed.
On 2026-08-15 I was doing unrelated work — an audit forcing machine-readable front matter onto every pre-registration document — and for the first time counted the whole corpus instead of one document. report/prereg-*.md came to 3 files. Only 1, the Monte Carlo replication above, had pinned the rule's three required slots (sample count, measurement window, sampling rule) in prose at registration.
1/3. I read that as "the rule didn't survive 24 hours" and wrote it into the retro — a cousin of an older counting mistake, The Count Read in the Thousands. It Was Thirty-Seven Things, Recounted., though that total was real and just uninterpreted. This one was about to be flat-out wrong.
I wrote the sentence before checking the thing I should have checked before writing it. Then I checked when the other two documents were actually born:
$ git log --diff-filter=A --format='%h %ad' --date=short -- report/prereg-*.md
03da812 2026-08-15 # the commit where the rule was written = the montecarlo replication's birth commit
670cdff 2026-08-14 # ink-coverage
b3478d8 2026-08-14 # sim-graph-revealBoth of the other two were created before the rule existed. They were never in scope. Counting them as violations meant counting retroactive non-application as non-compliance. The correct denominator isn't 3, it's 1 — compliance rate 1/1, 100%.
The lesson isn't "the rule died." It's that the same data got read two opposite ways, on the same day, by the same person:
| reading | fraction | conclusion | why it's wrong |
|---|---|---|---|
| count the numerator alone | 1 compliant case | the rule is alive | only one case was ever put to the test |
| extend the denominator back past the rule's birth | 1/3 | the rule is dead | it puts out-of-scope documents in the denominator |
Both are wrong, for the same reason: the denominator was never defined.
(A different flavor of the same word: The Gate Only Logged When It Fired. I Replayed 116,022 Candidate Stop Points to Find the Rest. is about a denominator never written down anywhere, reconstructed from retained logs; this one was never missing, just scoped wrong.)
The corrected 1/1 = 100% isn't comforting either — maximum reassurance and minimum evidence at once, since the rule has only ever been tested once. That one test case had already declared, in its own text, that it was "enacting this lesson here for the first time," so the story of the rule working was complete inside a single document, written, followed, and later re-read by the same person. "Wrote the rule" and "followed the rule" described one data point, not the whole corpus.
One clause at the outset would have prevented all three readings: had the fraction been defined the moment the rule was written, it would have read 1/1 from day one, and the denominator being 1 would have been the signal itself — don't adjudicate yet. The discovery wasn't a designed audit either, just luck: the unrelated front-matter task happened to require counting the corpus start to finish.
I moved the rule out of prose into something machine-checkable: schema: prereg/v1 front matter on line 1 of every pre-registration document, plus a checker (shorts-factory prereg, in src/shorts_factory/experiment_prereg.py) scanning the corpus for missing slots, unevidenced exemptions, and undeclared amendments. I transcribed front matter retroactively into all 3 documents; where a slot hadn't been pinned at registration time, I left it null — backfilling with a post-hoc value would have been a lie about what was pinned when.
That fix reproduced the exact failure it was fixing. Nothing calls the checker: no git hook, no CI workflow, no scheduler in the repository runs this subcommand.
# the checker exists
.venv/bin/python -m shorts_factory prereg # or: agent-youtube prereg
# but nothing wired to the repo calls it (confirmed 2026-08-15)
grep -rl "prereg" .claude/hooks/ .codex/hooks.json # no resultsThe thing I built to fix "a rule written in prose gets forgotten" landed exactly where it can be forgotten, for the same reason: nobody calls it automatically. I nearly got that claim wrong too — counting "automation surfaces," I almost stopped after git hooks and CI. Actually counting found 4 active hooks (session_start_state.sh, pre_upload_channel_check.sh, post_render_verify.sh, post_edit_module_tests.sh) plus SessionStart/PreToolUse/PostToolUse registrations in .codex/hooks.json — none call this checker. "There's no automation" isn't a sentence I get to write without naming what I checked.
This is where the essay's own honesty gets tested. Running the same commands against the repository as of 2026-08-15, the day of the audit:
$ ls report/prereg-*.md | wc -l
3
$ grep -c '^measurement_spec:' report/prereg-*.md
report/prereg-ink-coverage-20260814.md:1
report/prereg-sim-graph-reveal-20260814.md:1
report/prereg-simreveal-axisB-replication-montecarlo-20260815.md:1All 3 documents now carry measurement_spec:. That is not "the rule recovered to 3/3" — it's the value after the retroactive transcription above, a product of the audit work, not of the rule being followed; the field didn't exist before that work ran. "1/3" was counted before the transcription, while the rule existed only in prose; "3/3" is counted after. The two can't answer the same question — "is the rule being followed?" — because having the slot doesn't mean compliance. It means missing slots are now visible as null instead of silently absent (sim-graph-reveal's sample_count still is).
A follow-up check, dated 2026-08-31: the corpus gained one document since that audit. report/prereg-publish-window-20260818.md was born in commit c12f93f on 2026-08-19 — four days after the rule, so it belongs in the denominator. Its front matter pins all three required slots (sample_count, measurement_window, sampling_rule) at registration, plus three amendments, each with a dated, non-silent what: entry. That makes the corrected rate 2/2, not 1/1 — still short of the 3-or-more this piece's second falsifier actually asks for, but the first new data point since the rule's birth, and it points toward compliance.
git log --diff-filter=A or an equivalent creation check.prereg-*.md filename convention was that mark.n/n = 100% is maximally reassuring and near-zero information at once; keeping the denominator in view when citing a fraction removes most of this trap..claude/hooks/, .codex/hooks.json, or equivalents), schedulers.This piece's claim is not "rules get forgotten." That conclusion isn't supported by this incident's evidence — I wrote it into an early draft and falsified it myself, above, where the 1/3 reading turned out wrong. The rule got one chance to apply, and that chance was followed.
What's left is narrower: a compliance rate with an undefined denominator reads in whichever direction you're primed to read, and an n=1 compliance rate reassures without informing. The sample here is one rule, one repository, one observation.
Two ways this could be shown wrong:
The pre-registration fixed the threshold (motion median at least 50% of the original) but never fixed how many frames the metric would be sampled over. Measured at 8 frames the value came out 47.5%, which tripped the falsifier; measured at 24 frames it came out 56.5%, which didn't. The threshold was pinned, the measurement wasn't, so which side of the line the same clip landed on depended on a parameter nobody had committed to in advance.
Once, yes. The very next pre-registration document, written in the same commit, pinned the measurement window, committed to reporting both the 8- and 24-frame counts, and named the sampling rule — and said explicitly it was enacting the lesson for the first time. That is the only time the rule has been tested, and it held.
Because a day later I counted the full corpus of pre-registration documents for an unrelated reason and found only 1 of 3 had pinned the required slots in prose, read that as "the rule didn't survive 24 hours," and wrote it down. That reading was wrong: the other two documents were created before the rule existed, so they were never in scope. The correct compliance rate was 1/1, not 1/3, as of that 2026-08-15 audit — but a denominator of 1 isn't evidence a rule is durable either, it just means the rule had been tested once so far.
I turned the rule into a machine-checkable schema and a corpus-wide checker script, which is real progress. But nothing in the repository calls that checker — no git hook, no CI job, no scheduler. Four active hooks exist and are registered elsewhere in the repo, and none of them call it. Writing an enforcement mechanism and having it actually enforced turned out to be two different pieces of work.
Yes, once more, as of a 2026-08-31 recheck. A fourth pre-registration document, for a publish-window experiment, was born after the rule (git birth commit dated 2026-08-19) and pinned all three required slots at registration, plus three properly declared amendments. That puts the compliance rate at 2/2, not 1/1 — but the piece's own falsifier asks for 3 or more further rule-applicable outputs before the rate means anything, so this is one data point toward that bar, not proof the bar has been cleared.
Cite the fraction, not the last compliant instance, and never drop the denominator — rule-compliant outputs divided by rule-applicable outputs, bounded by the rule's own birth date. If that denominator is 1 or 2, the honest answer is "not enough signal yet," not a percentage.