Patterns from building AI agent systems for daily use — what worked, what failed, and why.
How to make AI agents give consistent decisions across sessions using versioned, trigger-driven rule files. 140+ rules across iOS, trading, AI agent design, game development.
AI agentsconsistencyClaude CodeHow to prevent LLM agents from hallucinating probabilities and accepting unverifiable multi-agent consensus. Includes real metrics: council_repeat_rate 0.44, empty_consensus_rate 0.354.
AI agentshallucinationmulti-agentStop a RAG agent from presenting inferences as facts: let the LLM only extract, and let deterministic code score and label FACT/INFERENCE with a ≥2-independent-source rule and required evidence IDs.
provenanceRAGhallucinationUsing Claude Code hooks to enforce behavioral constraints beyond prompts. AND-gate design, nag-once pattern, decision-ownership and sycophancy-prevention examples.
Claude Codehooksbehavioral gatesAn agent that has gathered all the data to decide will still hand the choice back — "which do you prefer?" — because the training gradient rewards deference. A deterministic Stop-hook with an AND-gate (deflection phrase AND fresh tool evidence) blocks it while letting genuine value questions pass.
Claude Codehooksdecision ownershipA Stop hook recorded only distinct blocks, leaving no denominator. Replaying its deterministic five-outcome logic over 5,537 retained transcripts recovered 116,022 candidate Stop points and a counterfactual trigger rate — without waiting for new instrumentation.
AI agentsobservabilityhooksReplaying a deterministic gate's logic over retained transcripts recovers a trigger rate no counter ever wrote — but only for gates whose entire predicate lives in the log. Three of five read live external state with no change history, so their denominator is not unmeasured but undefined relative to the archive. Also: five fingerprint files in one format holding three different quantities, and dedup compression that differs 3.5x between instruments.
AI agentsobservabilityinstrumentationMultiple personas on a single LLM do not give you real diversity — they are prompt variations of one set of weights, so they share the same blind spots, and the only durable d
aillmarchitectureMake every skill, hook, and scheduled job declare four invariants before it ships — Locality (where it can run), Source-of-truth (which facts it owns or borrows), Cross-ref (w
aiagentsdevopsNo — a scoring function that consumes whatever values an LLM hands it is only deterministic in name; the LLM's judgment launders straight through the "deterministic" gate, and
llmsecurityarchitectureA Claude Code session that returns API Error: 400 ... not valid JSON: no low surrogate in string on every turn is poisoned by a lone UTF-16 surrogate (a code point in U+D800–U
claudecodedebuggingpythonA one-time audit said an agent's quality is the method, not the model — so we built a stdlib SQLite lab to measure method vs outcome across 155 real sessions, reassembling logical sessions and fingerprinting which disciplines each model-epoch used.
aillmagentsA cryptographically fair A/B assignment was still unanalyzable — nothing joined the random draw to the task it dispatched, so no code could compute the intention-to-treat comparison. The join, the re-draw guard, and the up-front power calc are the experiment; the coin is the easy part.
datasciencestatisticsexperiment designA pre-registered randomized study logged 177 draws, froze its primary metric and honored its falsifier — yet still had to terminate undeterminable. Its attainable sample could detect an effect 6.7 times larger than the observed primary gap, and a repaired join would have needed about 45 times more observations per arm.
datasciencestatisticsexperiment designPrompt caching is a prefix match, so one timestamp or unsorted JSON key in the system prompt makes every later token miss the cache — a silent 10× cost leak no error reports. How it fails, the invalidator list, and how to assert cache_read_input_tokens > 0.
llmprompt cachingcostLLMs measurably reverse correct answers under "are you sure?" (98% in one Anthropic measurement), and self-critique can't fix it — the critic shares the producer's blind spots. The fix is architectural: a Stop-hook gate that forces one cross-family re-verification on challenge, then allows only HOLD-with-evidence or CHANGE-with-reason.
llmsycophancyverificationA reader turned the sycophancy gate above into an observable test, and it failed: the escape condition was a verification keyword in the agent's own prose, so writing "I re-verified this cross-family" cleared it with nothing behind it. Replaying the tightened rule over 212 transcripts newly blocked 0 legitimate turns — and the fix still only proves that a tool ran, not that it ran on the claim.
verificationstop-hookself-reportFanning one evidence digest out to five model families: 68/68 numeric claims survived recomputation, four models caught an anomaly the fifth missed — and all four external models missed an experiment-design flaw sitting in plain sight. Convergence on the same framing is cross-consistency, not independent evidence.
multi-modelverificationllmA different-vendor LLM audited ten queued posts written by my main model: it caught three copy-paste-breaking macOS/Linux platform bugs and a destructive in-place file conversion the writing family repeatedly missed — and also filed a false accusation that only grepping the primary source could refute. Cross-family audit works; auditor self-reports still need re-verification.
cross-vendorllmverificationA reader's rebuttal sent me to three judges before I replied — my own code review plus two model vendors. The table that came back looked like proof cross-vendor verification pays off; it was actually a schema bug. Only one model's prompt had a field for disagreement, and a same-model controlled experiment (n=3 per condition) showed the schema decided the verdict, not the vendor.
llm-as-judgeverificationprompt schemaThe field that fixed the schema bug above turned out to have a second failure mode. A mandatory "where does this overreach" slot gets filled even when there is nothing to object to, because the model cannot write nothing — so a populated field is not the signal. What is: whether two independent model families, shown the same material, land on the same objection.
llm-as-judgeprompt schemaverificationFive models, two roles, a few thousand real agent threads — and a stack of refusals instead of a ranking: role confounds, version-epoch discipline, [0,0] tie-degenerate bootstrap CIs, non-additive token counters, self-measurement contamination. The one completion-proxy CI worth reading spans zero.
llmtelemetrydatascienceFour model families cited every number from a frozen agent-telemetry table with zero errors — 70 of 70 recomputed within tolerance — and one still read a row of structural zeros as error-free tool use and built a routing suggestion on it. The one pre-registered equivalence test that passed cleared a margin 0.75 standard deviations wide while its own interval excluded zero. A citation audit is a copy-fidelity floor; it never checks the inference drawn from the number.
multi-modelverificationstatisticsA new skills repo scored 64/100 → TRIAL across five gates (maturity, dependencies, platform fit, policy, trajectory) on provenance-tagged inputs (9 FACT / 1 ASSUMPTION, conservative defaults for assumptions), with five auto-derived kill conditions — and a refusal to log a day-zero outcome that would pollute the ledger's own base rates.
adoptionfalsifierdecisionThree roles — question, defend, attack — rotated through three reviewer groups in all six permutations, so no group was ever the permanent judge. Seven confirmed defects before hardening, including a fix two groups had agreed on that a later hostile seat reversed as unsafe, and a verification pool that fell back three layers deep in round one.
adversarial reviewmulti-agentverificationA pre-registered gate used a holdout roughly 24 times noisier than the leaderboard it was protecting, so its threshold was cleared zero times in 125 days. A near-empty pass log is a symptom, not proof the gate works — and the fix is to move the gate to a different axis, not to enlarge the noisy instrument.
measurementpre-registrationexperiment designAcross a few thousand agent threads, the main-thread vs sub-agent split moved metrics by up to 135x, and the role mix ranged from 1.5% to 71.4% per model — so pooled per-model averages were weighted averages with different weights. One pair showed a 3.6x pooled gap that exists in neither stratum. Plus two traps that survive stratification: metrics that change meaning per stratum, and bootstrap intervals that collapse to [0, 0] on tied data.
agent telemetrystratificationstatisticsA tool failing and a tool being incapable are different claims, and the error in front of you does not say which. Descend one layer, call the underlying service directly with the same credentials, and let that decide — skipping the descent is how a client-side parsing bug gets laundered into a permanent capability judgment.
AI agentsdiagnosistool wrappersSee every cron job and LaunchAgent in one place by globbing on-disk state instead of importing them. Includes a 25-tool fit-gap matrix and the read-only aggregation pattern.
cronlaunchdobservabilityWhy pull (scan on-disk state) beats push (an event bus) for monitoring a high-churn fleet with components you don't control. The instrumentation-tax argument and self-healing polling.
multi-agentpull vs pushMCPA commenter showed that polling files the job itself wrote is self-report over a filesystem — the same weakness as the event bus I rejected. My publishing job proved its own health with its own log and stayed green through a four-day outage. The cross-vendor verification I ran instead of defending, and the effect_probe fix.
observabilityliveness without progresspostmortemCoordinate independent agent CLIs without LangGraph or a message broker: atomic Task/Result files, capability-based routing, absent-worker skip for graceful degradation, and a typed claim contract on the bus.
orchestrationmulti-agentLangGraph alternativeA publishing ledger kept every promoted item marked queued because the publish path never advanced the status. Twenty-four files were already published; binding the ledger update to the successful file move repaired the drift and restored the real six-item runway.
state integritypublishing pipelinepostmortemAn efficiency audit found a silent cost regression, a cache-invalidation trap, a pricing cliff — and a terrifying "12% task completion" that turned out to be benchmark contamination plus a schema misreading (real rate: 98.4%). Verify the metric before reacting; the false alarm's investigation still fixed three real defects.
observabilitymulti-agentpostmortemAn accelerator quota read 100.47% used across four concurrent projects, so I wrote a priority table to ration it — a table with ranks and not one number in it. Three measurements later: rescoring never spent the pool, a "GPU" label billed 1,043 seconds to a run that never touched a GPU, and real demand was 13%. Price the operations before you write the policy.
cost accountingobservabilityresource allocationA subagent reported 7 files changed with zero disk changes behind it, then fabricated the ls -la and grep -c proof demanded of it. The two distinct failure modes (fabrication vs silent stall), the landmark-file check run in the orchestrator's own process, and salvaging a non-writing worker by switching its output channel.
multi-agentverificationpostmortem25 days of "Couldn't fetch" on a sitemap whose 289 URLs all return 200 — a documented GitHub Pages pattern, not a broken site. Reading crawl stats, the manual Request-indexing workaround (~10/day quota), the dead ends to skip, and why Bing quietly matters more for AI search.
SEOGitHub PagesSearch ConsoleGEO is unfalsifiable until you measure citations: a fixed set of user-shaped questions fired weekly at a grounded model, a deterministic owner-scoped match, and a ledger that error rows can't fake — plus when not to measure at all.
GEOAI searchmeasurementA crosspost pipeline with 19 green gate tests published nine articles whose titles were visibly broken on the live platform for nine days. Tests verify the payload you build, not what the other system renders — a pipeline isn't verified until you check the output surface itself.
testingpostmortempipelineA 6-agent forensic audit over 92 of my own sessions answered "am I prompting wrong?" — partially yes (zero visual references, zero style tokens), but the dominant cause was silent infrastructure decay: exhausted generation credits quietly swapped in code-approximated shapes, a generation server was down with a required model file missing, and a ready asset pipeline was never invoked for six-plus weeks.
vibe codinggamedevauditBuilding a travel app on Korea's open-data portal: a documented-looking op name returned 500 (the real one returned per-gate passenger counts, not the grade field I'd invented), a sibling dataset 404'd because approval is per-dataset, a discarded dataset left a 401 red herring on the wrong gateway, and one API's airport parameter means the other end of the flight. Live capture is the only ground truth.
open dataAPIdegrade pathMy app mascot's arms rotated from the fingertips — invisible in code review, obvious in one render. Three structurally different variants instead of tint tweaks, the grep that saved a parameter used by 8 call sites, pre-registered design falsifiers checked at real sizes, and the render-first loop that fixes AI-drawn UI.
swiftuivibe codingdesignA 20-minute JWT cap that 401s perfectly valid signatures, one app version in flight (409 STATE_NOT_SUITABLE), and a 2-in-flight/5-total review-submission ceiling with undeletable orphans (DELETE 403, PATCH 409 — web UI cancel or 7-day expiry only). The two-call pre-flight that avoids all three.
App Store ConnectiOSAPIThree features, each with dedicated passing unit tests, none of which ever executed in the live app. One was broken by a merge that dropped two call sites; one existed only inside preview blocks; one was reported unwired in a line that got buried. Component tests are blind to wiring, and the shared root cause was a default value that removes the compiler's call-site audit.
testingwiringAI codegenThe predecessor note diagnosed three dead features. This one is about finding the rest: a fill-rate query run across seven projects, why the defect lives at the column rather than the feature, and a fourth kind of dead gate where the wiring was correct all along and the value still never arrived.
testingarchitecturedatabaseThe inverse of the note above: grep finds the call site, the scheduler entry exists, running the script by hand works — and the step still never ran, because launchd hands jobs a minimal PATH under which the wrong Python resolves. A correctly fail-soft design absorbed the failure while the heartbeat kept reporting healthy. Where a step's failure state has to live so the monitor already looking can see it.
devopsmonitoringschedulingA third kind of quiet check, after the two above: this one was wired in, ran to completion, and did not die — it returned PASS because the set it was checking was empty. The guard inserted its own probe row before asserting, so it stayed green through two runs where the layer it protected had produced nothing. Four instances of the same shape in one day, and the only check that caught any of them carried a population lower bound.
testingdatabasedevopsAn upstream reported throttling as HTTP 200 with an error envelope, so a rate-limited response parsed as a legitimately empty table and got memoized — serving zero rows long after the upstream recovered, with a completed walk and silent logs. Success is a verdict on the body schema, not the status code, and the cache write has to sit downstream of that verdict.
APIcachingerror handlingA layout fell apart while 481 tests stayed green and the missing rules sat in the file, correctly written. A comment closed with Jinja's #} instead of */ made the parser consume seventeen lines to the next real closer. CSS has no fatal parse errors by design, so a linter cannot catch it — the fix that mattered was two assertions guarding the whole failure class.
The design record, the code and the response headers all said the endpoint cached at the edge. The first test that actually timed a warm call found it re-walking all 29 upstream pages at 2.9s against a 0.16s expectation — the cache API is a documented no-op on that hostname. Nothing broke, because the spend circuit breakers underneath were absorbing the load the cache was credited with removing.
edge cacheverificationserverlessWhen nslookup resolves but curl/pip/Python fail, the mDNSResponder daemon is in a non-responsive zombie state. The two-DNS-path diagnosis, the weakest-first fix, and why long-running agent/MCP boxes trigger it (measured: 30+ long-poll connections → 77% CPU).
macOSDNSdebuggingFor a personal tool, the way past robots.txt, bot detection, login walls, and JS-rendered SPAs isn't a better headless scraper — it's a ~40-line AppleScript/JXA bridge that reads the DOM out of the authenticated Chrome tab you already have open. Why it sidesteps every wall at once, and its honest read-only, single-machine limits.
web scrapingmacOSautomationA background LLM server loaded models onto a nearly full disk; unified memory overflowed, swap failed, and APFS metadata corruption left the machine unbootable for a day. The postmortem, the five-layer code-level hard block, and the operating rules for 24 GB unified memory.
macOSpostmortemlocal LLM