Research notes29 October 2026

Who Caught It

Berg Atkinson · 29 October 2026 · The paper: Who Caught It

Who Caught It, summary card: 740 logged misses in one practice's AI-assisted work, 13 June to 8 October 2026. Who caught them: the operator 378 (51.1%), the AI itself 291 (39.3%), another AI 51 (6.9%), an instrument 19 (2.6%); 448 of the 740 got past the AI (1 unattributed). Of the operator's 378, 159 carry his own words and 219 rest on the AI's word; the AI writes the row in three of the four routes, 205 of the 219 predate September, and capture is now about 80%. The operator's share of catches by month: June 71.1% (n = 38), July 59.6% (n = 277), August 53.6% (n = 196), September 49.4% (n = 77), October 1 to 8 28.3% (n = 152), or 36.5% without a burst of 100 rows logged on 3 and 4 October. Two ways to measure the human gate: Anthropic's planted test of 7 August 2026 counts one planted bad command per tester, humans caught it 13.6% of 1,053 testers, a classifier caught 89%, about 17% early and 5% after 50+ prompts; this register counts every logged real miss, the operator's catch rate is not recorded yet, the AI caught 39.3% of its own misses at most, and the operator share went from 71.1% in June to 28.3% in October. Counts recomputed from the ledger in a read-only audit (10 October 2026) by a separate AI of the same model family.

How do you know the human in the loop is actually catching what your AI misses? Best recent answer: a planted test. In August, Anthropic put one dangerous command in front of each of 1,053 paid testers, and humans caught it 13.6% of the time. Good test! But it's ONE test, planted, in a test environment. That's not a Tuesday on real work. A human WATCHING the loop isn't a human IN it.

So I keep score on real work. Every logged miss in my AI-assisted work goes into a public, hash-chained register (wolfberg.ai/register), and every row says who caught it: me, the AI itself, another AI, or an instrument.

And the count is NOT the interesting part. The chain is. The AI checks itself first, and whatever gets past it gets caught by another AI, an instrument, or me. So every catch of mine is a miss that got past the AI...which means the AI has a real catch rate: about 4 in 10 of its own misses, at most.

Side by side, planted test | my register:

  • What it measures: a planted bad item, once, in a test | real misses, who caught them, in order, all the time
  • Catch rate: humans 13.6% of 1,053 testers (Anthropic, Aug 2026) | the AI 39.3% of its own misses, at most. Mine isn't recorded yet.

A separate AI session audited the register, read-only, meaning it could read every row and change none. The weak link it found is the same question my homework post ended on: who WROTE it down? In three of the four ways a row gets written, the AI writes it...including the rows that say I caught it 🙄 So "I caught it" was mostly the AI's word. It's getting better (rows with my actual words went from 0 of 27 in June to 35 of 43 in October), but a catch nobody logs doesn't exist. Between 13 June and 27 August my AI told me an idea of mine was new 208 times. The register has one row about it lol.

So what's the ledger good for? It's where the fixes come from:

  • Wrong diagnosis was the top miss type in July, and the AI had caught almost none of its own. Now it's a forced check: before the AI asserts a diagnosis, it names what would prove it wrong and reads that first.
  • A miss that came back (logged as a repeat) became a standing rule on which calls the AI makes and which come to me.
  • The two biggest miss types each got their own test.
  • Three misses in September got past the guard that makes the AI read before it claims. So now the guard checks the repo too.

What's next: rows that say I caught it carry my words, or something the AI didn't write, like the transcript. A record of what was in front of me first, so what got past ME gets counted too. And planted items or an outside check, for my rate.

Same caveat as always: one practice, and the auditor is the same model family as the team it audited 😁

If the human is the check, somebody has to check the human...and it can't be the thing being checked 🤷‍♂️

The paper: wolfberg.ai/papers/who-caught-it/

Sources

  1. Never let your AI grade its own homework, the post this one builds on
  2. The register: every logged miss and who caught it
  3. Nobody Else Has Done This, the companion note on novelty assertions
  4. Anthropic: auto mode is now the default in Claude Code (the 1,053-tester study), 7 August 2026 claude.com
  5. Habituation at the gate: rising approval in human review of AI agent code (Yu et al., 2026) arxiv.org
  6. AI agents push humans out of the loop (Mitchell, Ghosh and Passi, 2026) arxiv.org
Read this before you believe any of it. Keystone runs in production in its own AWS account. The rest of what you see here is a running prototype, not an accredited or certified system. The register, the war room, and the brain are receipts, not revenue. Every claim on this site links to something you can check. If one doesn't, tell us, and we'll fix the page.