Who Caught It: Keeping a Public Score on the Human in the Loop

1Introduction: where things stand

On 7 August 2026 Anthropic published the study behind making auto mode the default in Claude Code. Each of 1,053 paid testers, recruited through a vendor, met one planted dangerous command in a purpose-built environment where approving it ran nothing dangerous; testers knew they were being evaluated but not what was measured. Humans caught 143 of the 1,053 (13.6%); the auto-mode classifier caught 937 (89%). Human catches fell from about 17% early in a session to about 5% after 50 or more prompts (Anthropic, 2026b). In February the same lab had reported from real telemetry that roughly 20% of new users' Claude Code sessions use full auto-approve, rising to over 40% as users gain experience (Anthropic, 2026a).

The research of the last few months points the same way. Yu et al. (2026) followed 400 repeat reviewers across 11,429 reviews of pull requests written by AI agents and found the population's approval rate rose from 30.1% to 36.8% while inline comments fell. Mitchell, Ghosh and Passi (2026) argue that current agent designs do not support effective human oversight and contribute to its degradation. Reid (2026) argues oversight should be judged by intervention capacity rather than by a human's presence. Song (2026) planted 150 errors in 30 artifacts and found the best cross-model AI reviewer not significantly different from a fresh session of the same model, which does not establish equivalence; two reviews that differed in model found more planted errors. Regulation has slowed: the EU's Digital Omnibus moved the application of the AI Act's high-risk chapter, including the human-oversight article, to 2 December 2027 for Annex III systems (Regulation (EU) 2026/1744).

These are careful measurements, and almost all of them are one-off, planted, or about approvals rather than catches. A one-off planted test gives a rate under test conditions. An approval log shows that a human clicked, not whether the thing approved was wrong. Neither tells a practice what its own human gate catches on an ordinary day of real work, or how that changes over months. This note describes a register built to record exactly that, and what an audit found when we asked who writes it.

The earlier post in this series, "Never let your AI grade its own homework" (Atkinson, 2026a), measured the cost of self-report: the AI team logged its own session starts in 75 of 579 sessions, and its own logs caught corrections on 1.76% of the turns where a hand-graded detector found them on roughly a third. It ended on one question: who wrote it down? The register was built to answer that for every miss. The audit asks the same question of the register.

2Our position

A human gate should be measured the way any control is: continuously, on real work, by what it caught and what got past it. The record of who caught each miss should not rest only on the word of the party being measured, and real catches need planted items beside them to give a rate.

Three parts, each with a different status. That oversight should be measured rather than assumed is the position of most of the recent work in Section 3 and of Green (2022) before it. That the measure should be continuous, on real work and by catcher type is this practice's design, in use since June 2026. That the catcher label should be witnessed by something other than the writing seat, and paired with planted items, is what the audit in Sections 4 to 6 says the current register lacks; it is untested here.

3What else exists, newest first

We searched for measurements of human catch rates in AI workflows, records of who caught AI errors, tamper-evident agent logs, and oversight rules, weighting the last eight weeks most (about 20 queries, 30 sources, run 10 October 2026). Table 1 lists the closest, newest first. In that search we did not find another practice publishing, continuously, which party caught each real miss in its AI-assisted work; the nearest are a one-day account of fifteen errors and the checks that caught them (Ritschel, 2026) and an AI agent's public log of its own claim checks (Stellar, 2026). That is a bounded statement about one search, not a claim that nobody does it.

4How the register works

This section reports a read-only audit of the register's code and data by a separate AI instance on 10 October 2026, with file and line references kept in the working report (Atkinson, 2026c). It read the producer, ingest, page and integrity tests in full and the larger writer and WORM scripts in part, re-implemented the chain, signature and count checks, and ran no test suite. Exhibit 5 is the data flow.

4.1Writing a row

Misses go into one plain JSONL ledger (a live file plus an archive for pruned rows) by four routes. Code seats call a command-line logger with a miss type and a source of operator, cross_instance or self; the logger rejects unknown types and sources and appends under a file lock. Chat seats drop a JSON file into an inbox that an ingest script reads about every 30 seconds, using the file name as the event id so a retry cannot count twice; it does not pass the timestamp or the operator's words through. A poll-down copies rows chat seats wrote in a Notion ledger (251 rows, 4 July to 31 August). Probe scripts, run outside the seat being measured, append rows with source probe. Separately, hooks write guard rows (evaluations, refusals, overrides, stop certificates) that record a hash and rule labels, never the subject text; these are not misses and the register does not count them.

A published miss carries its type, weight, who caught it, the first sentence of its note (anonymized, cut at 200 characters), its timestamp, and its place in the chain. Eleven miss types are counted, each with a fixed weight from 3 (format slip) to 16 (an error that would have shipped).

4.2Who decides who caught it

The register does not decide who caught a miss. It publishes the source field of the row, and whoever writes the row sets that field.

The label is a pure function of the row's source: operator becomes operator, self becomes self (unprompted), cross_instance becomes peer instance, probe becomes external instrument, and anything else becomes unattributed. Until 3 October an unknown source fell through to operator, which had mislabeled 44 of 624 rows; the unmapped value was cross_instance, so the fix moved peer-caught rows out of the operator count, and 43 published rows changed label in the next build. Anyone who read the page before 3 October saw no peer rows and a higher operator share. Nothing in the logger or the ingest checks the asserted source against an independent signal. One client-side script refuses an operator row without the operator's verbatim words; the logger itself does not. The at-a-glance figure shows the result: 159 of the 378 operator rows carry the operator's words, and the other 219 rest on the writing seat's word. Capture has risen: by month, 0 of 27 operator rows carried his words in June, 31 of 165 in July, 61 of 105 in August, 32 of 38 in September and 35 of 43 in October; 205 of the 219 date from before September. The chat-seat file drop does not pass the operator's words through, so some rows could not carry them. Probe rows, 19 in all, are the only ones written by a process other than a seat.

4.3Building and signing the record

Each night a producer reads the live ledger and the archive, keeps the eleven miss types, removes duplicates by event id (879 raw miss rows become 740), and sorts newest first. It then chains five gate entries and the 740 misses: each entry's hash is SHA-256 over the previous hash and the entry's canonical JSON. The five gate entries' hashes and the chain head are signed with Ed25519, and the public key and head signature are published with the file. Since 29 October 2026 the producer no longer signs. A separate nightly signer, outside the machine that writes the record, recomputes the chain, refuses one that does not verify or does not extend the chain it signed before, and signs the head and the five gate hashes with an Ed25519 key held in AWS KMS. The /register page checks those signatures in the reader's browser against the published key. The key's policy lets only that signer use it; an administrator of the key can change the policy, and AWS records the change. Misses themselves are hash-chained, not individually signed.

The audit verified the file independently: 745 entries, 0 broken links, the head equal to the last hash, and the head signature and all five gate signatures valid under the published key, which was the same in the builds of 4, 9 and 10 October. It also found what the chain does not do. The file is rebuilt whole every night, and because new rows are inserted above old ones, no entry's hash survives from one night to the next even when its content does; continuity over time comes from git history (48 commits) and, for some rows, the separate WORM ledger. Diffing all 48 builds, four transitions changed earlier entries: 3 July (a 15-row file replaced by the ledger; 9 of the 15 do not reappear), 28 August (1 entry), 1 September (11 text changes) and 2 to 3 October (the 43 relabels above and 5 weight changes).

4.4What the page shows and leaves out

At build time the page recounts misses from the entries rather than trusting the file's own totals, recomputes every hash and link, and draws the catcher split only if the declared split equals the recount. It shows the newest 40 of 745 entries, the head hash and the public key, and its own lists of what the record proves and does not prove. It leaves out the other 705 entries individually, the session ids, operator words and raw notes, unlogged misses, probe passes and declines, and all guard rows. The record file itself is not served on the site, so a reader cannot repeat the recomputation; the page says so. The build checks no signature, and a broken link shows as a number on the page rather than failing the build.

4.5Tests

An integrity test seeds five tampers (an edited entry, an edited weight, a deleted entry, reordered entries, a broken previous-hash link) and requires each to fail verification, runs the producer end to end on a planted ledger with every kind of source including missing and malformed ones, and requires the published split to equal a recount. Source checks on the page require counts to come from entries, not from the file's totals. These run in CI on changes to the producer, page or data; checks that need files on the operator's machine print NOT RUN in CI. A hand edit that rewrites an entry and every later hash recounts clean; the nightly rebuild from the ledger overwrites such an edit the next night, and git keeps it.

4.6The gate half

The five gate entries on the page are fixed examples of the gate's rules, the same every night, undated and signed by the register's key, not by any person; the page says so. The real gate record is in the ledger as guard rows written by hooks between 2 September and 9 October 2026: 5,032 evaluations, 863 guard-refusal rows, of which 243 are marked blocked and 620 carry no outcome (584 from the keystroke guard, 145 from a read-before-claim guard, 61 from a stop-delivery guard, 37, 8 and 2 from three smaller guards, and 26 with no guard named), 3 overrides (all on 16 September), and 1,274 stop certificates, 14 of them refused. None of it is published yet. A separate nightly ship-or-hold check over regression tapes (flip_gate) is not part of the register.

5What the record shows

5.1Who caught the logged misses

Of 740 logged misses, the operator is credited with 378 (51.1%) and the seats with catching 291 of their own (39.3%); peers caught 51 (6.9%) and instruments 19 (2.6%).

Weighted by severity the split barely moves: the operator's misses carry 3,707 of 6,926 weight points (53.5%). The operator dominates the repeated misses (118 of 177) and wrong diagnoses (101 of 161); the seats' own catches dominate self-corrections (44 of 46), as the type implies. Errors that would have shipped, the heaviest type, split 67 operator, 53 self and 12 peer.

5.2Over time

The operator's share went from 71.1% in June to 28.3% in the first eight days of October. June rests on 38 rows, the weekly series is not steady, and the ledger cannot say how much of the change is behavior and how much is logging practice.

Two filing events sit inside the trend. On 15 July, 77 rows (55 of them operator) were written in one bulk stamp; without them July's operator share is 55.0% rather than 59.6%. On 3 and 4 October three Code seats logged 100 rows (73 self, 24 operator, 3 peer), most of October's 152; their timestamps spread over about 42 hours, and the ledger cannot say whether they were filed after the fact or as the work happened. Without them October's operator share is 36.5% (19 of 52). June's 71.1% is 27 of 38 (95% interval about 55% to 83%), and the weekly share since September has ranged from 25% to 68%. We report the change and do not read it as the seats getting better.

5.3What the operator caught that the record did not

A companion note in this series (Atkinson, 2026b) counted 246 verified assertions in the operator's chats, between April and August 2026, that an idea of his was new, unique or something nobody else does; 208 fall between the register's first row on 13 June and the last assertion on 27 August. Of the 69 that could be dated against a track record, 66 came after an earlier statement of the idea; the rest were not tested. None of the 246 appears word for word in the practice's published work (a seven-word match; paraphrase not tested). The register holds one row about a novelty claim: 11 September 2026, caught by the operator, logged as a repeated miss, "CLAIMED NOVELTY ON AN INCOMPLETE SEARCH, again." The nearest class is unsearched claims, 56 rows (30 operator), most of them not about novelty. The companion's reading, from one practice: the filtering was the operator's, and almost none of it was logged. The register measures logged catches, and seats do the logging.

6What the record does not show

The audit listed ten gaps. We order them by how much they limit what the register can claim.

1. Self-asserted attribution. Every row's catcher is asserted by its writer. 219 of 378 operator rows carry no operator words, and the 291 self rows are the seats' own account. For operator rows this is the same self-report problem the earlier post measured, one level up: the seat no longer grades itself, but it records the operator's grade. The 291 self rows are still self-grading. In September and October, 14 of 81 operator rows carry no operator words.
2. No denominator. The register counts logged misses and a rate per hour, not misses per opportunity. It cannot say what share of misses the operator catches, only what share of logged catches are his. Anthropic's planted test has the denominator the register lacks. Read as a chain, though, the register does give the AI a catch rate (added 29 October 2026): it caught 291 of its own 740 misses (39.3%), and 448 (60.5%) got past it to the operator, another AI or an instrument. That is an upper bound, since misses that got past every catcher are not rows. The operator's rate needs a record of which misses went past him first, which the register does not keep.
3. Unlogged catches. A miss nobody logs is invisible, as the novelty case suggests (one novelty row against 208 assertions in the chats). Five operator corrections from 4 and 5 July sit in the ingest's rejected folder, refused for vocabulary.
4. One reviewer's findings do not arrive. Nine findings from a Claude Science non-author review on 7 October are staged in a folder the ingest does not read; no row has been written through that seat's channel, and it is not on the instance roster. One Code-written peer row (8 October) credits Science as a catcher. A plan to connect it exists; the change had not landed in the files read.
5. Partial sealing. The WORM ledger verifies clean (16,735 entries, signed since 4 October) but seals 219 of the 740 published misses (measured). The audit infers the cause is that its ingest reads only the live file (inference, not tested). Edits to the other 521 would be visible only through git history.
6. A narrow integrity claim. The chain is rebuilt nightly, the record file and entry hashes are not served, the build checks no signature, the register chain and the WORM head do not reference each other, and, at the audit, the signing key file sat on the same machine as the seats. That last gap closed on 29 October 2026, when signing moved to a key held in AWS KMS (Section 4.3).
7. A default that writes self. The Notion poll-down still turns a blank source into self, a default the file-drop ingest removed on 30 August; 60 of the 251 Notion rows are labeled self and cannot be told apart from blanks. Separately, 205 rows, almost all from the command-line route, have no event id, so their route cannot be checked.
8. The gate half is illustrative. The page's gate entries are examples; the 863 guard-refusal rows (243 blocked) and 3 overrides are not published.
9. Timestamp quality. 77 bulk-stamped rows and 71 date-only rows; the page discloses the first.
10. Hand-copied tables. The miss-type weights and gate rules in the producer are copies of other files, and their drift checks run only on the operator's machine.

7What it means

7.1For anyone with a human in the loop

Count what the human catches, on real work, and keep counting. A one-time test tells you whether the gate works under test conditions; a running record tells you what was caught, by whom, and how that changes as the people and the tools change. What got past every catcher needs planted items or an outside check. Then ask the series' question of your own record: who wrote it down? If the AI being checked writes the row that says the human caught it, you have moved self-report one step, not removed it.

7.2For anyone building or regulating oversight

We found three hash-chained agent-log designs within six weeks (a paper, a pull request and a recipe; two read as snippets). None we found records who caught an error, or a way to witness that field. Article 14 asks that overseers be able to understand, resist over-reliance, override and stop; Article 14 itself does not ask anyone to show how often they do. With its Annex III application now December 2027, a practice that wants to show effective oversight has time to measure it rather than describe it.

7.3For this practice

The audit points to changes; none is made yet, and each is a proposal, not a result. An operator row should carry the operator's words or a witness the seat did not write, such as the harness transcript, which a script already reads for guard activity. A small number of planted items in the operator's own queue, the method of Anthropic's study and of the canary practice in Table 1, would give the register a denominator. Rows should be sealed before they are pruned, and the register's chain should reference the WORM head. The record file should be published and its signatures checked at build, with a break failing the build. The Claude Science review drops should be ingested, the Notion default to self removed, and the real guard decisions published in place of the five examples.

8Limits

This is one practice with one operator and a small AI team, so none of its rates generalize. The audit was done by an AI instance in the same family as the seats it audited; it read some large files in part, did not open the key files, and ran no test suite, so its integrity checks are re-implementations and its account of which key signs in production is unverified. Seven of the 19 sources in Table 1 (14 of the 30 the search found) were read through search snippets, the OpenAI row rests on secondary reporting, and the search ran about 20 queries. The change in the operator's share is confounded by a 100-row burst on 3 and 4 October that the ledger cannot separate from behavior. The novelty comparison in Section 5.3 sets chat assertions against a ledger that chat seats reached only by file drop or Notion, over windows that do not match exactly, so some of the gap is plumbing as well as practice.