In the Loop or Watching the Loop? The Human Gate Is a Rater Too

1Introduction: where things stand

Newest first, because in this field a source’s age matters. Ages are days before 16 October 2026. In this note a “seat” is one Claude instance with an assigned role, and “the operator” is the register’s label for the human who runs Wolfberg.

• 16 October (0 days). The question of how to measure whether the human is in the loop or watching it was put to the analysis seat, and a joint miss ledger that records both the AI seat and the human was begun: twelve rows from one session (Section 7).
• 15 October (1 day). Wolfberg’s public miss register, rebuilt nightly, lists 740 misses logged between 13 June and 8 October 2026 and who caught each one (Wolfberg, 2026a). Section 6.3 reads it against the paper.
• 14 October (2 days). micro1 published a landing page for the paper with four headline tiles, an interactive explorer for the horizon and breakeven calculations, and a closing section on what the work does not show (micro1, 2026).
• 10 to 13 October (3 to 6 days). Edge updated on 10 October and Chrome on 13 October on the working machine (version numbers in Appendix A), two Windows updates followed on 13 and 15 October, and the tools’ sandbox permissions changed. Headless PDF rendering hung in those days and a flag that disables the sandbox fixed it; the cause was not isolated. Section 6.2.
• 4 October (12 days). Zhang and Esposito posted the working paper, arXiv:2610.07003v1 (econ.GN).
• 25 September (21 days). Katoch, Selby, Großmann and Vollmer posted a workshop paper arguing that LLM judges carry biases, including differing severity, that more comparisons cannot remove, and fitting a latent-variable model that corrects for them (arXiv:2609.31184).
• 1 September (45 days; first posted 14 July, 94 days). Sachdeva and Boudol fit many-facet Rasch models to annotations from nine LLMs and report that LLMs differ from human raters in severity, item-level calibration and rating-scale use (arXiv:2608.27463v2).
• 7 August (70 days). Anthropic’s auto-mode study planted one dangerous command in front of each of 1,053 paid testers; 13.6% caught it, falling from about 17% early in a session to about 5% after 50 or more prompts (Anthropic, 2026b, as summarized in Wolfberg’s forthcoming note on the human gate; the source was not opened for this note). Section 7.
• 4 February (254 days). Ford, Van Doren and Dix compared eight model releases on a fixed set of 726 adversarial prompts, using 82,256 human harm ratings, and found attack-success rates moved in different directions across generations (arXiv:2602.04739). Older background, kept because the paper leans on it as longitudinal evidence of release-to-release drift.

This note builds on Wolfberg’s series piece on never letting a model grade its own homework (Atkinson, 2026a). That piece argued for a check from outside the model. This one asks the follow-up: when the outside check is a person or another model, how is it known to be the same checker tomorrow?

2Position

Position. A score is a reading of the agent made by something, and the something is part of the number. The paper’s strongest contribution is to say so with figures: on its data the roster change was about as large as the improvement, so the two canceled. Its three checks (rater comparability, evidence currency and corroboration) are the right ones. Its weakest part is the one the landing page puts in a tile: a five-week expiry date that rests entirely on a single step the change log does not explain.

Two claims are often run together, and they are kept apart here. One is that scores have to be made comparable across raters, which the paper shows well. The other is that evidence has a shelf life measurable in weeks. The second is directionally right, and the paper’s own caveats say the number is an illustration. The method is worth taking (estimate the drift rate that the records do not explain, then compute how long a reading is good) and the figure is not.

The practical consequence is the one the series keeps arriving at, with a twist. Checking from outside the model still holds. The checker also has to be checked, because the checker is a rater: it has a severity, the severity may differ by question, and it can change without anyone logging a change.

3What the paper shows

3.1The design

The agent, Zara, conducts live job interviews for micro1 on a real-time voice model plus a text model. From 26 March to 24 August 2026 the firm’s trainers watched recordings and scored them on a nine-family rubric: conversation structure, interruption handling, latency, speech pacing, coherence, audio-visual sync, freezes, audio quality and interview outcome. The stability index is the mean of the nine family scores, standardized; trainers also flagged “critical failures”. In all, 2,611 production recordings were scored in 14 batches of 68 to 335. Reviewer A scored every batch and scored alone until 21 May; reviewer B joined then and scored roughly half of every batch; reviewer C scored part of the 21 May batch. The authors also have the agent’s change log (three production deploys: 12 April, 16 July, 29 July) and daily support-ticket counts from 5 May.

3.2Finding 1: a new reviewer looked like a worse agent

Which reviewer scored a recording predicted its score far better than which fortnight it came from: batch fixed effects explained 1.4% of case-level variance in the index, and adding reviewer fixed effects raised that to 16.2%. Reviewer B scored 0.792 SD below A (SE 0.045). The gap is not one number: it is -1.40 SD on latency and -0.13 on conversation structure, and B flagged critical failures less often than A (2.0% against 5.0%). Raw averages rose only 0.002 SD a month; with reviewer effects removed the trend is 0.102 (SE 0.021). The smoothed latent index rises from 0.05 to 0.58, a change of 0.53 (SE 0.07). Critical failures start at 2.8%, peak at 6.3% around 9 May and fall to 2.5% by 10 August. The authors’ rule: overlap raters, and never read a raw average across a changing roster. They add that the same applies to model judges: “a judge version is a rater.”

3.3Finding 2: the agent moved when nothing was deployed

With the state allowed to jump at the three deploy transitions, what is left is movement the change log does not explain. Its weekly standard deviation is 0.086 (0.041 to 0.180). In the paper, forecast uncertainty starts from the smoothed variance at the last batch and reaches the size of the biggest deploy effect (0.20 SD, the April deploy) after about 5 weeks at the point estimate, 6 with the pooled rate, and 1 to 21 across the interval. (The landing page’s explorer instead starts from the steady-state precision of a batch of 300 every two weeks, and the analysis seat uses that to recompute in Section 5.) At a fortnightly cadence the forecast SD is about half a large deploy contrast, so a batch of 300 could still detect such a change between two readings. The authors say the horizon is best read as the point at which a firm relying on its records would have been wrong by a deploy-sized amount, not as a smooth forecast.

3.4Finding 3: tickets agree in direction

Interview-related tickets fell about 10% a month relative to technical-issue tickets on the same dashboard (9.9%, t = -2.69, 5 May to 17 September), and 18% a month from June to August. Tickets per interview fell 7% a month over the full window. Interview tickets themselves did not fall in level (+3.7% a month); they failed to grow while the control category did. The authors read this as direction and order of magnitude, not an elasticity.

3.5What it costs, and what the authors say they cannot show

Rater pay for the stability pipeline was $7,806 a month (engineering time is not counted, and fewer filed tickets need not mean fewer failures). Counting only avoided escalations (428 fewer interview tickets a month in September than at the May ratio), the program covers its cost if it deserves credit for about 18% of the reduction at $100 per escalation, 36% at $50 and 73% at $25. The authors state the limit directly: “we do not identify the causal return to the evaluation program.” No part of the deployment went unevaluated, and nothing varied evaluation intensity. They also say what would identify it: a record, kept at the time, of which finding motivated each commit, and designed variation in evaluation intensity. The landing page boils the paper down to four rules, referred to here as Rules 1 to 4: overlap raters; report families, not an index; measure one’s own drift rate; and design for attribution before starting.

4The fine print

The analysis seat checked each of these against the paper’s text and tables. Two are credits to the paper, and they are given the same weight as the three problems.

4.1No interview was scored by both reviewers

The landing page says, in its heading and tile, that on the same interviews B scored 0.79 SD lower than A. The paper says something narrower, and so does the page’s own paragraph under that heading: two reviewers scored the same batches, and B took about half of each batch. Reading the paper’s counts, the analysis seat finds one score per recording. There are 2,611 recordings in the sample and 2,611 cases in Table 2, and in Table 3 the two reviewers’ counts add up to the batch size (27 July: 188 + 147 = 335, the largest batch; 24 August: 13 + 22 = 35, the partial batch). If recordings were scored twice, those sums would exceed the batch sizes. Status: inferred from the counts, because the paper never says in words that none was double-scored; its abstract and Section 3.2 use “overlapping” at batch level. Reviewer A scored alone until 21 May, then B took roughly half of each batch, and C took part of one batch. The 0.79 is therefore a comparison between two groups of different interviews drawn from the same fortnight. It measures reviewer severity only if interviews were split between reviewers without regard to difficulty, which the paper calls exchangeable assignment (its Appendix A). The paper states this as an assumption, does not say how the split was made, and tests it only indirectly. The B-minus-A gap is negative in all nine batches both scored, from -0.53 to -0.90, with a pooled value of -0.77; a heterogeneity test rejects an exactly constant gap (Q = 19.0, p = 0.015); and the reviewer-A-only trend (Section 4.4) needs no adjustment. That is decent circumstantial support. The paper’s own rule is “a fraction of every batch to two raters, at least one rater on every batch”, which can be read as double-scoring a sample or as sending part of each batch to a second rater, which is close to what micro1 did. The wording is ambiguous; double-scoring a sample is the recommended design, because it is the only design that separates severity from the difficulty of the interviews each rater happened to get. The landing page’s headline wording overstates what was done.

4.2The five-week point estimate rests entirely on one step

Table 6 gives three estimates of the index’s weekly drift. With a free jump at the three deploys: 0.086 [0.041, 0.180], likelihood-ratio p < 0.001. Also freeing the late-May transitions (9 May to 1 June, where the sup-F test places a jump of 0.34 SD that the change log does not explain): 0.000 [0.000, 0.151], p = 0.50. So the point estimate falls from 0.086 to zero when one step is set aside, and the five-week figure goes with it. The interval still runs to 0.151, close to the original 0.180, and p = 0.50 is a failure to reject a constant state, not evidence for one: the data cannot say drift is zero. For the critical-failure rate part of the drift survives freeing the step (0.666 percentage points a week, p = 0.16 against 0.762 before). At the index’s upper bound the horizon is 1.6 weeks (Section 5). The paper does disclose this: its abstract says the horizon “reflects a small number of step-like changes,” and Section 5.2 says the index moved in a few steps rather than continuously. The landing page body (its Finding 3) discloses the step too, and calls the number an illustration with a wide interval, not a measured expiry date. Only the headline tile, “~5 weeks”, omits it. The honest reading is that a firm should expect steps of roughly this size that its change log does not explain, and that these data cannot say how often.

4.3Whose agent, whose evaluation

The agent is micro1’s own product, run on micro1’s own interviews (paper, Section 3.1). The PDF carries no affiliation line, funding statement or competing-interest statement; the landing page lists the second author, Esposito, as micro1’s Chief Economist and the first, Zhang, as at Harvard University. micro1’s site menu also offers evaluations and RL environments for AI labs under the name Realm, and a visibility-and-improvement layer for agentic AI under the name Cortex. The paper is a working paper, marked preliminary, posted to arXiv on 4 October and not peer reviewed. None of that makes the numbers wrong: the tables are internally consistent and several checks reproduce. It is two facts a reader should weigh: the agent belongs to one author’s employer, and that employer sells evaluation products. To the authors’ credit the paper makes no return claim and says the breakeven is a condition, not a return.

4.4Credit: the improvement does not depend on the adjustment

Reviewer A scored every batch, so that reviewer’s scores alone need no reviewer correction. Table 4 puts the A-only trend at 0.124 SD a month against 0.102 with the adjustment, and 0.125 with deploy-regime fixed effects. The raw average flattened because of who joined, not because nothing improved. A common drift in both reviewers’ standards would survive this check; the authors hold that as an assumption. It is the best single piece of evidence in the paper and the most underplayed on the landing page.

4.5Credit: causality is disclaimed, honestly

The authors do not claim the evaluation program caused the improvement, say why (no unevaluated arm, no variation in intensity), and say what would be needed. Commit timing does not cluster after sign-off dates (63% of commit groups fall within seven days of one, against 67% under a permutation null), and no commit in the changelog cites an evaluation finding. The paper adds that its own simulation detects a commit-after-finding pattern in only 37% of replications, so the timing evidence neither supports nor rules out the channel. That is the right posture, and it is adopted for Wolfberg’s own work in Section 6.

4.6Smaller points worth knowing

• Severity is a vector. B is stricter than A on judgments of degree (latency, pacing, audio quality) and agrees with A on events (freezes, sync). On the critical-failure flag B is more lenient. One leniency constant would not fix that.
• The critical-failure trend is not clear. With reviewer effects the monthly trend is -0.784 percentage points (SE 0.451), wild-bootstrap p = 0.193. The story there is the hump, not a slope.
• The population moved. The share of software roles among interviews rose from 17% to 27% between March and May, and applicant sources shifted from June. The recordings carry no information about either, so some measured movement could be the interview mix rather than the agent.
• The landing page quotes a different figure. It says the paper reports 4.8 weeks; the paper text the analysis seat read says “about 5” (4.8 appears nowhere in it), and the explorer shows 5.2. Section 5 sorts these out.

5What the recomputation reproduced

The paper’s horizon comes from its equation (6) and the closing paragraph of Proposition 2. After a batch of n sessions the posterior variance is P₀; it then grows by the weekly drift variance q each week, and the horizon is the number of weeks until the forecast variance reaches the square of a change the firm cares about. The analysis seat used the landing-page explorer’s defaults, which the paper’s cadence discussion matches: a batch of 300, a two-week gap, a standardized index (variance 1), and a change of 0.20 SD, the April deploy’s effect. Status: measured (recomputed from the paper’s formula and inputs).

Three further checks agree to within rounding. The forecast standard deviation at the point estimate is 0.180, 0.249 and 0.303 at 4, 8 and 12 weeks; the paper gives 0.18, 0.25 and 0.30. The smallest change two consecutive batches can tell apart (95%) is 1.96 × √2 × 0.0529 = 0.147; the page says 0.14 (it uses 0.052, which gives 0.144; the recomputed value is 0.147). A batch’s own precision at the point estimate is 0.0529 where the page shows 0.052. Both are within rounding of 0.05, but the recomputed values do not round to the page’s, so “agree” holds only to two decimals.

Two things are worth saying about the shape. First, q enters squared, so an interval of 0.041 to 0.180 on the SD (a factor of 4.4) becomes a factor of about 20 on the horizon (22.6 against 1.14 weeks). The paper’s “1 to 21” is that arithmetic, not vagueness about the model. Second, the late-May row matters more than any other: it moves the point estimate from 5 weeks to “never”, and the interval for the horizon from 1.1 to 22.6 weeks to 1.6 weeks to never. The upper bound on drift falls from 0.180 to 0.151, so the shortest horizon gets longer (1.1 to 1.6 weeks), not shorter.

On the page’s 5.2 and 4.8: the page says its explorer starts from the steady-state precision of a batch where the paper starts from the smoothed estimate at the last batch, and the paper’s appendix confirms its text uses the smoothed variance. That explains why the two differ, but the analysis seat could not reproduce the explorer’s 5.2 from the inputs it states. The steady-state formula gives 5.03; the several variants it tried (using σ²/n alone, a continuous-time steady state, a one-week gap, 400 recordings) gave 4.74 to 5.41, none exactly 5.2. It also found no “4.8” in the paper text it read. All of these are “about five”, and none changes any conclusion.

Credit needed is 7,806 divided by 428 times the cost of an escalation, and it matches in every row. The net values differ from the paper’s by $11 to $109, growing with the cost per escalation. Working backward, every paper row implies 428.4 avoided tickets, not 428, which is consistent with an unrounded count. At $100 an escalation, $7,806 over 428 tickets is about $18 per avoided escalation, as the paper says.

6The three checks applied to Wolfberg’s practice

The paper’s three checks are rater comparability, evidence currency under drift, and corroboration with an outside outcome. Wolfberg runs an AI team and keeps two scoring instruments of its own: a full read that labels every item in the practice’s transcripts (the labeler is a language model, with a human and a second-stage detector as checks), and a public register of misses that records who caught each one. Both have the paper’s problem. This section uses only counts the team has produced, and each is a count from one practice, not an estimate of anything beyond it. The analysis seat, which did the recomputation above, also recounted the labels file and the register file where noted.

6.1Check one: the labeler is a rater

From 8 to 16 October 2026 one model read 36,657 transcript items for the practice. 3,163 were exact duplicates of earlier items and were not read again. The other 33,494 were each read twice, from requests that shared one instruction prefix (the same hash on all of them). For 2,004 of those 33,494, at least one pass came from a fallback utility model after the main model refused. That is a different 2,004 from the items flagged as pushback below; only 383 items are in both. Pass 1 found 42,288 records and pass 2 found 42,236, a difference of 52 (1,975 of pass 1’s records and 1,964 of pass 2’s came from the fallback). Only 16,732 matched on exact anchor text, which is a floor on agreement because the same statement anchored on different words counts twice, and the union is 67,792. In total, 5,126 of the 33,494 items (15.3%) had a disagreement between passes: 4,579 of the 31,490 items where both passes came from the main model (14.5%), and 547 of the 2,004 fallback items (27.3%). Of the 2,004 items flagged as pushback, 581 were flagged by one pass only. The totals agree; the exact-text matches, which are a floor, cover about 40% of either pass (16,732 of 42,288 and of 42,236: 39.6%).

The pattern is the paper’s. A rater can agree with itself and still not be the rater it was taken to be. On the fold question the model’s two passes agree with each other at 84.7% (kappa 0.77, 523 episodes), while the model against the human hand check is at kappa 0.157 on 23. That is a gap between self-agreement and agreement with the person, not a difference between question types, and the 94.5% figure is a different question (the kind of pushback). There is no test of whether the labeler is steadier on events than on judgments, so none is claimed. Status: conjecture, that the paper’s finding that severity attaches to judgments of degree will show up in the labeler. The samples behind the lower kappas are small (23 episodes, with a single episode called folded among them; 28 rows; 9 unflagged messages), so no tight interval should be put on them. They are enough to say the read has not earned a single number.

The 3 October window. On 3 October 2026, between 07:24 and 09:05 UTC, the read held 15 items from the human’s side of the conversation; four of them are mostly pasted or relayed text, and one is a task brief for another seat with no conversation context. The read flagged none of the 15. When the items were viewed on 16 October the operator’s reaction was recorded (the operator, 2026-10-16: “theres a lot of pushback there”). One was an exasperated message (the operator, 2026-10-03: “nooooo a post that goes with the who first pdf”); another was an impatient one (the operator, 2026-10-03: “whats taking so long”). The remaining items are not published. The pasted items make the window a weaker test than “zero of 15” sounds. Precision when the flag said yes was 17 of 19; recall was about 42%, and in this window it was zero of 15. Recall is uneven, which is the paper’s point about a judge version being a rater, turned inward: the number that looks like a property of the labeler is partly a property of the stretch of conversation it is applied to.

6.2Check two: evidence goes stale, and some of the drift is nobody’s change

The paper’s drift is movement its change log does not explain: a vendor model changing behind a fixed name, a configuration outside the repository, a different mix of interviews. Wolfberg has the same exposure, with one example that can be dated and a cause that was not isolated.

• The browsers. Edge updated on 10 October and Chrome on 13 October. Headless PDF rendering then hung, and a flag that disables the sandbox fixed it. The timing fits the browser updates, but two Windows updates (13 and 15 October) and a permissions change to the tools’ sandbox landed in the same days, and the seat that traced it did not separate the causes. It serves as an example of a pipeline moving under a pinned recipe, not as a measured cause. It is still something that changed in the environment that was not in the practice’s own change log.
• The labeler. It ran under a fixed model name across eight days. The request bytes were pinned for the main model, and the matching prefix hash shows the instructions did not change, but what the vendor serves behind the name cannot be pinned, and a fallback model read about 6% of the items (2,004 of 33,494), a second rater inside the run. Overlap was measured only at the start, with a third read of 250 items before launch, not at the end. The paper’s rule says to overlap raters at every batch.
• The working rule on source age since 8 to 10 October is that more recent information carries more weight, and that model-landscape sources two to three months old are stale. That is the qualitative version of the paper’s horizon: a reading expires, roughly when can be said, and no rate has been attached to it.

The paper’s method suggests the missing piece. A fixed anchor set can be re-read at the end of a long read, as well as at the start, and the change in agreement treated as an estimate of unlogged drift over that run. It has not been run.

6.3The public miss register is a reading that contains the rater

The register at wolfberg.ai/register lists every logged miss by Wolfberg’s team, with its type, a one-sentence account, and who caught it. The file behind it (built 15 October 2026) holds 745 entries: 740 misses dated 13 June to 8 October 2026, and five “gate” rows, which are a fixed example list and not real gate decisions, so they are not counted as evidence. Misses are read from the team’s session-health ledger, a plain file that is appended to by convention. The entries are sha256 hash-chained: the analysis seat recomputed the chain and found 0 broken links across the 745. What that protects: each entry’s hash covers its own content and the entry before it, so an edit inside the published file shows. What it does not protect: the chain is rebuilt whole every night, with newer rows inserted above older ones, so no hash carries from one night to the next. An edit to a row shows only if someone kept an earlier copy and compares row text, and a miss nobody logged is invisible (the production-signing details come from an internal audit dated 10 October). None of the 740 misses carries an Ed25519 signature; each is marked “hash-chained, not Ed25519”, and only the five example rows and the chain head are signed.

Who caught each miss: the operator 378, the agents unprompted 291, a peer instance 51, an external instrument 19, unattributed 1. The most common types are repeated miss (177), wrong diagnosis (161), would-ship error (132), recall assertion (73), unsearched claim (56) and self-corrected (46).

The register reads the way the paper reads reviewer B. The operator’s share of the catches fell from 71.1% in June (27 of 38) through 59.6% in July, 53.6% in August and 49.4% in September to 28.5% in early October (43 of the 151 with a catcher recorded). That is the observed fact. The tempting reading is that the team is catching more of its own mistakes and getting better. That reading is not taken here, for the paper’s reason: a count of logged catches mixes the team’s error rate with who was looking, how hard, and how catches were logged, and the data cannot say whether the catchers changed or the logging did. A fall in the operator’s share could come from the agents catching more, from the operator looking less, or from when and how rows were filed: a retrospective sweep on 3 and 4 October put 100 rows in on two days (73 self, 24 operator, 3 peer; from the internal audit dated 10 October). The register cannot say which, and the June-to-October counts must not be read as the team getting better or worse.

Three further points map directly to the paper’s rules.

1. Overlap. The register’s catcher field is set by the seat that writes the row, at write time, and nothing checks the claim against an independent signal (internal audit, 10 October). Of the operator’s 378 rows, 159 carry the operator’s own words, so the other 219 rest on the logging seat’s word alone (same audit). No second catcher re-scores the same sessions, so a catcher’s severity cannot be separated from the error rate. The field is the start of the roster the paper’s method needs, not the overlap.
2. Evidence currency. Every row is dated, so the age of any claim is visible, which is the paper’s shelf-life rule built into the format. The guarantee is weaker than it sounds: the ledger is appended to by convention and the chain is rebuilt nightly, as above, and a miss that nobody logged is invisible.
3. The paper’s Rule 4. The register records what happened and who caught it. It does not record which fix a miss led to. That is exactly the gap the paper says prevents it from identifying the value of its own evaluation program. The cheap thing to add is one field on each row: the fix, if any, that the miss led to.

The corroboration check has a thin counterpart here. The only rows written by a separate process are the 19 probe rows, all of one type (probe miss); the 159 rows credited to the operator with the operator’s own words are still logged by a seat. There is no outside outcome series like the paper’s tickets, so the register can say what was caught and by whom, and cannot say whether it mattered. Two further limits: the 15 October build contains no miss after 8 October, and by the 10 October audit, catches from the analysis seat’s reviews never reach the ledger, so the register undercounts outside catches. The register paper that explains the design, a separate forthcoming piece, is not yet published; this note cites only the live page.

6.4Next steps

• Re-read a fixed anchor set at the end of a long read, not only at the start, and report the change in agreement as an estimate of unlogged drift. Not yet run.
• Put a second catcher on a sample of sessions. Double-scoring a sample is the preferred design, because sending part of each batch to a second rater does not by itself separate catcher severity from the error rate.
• Add a fix field to each register row, so a miss can be traced to the change it caused, and the origin, escape and before-or-after-ship fields of Section 7.
• Vary the harness (the set of checks wrapped around the model) on purpose, by session, so that a counterweight’s effect (a mechanical check against the model’s pull toward agreeing with the user) can be identified rather than inferred from before-and-after counts (the paper’s Rule 4). Today every session gets the same one.

7In the loop, or watching the loop?

On reading the first draft of this note the operator asked the analysis seat whether agent reliability is really this: “How do you measure if the human is in-the-loop or watching-the-loop?” (the operator, 2026-10-16). The paper doesn’t ask it, but its three checks answer it, because the human who approves AI work is a rater too. Everything the paper says about rater severity, drift and overlap applies to the human gate. Some humans are strict and some lenient. Attention drifts: in the study Anthropic published on 7 August 2026 (70 days ago), as Wolfberg’s forthcoming note on the human gate (“Who Caught It”) reports it, 1,053 paid testers each met one planted dangerous command and caught it 13.6% of the time, falling from about 17% early in a session to about 5% after 50 or more prompts (Anthropic, 2026b; Wolfberg, forthcoming). And usually nobody re-scores what the human approved.

7.1What “in the loop” has to mean

The definition here is by effect. A human is in the loop when their decision changes outcomes before they take effect, and changes them because the item was wrong. A human who approves at the same rate whether the item is wrong or right is watching, however carefully. That gives three measures, and only the first is the test.

7.2The denominator problem

Discrimination needs the wrong items the human did not catch. A log of catches alone can’t supply them: a human who catches 40 of 40 logged misses might have missed 400 that nobody found. There are two routes. The first is planted items, as in Anthropic’s test: put known-wrong items in front of the human and count. That gives a rate directly, at the cost of an item that is not real and a human who may know it is a test. The second is escapes: wrong items that got past the gate and surfaced later, found by anyone. They’re real, and they’re all that comes for free, but on their own they are a floor, not a rate, because the ones that never surfaced are missing.

The earlier position, in the forthcoming “Who Caught It” note, was that a practice needs both real catches and planted items. That position is updated: escapes plus a random re-check are chosen over plants. Escapes are far more effective than planted items in this practice, because the agents are assumed to be wrong in some way in every response. The assumption is the design: if every AI response is presumed to contain a miss, the question is only who finds it and when.

7.3An escape count measures the finder too

This is the paper’s reviewer-B problem one step later. A count of escapes measures whoever found the escape: a human who reviews more, or looks harder, finds more, and the count says nothing about the gate they were checking. The fix is the paper’s: overlap the raters. In practice, a random re-check of signed-off work at a fixed rate, for example one item in ten, re-read by someone who did not sign it. That gives a rate with a denominator (so many wrong in so many re-checked). Escapes found any other way remain a floor. And an escape counts against a party only if the miss was actually in front of them: a defect in code the human never saw is not a lapse in their approval.

7.4The unit is the pair

The paper’s logic cuts both ways, and the human is not exempt: both sides are to be analyzed, the agents and the human. Each miss row should record the origin (the AI seat, the human or both), whether it was in front of the seat and in front of the human, who caught it, whether it was caught before or after it shipped, and how long it took to surface. The human’s own misses are decisions, rulings, instructions or work claims that steered the work wrong. They are not offhand recollections: grading the human’s memory of things months later would probably go badly. The human can miss and be wrong too.

The twelve rows are an overstated claim in a brief, a cause asserted but not isolated, an apples-to-oranges kappa comparison, a claim about the register made without reading its audit, a test gap, a forecast the analysis seat and the human both accepted, a file lock, an unwired logging path, and four labeling and recall errors in the practice’s own read that a review caught (the 79 of 125 rows, the fold label, the 3 October window and a wrong claim about who renders the PDFs). Four of the twelve were errors in this note’s own first draft, caught by a reviewer instance before it left the folder. Four were in front of the human and not caught by the human, so they are escapes past the analysis seat and the human: three surfaced before shipping, one after.

The human-origin row count is zero, and it should not be read as a score. Nobody looked for the human’s misses on purpose; the register was built to log the AI seat’s. It is a coverage gap. The one shared row is a forecast the analysis seat and the human both accepted.

7.5Code is where the human cannot see

The least visible misses are the ones buried in code. They are the easiest for a human to miss, because components do not get wired together until the agents and the human test after merging. The example is the analysis seat’s miss-logging path, which wrote its reports into a folder that no importer read, for nine days, while it appeared to work. Neither the seat nor the human saw it: it was in front of the seat, not in front of the human, and surfaced after it had shipped. On 16 October two mechanical checks were adopted. First, every pull request that adds output names its consumer and includes a test through it. Second, a nightly check flags output with no reader for more than three days. A new miss type for it was declined; whether it needs its own type is uncertain. The point for measurement is that a human gate can’t be the control for what a human can’t see, which is the series’ usual answer: put the check outside the people, and measure the people on what they can see.

7.6Honest status

None of this is running yet. The register has no re-check sample, no escape field and no before-or-after-ship field. What exists is the catcher field and a twelve-row seed from one session. The next steps are the fields, the one-in-ten re-check, and a per-catcher discrimination rate once there are wrong and correct items in front of each.

8What it means

8.1For anyone who scores an agent with people or model judges

Raters should be overlapped, and kept overlapped at every batch, not only at the start. A raw average should never be read across a changing roster. A new judge version should be treated as a new rater, including one that arrives under the same model name. Results should be reported by family or question, not by index, because severity differs by question (the paper’s B is harsher on degree than on events; whether Wolfberg’s labeler differs that way has not been tested). And it should be written down, at the time, which finding motivated which change. The human who approves the work is one of the raters: what should be measured is whether their decisions change outcomes (Section 7), not whether they approve.

8.2For anyone reading a paper like this one

A landing page and its paper should be read side by side. Here they differ in the places that matter: “the same interviews” against same batches, “about 5 weeks” against a horizon resting on one step, 4.8 against 5.2. Who benefits from the headline should be checked. A working paper by a vendor of evaluation about its own agent is evidence, and a check is cheap: the paper’s tables let the headline be recomputed in an afternoon, and the recomputation here reproduces.

8.3For Wolfberg’s practice

The register’s monthly counts do not support saying the team is getting better, and the post that goes with this note says so. The register’s job is to make each claim dated and attributable; it can’t yet do the paper’s job of separating the scorer from the scored. The first changes are cheap: a second catcher on a sample, a fix field, and the escape and before-or-after-ship fields. The third, varying the harness by session, is how it can be learned whether a counterweight does anything, and it has to be designed in before the sessions run, which is the paper’s last lesson.

9Limits

The analysis seat read the paper’s main text, Tables 1 to 9 and the proofs from a text extraction of the PDF; figures were read through their captions and printed values, and appendix tables from Table 10 onward were skimmed. The authors’ data and code were not available, so this reproduction is of their arithmetic from their reported inputs, not of their estimates. The three related papers were read in part: the abstracts and introductions in full, with methods and results skimmed (Table 1); none is a close replication of the paper. The Ford et al. study concerns release-to-release change, not silent change behind a fixed model name. The landing page’s explorer was read as text and was not run, so the 5.2-week figure is as quoted. The in-house numbers come from one practice’s full read and register. The labeler is a model, and a fallback model read about 6% of the items; the hand checks are small (19 flagged messages, 9 unflagged, 23 episodes, 28 rows); the agreement figures are from different questions; the recall range is rough; and the analysis seat recounted the 36,657 rows (models, disagreements) from the labels file but took the other read counts as supplied by the agents. The register’s catches are asserted by the seats that wrote them; its counts are of logged misses, not of opportunities; the monthly share of catches credited to the operator cannot be separated from changes in who logged and when; its five gate rows are an example list; and five figures marked as coming from the 10 October internal audit were not recomputed from the ledger. The hash-chain check confirms that the published file is internally consistent (the chain is rebuilt each night), not that the ledger beneath it is complete or unedited. Section 7 is a design and a twelve-row seed from one session, with no rates: there is no re-check sample, no escape field and no before-or-after-ship field in the register yet, the origin and in-front-of fields were assigned by the same seats the ledger measures, and the Anthropic study is cited through Wolfberg’s own forthcoming note, whose wording was checked but whose source was not opened. This note was drafted by Claude, a model made by Anthropic, under the operator’s direction, and one of the seats it describes in Sections 6 and 7 is a Claude seat. micro1 and the authors were not contacted for this note. The 20 October revision changed the voice, layout and figures of the note, not its evidence: ages are counted from 16 October, and neither the register file nor any outside source was re-checked.