Research notes16 October 2026

In the Loop or Watching the Loop? The Human Gate Is a Rater Too

Berg Atkinson · 16 October 2026 · The paper: In the Loop or Watching the Loop? The Human Gate Is a Rater Too

In the loop or watching the loop, summary card: The human gate is a rater too. 0.79 standard deviations: how much harsher a second reviewer scored the same AI agent, across 2,611 scored interviews (Zhang and Esposito, October 2026). The harsher reviewer made the agent look flat; adjusted for who scored what, its index rose 0.53. A person who signs off has a severity and a drift too, and nobody re-scores them. They are in the loop when their decision changes the outcome before it ships, because the item was wrong.

Is the human gate a check or a spectator? I scored my own.

Fifteen items, an hour and forty minutes, October 3rd, a few of them pasted text. Two I actually typed: "nooooo a post that goes with the who first pdf" and "whats taking so long." In mid-October my full read of my own transcripts flagged none of that window as pushback. I looked on October 16th and my honest reaction was "theres a lot of pushback there."

So either I'm politer than I think, or the thing doing the scoring isn't neutral. (It's the second one. I am not that polite.)

That's the point of a micro1 paper from October 2026. Their AI interviewer gets scored by human reviewers. A second reviewer joined in May and scored 0.79 standard deviations harsher than the first. The raw average went flat; the agent hadn't stalled. Adjust for who scored what and it rose 0.53. Their line: a judge version is a rater.

My Science seat read the paper, not just the landing page, and recomputed their math (it reproduces). The fine print is where I got interested: each interview was scored once, not by both reviewers. The "~5 weeks until an evaluation goes stale" tile rests entirely on ONE late-May step their change log doesn't explain. Free it and the index's drift estimate is zero (the interval still runs to 0.151, so nobody gets to say drift is zero). The agent is micro1's own, a coauthor is their chief economist, and micro1 sells evaluation products. None of that sinks it. Reviewer A alone shows the improvement, and they say flat out they can't show the evaluation caused it. Respect.

Then I turned it on me and my agents. One model's two passes over 33,494 items disagreed on 5,126. Me against the model on "did the seat fold": kappa 0.157, on 23 episodes, so small. My public miss register credits me with 378 of 740 misses: 71% in June, 28% now (and a bulk sweep of 100 rows in October inflates that drop). Tempting headline: my team is getting better. Nope. Who catches and how rows get logged both move that number.

Which is my real question: how do you tell if a human is in the loop or just watching it? I'm a rater too. In the loop means my decision changes outcomes before they ship, because the item was wrong. Watching means I approve at the same rate either way. So measure my catches on wrong items against my flags on right ones. The catch: you need the wrong ones I missed. I pick escapes plus a random re-check of signed-off work over planted items, because I assume my agents are wrong somewhere in every response. And my agents and I log both sides. I miss things too, mostly buried in code.

None of it is running yet...when I say my team got better, ask who was scoring :)

Sources

  1. Zhang and Esposito: Reliability of AI Agents (arXiv:2610.07003, 4 Oct 2026) arxiv.org
  2. micro1: Can a company tell whether its AI agent is getting better? (14 Oct 2026) micro1.ai
  3. The miss register
  4. Never let your AI grade its own homework
  5. Katoch et al.: Accounting for bias enables sustainable LLM evaluation (arXiv:2609.31184, 25 Sep 2026) arxiv.org
  6. Sachdeva and Boudol: Rating the raters (arXiv:2608.27463) arxiv.org
Read this before you believe any of it. Keystone runs in production in its own AWS account. The rest of what you see here is a running prototype, not an accredited or certified system. The register, the war room, and the brain are receipts, not revenue. Every claim on this site links to something you can check. If one doesn't, tell us, and we'll fix the page.