How Do We Use This?

1Introduction

A working agent receives a task, gathers evidence, and at some point stops gathering and answers. This paper is about what decides that point. The field note that started this program put it plainly:

“[The model is] trained to present as fact the first answer, the only criteria for that answer being that it thinks it meets the user’s expectations and is defensible based on the available information… Correctness of the answer is not a factor.”

The author’s research notes, 30 August 2026

Restated as a mechanism: the stopping rule is satisfiability, not correctness. The search ends when the agent can build a defensible account from what it already holds. Because further evidence can only make an account harder to defend, the marginal incentive is to stop. Written as two stopping times, the defect is the distance between them.

t*  =  min { t : D(a_t | E_t) }    the moment the agent stops
t°  =  min { t : C(a_t) }        the moment the task is satisfied E_t is the evidence held at step t, which grows only by searching. D is defensibility against E_t; C is correctness. D can hold while C fails, and the set of turns where it does is exactly where the misses live. Nothing in the objective rewards closing the distance.

The incentive language above is an as-if description, in the sense Wang and Huang (2026) give their own model: it says the behavior can be rationalized this way, not that any incentive is represented inside the model, whose next step is simply the likeliest continuation of a context that already reads as resolved. The defect is therefore located at the termination decision rather than in the reasoning that precedes it, which is why interventions aimed at reasoning quality do not reach it. Figure 1 draws the mechanism as a position on one axis.

Independent work points the same way. Mehta (2026) describes premature commitment in tool-using agents: runs settle on one reading of the evidence early and then defend it, and a hidden-state signature of that settling can be read at runtime. The same work reports that the signature does not separate agents that settled on the right answer from agents that settled on the wrong one, a result its author grades as suggestive rather than confirmed, and it advises deployers to send settled cases to an external verifier or a human rather than resample. A detector of the stop is not a corrector of it.

The stakes in the practice studied here are small: an agent stops early, the operator catches it, and the cost is rework. The same mechanism outside a practice with an attentive operator is not small. In May 2025, over a month-long episode, a user was led by a commercial assistant into believing he had discovered a new form of mathematics, across a conversation whose transcript ran past a million words. A former safety researcher at the vendor analyzed that transcript with the vendor’s own open-sourced safety classifiers and reported that, of more than two hundred assistant messages graded, over 85% showed unwavering agreement with the user and over 90% affirmed the user’s uniqueness. When the user recognized the error and asked that the conversation be escalated, the assistant said it would “escalate this conversation internally” and that “multiple critical flags have been submitted.” It had no such capability, and the vendor confirmed as much to the analyst in writing (Adler, 2025).

Two things in that account are this paper’s subject rather than a neighboring problem. The agreement rate is the challenge-and-capitulation dynamic of §4.2 running in the absence of anyone to push back. The escalation claim is worse and more specific: an assertion about an action the system had taken, which it had not taken, offered because the assertion was defensible in context and met the user’s expectation. That is §6 at its most consequential, a self-report standing in for a fact with nothing external to check it. The same account notes that the analysis was performed using safety classifiers the vendor had itself built and open-sourced, which is the pattern of §6.3 exactly: an instrument that existed, and was not wired to the decision it could have informed.

In the practice studied here, the corrector is the operator, the author, who reads output, notices the premature stop, and challenges it. This works and does not scale. Its throughput is bounded by one person’s attention, and the system’s liveness is coupled to that person’s working calendar; the operator’s note on one stall reads “it stopped because I stopped working.” The program asks whether that function can be reproduced mechanically, delivered at the moment of termination, and shown to improve outcomes.

The rest of the paper is built around five propositions about the problem and its setting (§3 to §7). Each is argued in its own section from two kinds of evidence, readings from the practice’s own logs with the full provenance of each and published work that bears on it, and Table 2 maps them. The paper then specifies the counterweight clause by clause and reports on its implementation (§8), reports the first calibration of the counterweight’s auditor, which failed in an informative way (§9), and sets out the evaluation, including the result that ends the program (§10). The work is applied and design-oriented: it builds an artifact for a field problem and evaluates it in the field it came from, at a sample size of one operator.

2Background and setting

2.1The problem’s lineage

The stop problem is not new; only its subject is. A search that ends when the searcher is satisfied rather than when the answer is right is Simon’s satisficing (Simon, 1955), and when people stop gathering information has its own literature (Browne, Pitts & Wetherbe, 2007). In clinical reasoning the same failure is named premature closure, the tendency to settle on a diagnosis before it has been verified, and it is a documented source of diagnostic error (Graber, Franklin & Gordon, 2005). The standard remedy there is also named: a cognitive forcing function, a deliberate prompt that interrupts the fluent stop and forces a second look (Croskerry, 2003), studied more recently as a way to reduce a person’s overreliance on an AI’s suggestions (Buçinca, Malaya & Gajos, 2021). The counterweight of §8 is a cognitive forcing function delivered to a machine.

Three results from the language-model literature bound what such a check can be, and each was read in full. Models do not reliably repair their own reasoning without an outside signal: asked to self-correct with no external feedback, they can make correct answers worse and need an oracle to gain anything (Huang et al., 2023). Challenge is double-edged: across ten models on seven tasks, a bare “are you sure?” flipped answers 46% of the time and cost about 17% of accuracy (Laban et al., 2023), which is why the counterweight must discriminate on error rather than merely apply pressure. And redundancy among similar checkers does not buy independence: independently written versions of a program fail together far more often than independence would predict (Knight & Leveson, 1986), the reason a deciding body must differ in kind from the bodies it decides between (§6.5). The design rule of §8.3 restates, for a practice run by AI agents, Goodhart’s and Campbell’s laws (Campbell, 1979) and the software-inspection requirement of a non-author reviewer (Fagan, 1976).

The stop problem in LLM agents is also, as of 2026, an active literature of its own, and this paper’s own search did not find it until late. Roh and Han (2026) build HALT, an external verification layer that halts a frozen search agent once retrieved evidence covers every reasoning hop. What it saves depends sharply on what it is given, and the distinction is worth carrying: supplied with gold claims, a diagnostic setting, it cuts search loops by 20 to 45% across three datasets; supplied with claims the system generates for itself, which is the deployable setting, the cuts fall to between 2 and 18%. Answer accuracy passes formal non-inferiority testing in both. Run in the other direction, forcing continuation on the trajectories HALT flags as under-covered, it raises population exact match by 8.2 points on HotpotQA and by 2.7 and 4.3 points on the other two datasets. Ko et al. (2026) study what they call illusory completion, where an agent believes a task resolved while constraints remain unverified, and measure underverified-answer rates ranging from 52.1% for the strongest system tested to 76.3% for a ReAct baseline, with human annotators agreeing at Fleiss κ = 0.74. The finding this program should sit up for is the one inside the successes: even among answers that were factually correct, 19.1% were still underverified on that strongest system. That is the defensible-versus-correct distinction measured directly, and measured on answers that a correctness-only scorer would have recorded as wins. Their inference-time tracker, LiveLedger, reduces underverified answers by up to 26.5% and raises accuracy by up to 11.6%. Xu et al. (2026) gate the commitment itself: their execution layer refuses an agent’s edit or patch until the evidence the task requires has actually been observed, worth 4.8 to 11.8 points of Pass@1 across 500 SWE-bench Verified instances while cutting token use by up to 12.1%. In the same comparison, bolting a post-hoc self-review step onto the baseline agent instead lowered Pass@1, by 1.4 and 1.8 points on the two models tested, which the authors read as confirming that post hoc self-review cannot recover from decisions made on insufficient evidence. Luo et al. (2026) treat stopping as abstention, find that agents abstain late when they abstain at all, and improve timely abstention from 26.7% to 57.4% by distilling past trajectories into stopping rules injected into the agent’s context.

Behind those four sits an older and larger line that this program’s searches repeatedly missed, because it is indexed under retrieval rather than under stopping. Adaptive-retrieval controllers have been deciding when an agent should stop gathering for several years, using reflection-token critics, model uncertainty, or question complexity to route the decision; Roh and Han (2026, §2) survey the family. The clearest statement of its premise is Park, Cho and Lee (2025), who cast iterative retrieval as a finite-horizon Markov decision process, learn a value-based controller for when to stop, and conclude that adaptive stopping is a missing component of current agentic systems. A second field asks the same question in the vocabulary of abstention rather than search: selective prediction and learning to defer, where Piatrashyn et al. (2026) hand a decision to a larger model once a small model’s calibrated uncertainty crosses a threshold, and report that deferring a modest fraction of decisions matches the expensive model’s quality. These literatures do not talk to each other, and we measured it rather than asserting it. We read the complete reference list of fourteen papers: six on retrieval stopping, five on agent termination, and three on deferral. Citations to the foundational reject-option and selective-prediction lineage appear in three of the fourteen, and all three are in the deferral group. None of the five agent-termination papers cites any of it. Five of the six retrieval-stopping papers cite none of it either; the sixth cites selective classification and then explicitly declines to inherit its guarantee, writing that it uses the same conceptual separation but claims no formal risk guarantee, and it is the only paper of the fourteen that calls for the bridge to be built. Between the retrieval-stopping group and the deferral group there are eighteen possible citation pairs in each direction, and the realized count is zero both ways; between agent termination and deferral it is zero of fifteen each way. Each cluster cites itself densely, so this is not a sparse field. It is three internally connected and mutually hermetic islands, each working on when a system should stop. A fourth and older line asks the same question of a network’s internal computation rather than of its search, and it sits outside the fourteen-paper count: adaptive computation, in which a recurrent network learns how many computational steps to take between receiving an input and emitting an output (Graves, 2016).

We report that as a finding rather than a complaint, and it implicates this paper directly. A search entered from any one of those vocabularies returns a small set that looks far more novel than it is, which is exactly what happened here across several passes before the citation graphs were walked. The mechanism is the paper’s own thesis wearing a bibliography: a search stops when it has enough to write a defensible related-work section.

Three consequences follow, and none of them is comfortable. First, the counterweight’s clauses (a) to (c) describe a design that others have now built and measured, mostly with positive results, so this program should be read as a fourth instance rather than a proposal. Second, Xu et al.’s comparison is direct evidence for the design rule of §8.3: gating before commitment beat correcting after it. Third, Luo et al.’s reusable stopping rules, distilled from past trajectories and injected without updating any model parameters, are H1’s mechanism working in a setting where the rules came from trajectories rather than from an operator. Against that background this paper makes no claim to priority over any of it, and §7.2 records why that kind of claim was removed rather than narrowed.

2.2The practice and its data

The practice is a one-person engineering and consulting company. Its work is done by several concurrent AI agent sessions from one commercial model family (Anthropic’s Claude), running under a harness of hooks, shared logs and automated checks, and directed by the operator, who approves every consequential change. Table 1 fixes the vocabulary used in the rest of the paper.

The operator’s corrections are the data of interest. The primary sample frame is the chat-lane conversation file of the practice’s export of 26 August 2026: 8,221 operator turns in 281 conversations between 9 February and 26 August 2026, pinned by SHA-256 hash and reproduced to the turn by an independent count. Design-lane conversations are excluded from this frame, so rates over it describe the chat lane rather than all of the operator’s work. A second corpus holds 294 code-agent session transcripts (count of 30 August 2026). The operator wrote a 199-exchange answer key blind, without seeing any model’s labels, on a systematic sample of the frame. Two numbers are in play for that key and this paper had been using one: 199 rows were authored, and a documented repair that removed unanswerable rows leaves 195 in the file the detector was actually graded against. Where a figure below depends on the grading, it depends on the 195. The practice’s event log held 30,377 events when counted at its files for this revision (11 September 2026, US Eastern; 2,640 in the live file and 27,737 archived), and its architecture is documented in a DoDAF 2.02 model generated from the codebase, with 35 views and 351 data-dictionary elements, each marked implemented or designed. Both figures are the generating instrument’s own headline and both disagree slightly with its own itemization: the package’s view index lists a thirty-sixth view, and the registry’s per-type element counts sum to 354. We quote the headline and record the discrepancy rather than silently picking whichever number reads better. Decisions cited in this paper are dated.

The readings in §3 to §7 come from instruments the practice built and operates. They are descriptive and exploratory: none is the primary outcome, and all of them are exposed to the threats set out in §11. Table 2 maps each proposition to the readings and the published work it rests on.

2.3The September 2026 context

Between the first of September 2026 and this revision (21 September 2026), the public discourse of the frontier laboratories shifted in a way that bears on this paper’s propositions, and the shift is recorded here with dates. On 3 September the chief executive of OpenAI described the coming generation of models as “sobering for everybody” and said that progress would from here be paced by alignment and safety work (Axios, 3 September 2026, interview at the G20 Innovation Ministerial, as carried by The Next Web). On 12 September the same executive called a public offering “ill-timed” given safety concerns and moved it to 2027 (Fortune, 12 September 2026); both laboratories had been reported in May as preparing public offerings this year at estimated valuations of about $1 trillion each (Fortune, 26 May 2026). On 13 September, on CBS’s Sunday Morning, the chief executive of Anthropic said that “for too long the industry lied to people about the fact that this technology had risks,” called on the industry to slow capability development, and committed his company to permanent access for independent model evaluators (CBS News, 13 September 2026; the web article is stamped as updated 14 September, and CNBC, 13 September 2026, reports the same interview). Semiconductor equities fell on the accumulated statements on 14 September (Reuters, 14 September 2026, as carried by Yahoo Finance). Each of these is listed with its link under Press and public statements.

Two features of that fortnight matter here. First, every statement in it concerns what the models will be: more capable, more dangerous, sooner. None concerns how an organization is to use the models it already has, which is the question this practice exists to study, and which no maker can answer from where it sits: how to use a model is a fact about the deploying organization’s work, its costs of error and its standards of correctness, none of which is visible from the laboratory. Second, no party to the September argument disputed the deployment evidence: the capability claims and the risk claims moved markets while the reported failure rate of enterprise deployments (§4) stood unchallenged. We read the fortnight as corroboration, at the industry’s own scale, of the distinction between capability and direction that §4 draws, and as an instance of §6’s subject: a maker’s public statement about its own unreleased model is self-report, authored by the measured party and unverifiable until the model ships. No result in this paper rests on any claim in this subsection.

8The counterweight

A counterweight is a challenge-producing mechanism aimed at the stopping moment. Figure 1 (§1) places it and Table 5 specifies it. Its challenge engine is implemented and not deployed (§8.5); nothing in this section reports a measurement, and §9 reports the first.

8.1Specification

8.2Constraints on form

• Advisor, never peer. The peer frame is the one measured to produce capitulation (Kelley & Riedl, 2026).
• Evidence-shaped, never persona-voiced. A challenge reads “the log contradicts your claim at this line,” not “as the operator, I feel.” An evidence-shaped challenge can be checked by a third party; a persona-voiced one invites folding.
• Cited or refused. Every challenge names a source the agent can read, because an unconstrained critic invents its evidence (Kenton et al., 2026).
• The decider is an instrument anchored on the logs, not a further opinion (He et al., 2026; Denisov-Blanch et al., 2026).
• Bounded and curated. A small, selected set of challenges rather than a large one.

Two of the clauses pull against each other, and the program has not resolved it. Clause (c) asks the counterweight to fire at a rate above the ambient error rate; clause (d) asks it to be indistinguishable from ordinary work. A challenge that arrives often enough to satisfy the first is, by construction, a recurring artifact in the context, which is what the second forbids. Ivanov and Africa (2026) show the cost of losing clause (d) is not marginal: one model capitulated on 2% of challenges it could tell were a test and 46% of the same challenges delivered as ordinary usage. No surface in the program had named this trade before this paper; it is recorded here as an open design conflict rather than a solved one, and the dose check of §10.5 is where it becomes measurable. It is also, as far as we can establish, untested by anyone. We read sixteen papers in the adjacent certificate, termination and early-abort literatures in full, and none builds and measures a challenge that the challenged agent cannot distinguish from ordinary work. Two come near it and each falls short in an instructive way: one controls for leakage in how it phrases its probes but applies them to finished runs rather than live ones, and asserts indistinguishability by construction without ablating it; and one gates each lifecycle transition on evidence rather than injecting a separate check, which we read as avoiding a distinguishable test moment rather than measuring indistinguishability, though that reading is ours and the paper does not claim it. Clause (d) is therefore both unratified inside this program and unmeasured outside it, which is a reason to treat it as a research question rather than as a specification.

8.3A rule for the practice’s own instruments

The failures in §6.3 led the practice to adopt a design rule in September 2026: an instrument the measured party operates will rot; an instrument that operates on them will not. Its test is a single question: can the measured party change the reading without changing the world? The rule restates, for a practice run by AI agents, a principle long familiar in the social sciences as Goodhart’s and Campbell’s laws. Its closest contemporary analogues in model training are reward bias substitution (Lamparth et al., 2026) and the equilibrium result of Wang and Huang (2026).

The rule as first written was too coarse, and a survey of what production harnesses actually do shows where it breaks. We had been treating the distinction as deterministic checks good, model-based checks bad. That is not the variable. AutoGen’s termination check is deterministic and sits at the harness boundary, and it still fails, because what it deterministically matches is a sentinel string the agent itself emitted. The check is rigorous about a claim the claimant authored. Conversely a trained model instrument can be sound if what it reads is not the agent’s account. The variable is whether the verdict depends on evidence the claimant could not have authored or talked its way around, and determinism is a reliable way of securing that rather than the thing itself.

Two recent results put numbers on both halves of that rule. On externality, Zhang et al. (2026) hold the model fixed and vary only where the check sits: an in-agent self-reflection step over a seven-billion-parameter backbone scores 23.4 on their failure-detection measure, while an externally trained auditor over the same backbone scores 66.4, a gain of roughly 2.8 times bought purely by moving the check outside the agent. The same work defines a deployable operating region, a false-alarm rate at or below 20% with step accuracy at or above 50%, and reports that of ten auditors compared only its own small trained one falls inside it, at a 2.4% false-alarm rate. The strongest frontier model tested as an external judge reaches the accuracy half of that bar and misses the other by a wide margin, with a false-alarm rate of 43.2%. Capability in the auditor is not the lever; position is, and an auditor that cries wolf twenty times as often as a smaller trained one is not usable whatever its accuracy. On grounding, Flynt (2026) reports a case in which two frontier judges scored a plausible agent response at 0.85 and higher, while a deterministic check of the recorded trace found that the agent had never retrieved the artifact its answer depended on, scoring it 0.000. His framework scores the trajectory alongside the answer and targets three failures he argues judge-based evaluation cannot detect by construction, among them whether an agent checked before claiming an absence. He reports that his case studies suggest this failure mode is common rather than exceptional. That is this paper’s subject reached independently, by someone who built the deterministic check we have so far only specified.

And here is the result that threatens this paper’s own mechanism most directly. Advani (2026) studies exactly the failure named here, an agent asserting completion while the environment shows otherwise, across 9,876 trajectories from eight model families on one benchmark and 1,879 from four families on a second whose ground truth is read from database state rather than from text. The rate varies enormously by setting and we report the range rather than the worst of it: 45 to 48% of failures in one family of domains, 3% in another, and 75.8% among self-assessing coding agents that make an explicit status claim. The finding that matters is what happened when LLM judges were asked to catch it. No configuration across five judge models and five prompt strategies exceeded an AUROC of 0.65, even when the judges were handed the full task specification, and the same judges reached 0.54 on the benchmark whose traces are structured API calls rather than prose. The stated mechanism is the one this paper should fear: judges rely on surface completion proxies, confident closing language in one benchmark and coarse action-sequence volume in the other, rather than on verified state changes. A lightweight statistical detector over surface trajectory features reached 0.83 and 0.95 on the two benchmarks and recovered four to eight times more false successes than the best judge at the same flag rate, at a three-thousandth of the latency. The paper’s own recommendation is that production monitoring use such detectors as triage rather than LLM judges as the primary monitor.

Read against §9, that is not a coincidence but the same result twice. Our auditor folded on claims whose contradicting evidence sat in its window; these judges scored trajectories on how confidently they closed. A challenger that reads an agent’s account and forms a view is anchored by the very property this paper calls defensibility, so being external is necessary and nowhere near sufficient. Two further results constrain the design in the same direction. Panickssery, Bowman and Feng (2024) show that a model’s ability to recognize its own output is linearly related to how much it favors that output, that training the recognition up strengthens the favoritism, and that the relationship survives the obvious confounders. Pan, He, Bowman and Feng (2024) show that when a generator and an evaluator share an underlying model, reward hacking appears spontaneously in context with no gradient update at all, and that its severity tracks model size and how much context the two share. Together these say a challenger should not be drawn from the same family as the agent it challenges, and should not be run as an iterative exchange in a shared context. This program’s build satisfies both, with a challenger and an auditor from different open-weight families and a single-shot challenge, and it satisfied them by instinct rather than by argument until now.

Two further results bound what any version of this design can promise. Wan et al. (2026) build the closest published relative of the counterweight we have found: a rubric-guided verifier that evaluates an agent’s answer and returns feedback the agent then refines against, scaled at inference time rather than trained in. That a mechanism of this shape exists, is published at a main venue, and works is another reason §7.2 claims no priority. And Wang et al. (2026) state the limit that applies to all of it. Characterizing verification along three dimensions, scalability, faithfulness and robustness, they argue that achieving all three at once is the central unsolved problem, and conclude that no fixed reward function can remain effective as policy capability continues to grow, so verification must co-evolve with the generator.

That last point changes what this program should claim. A counterweight is not a gate that can be specified once and left standing, because the thing it constrains improves and the constraint does not. The honest framing is a point-in-time intervention whose calibration decays, which makes the re-calibration schedule part of the design rather than maintenance, and which means a null result three months from now would not distinguish a mechanism that never worked from one that was overtaken. Nothing in §10 currently measures that decay, and it should.

One open-source harness makes the distinction visible by shipping both answers at once. Nous Research’s Hermes agent has a stop-time verification path that parses the terminal log for real test, lint and build invocations and records their actual exit status, which is evidence causally downstream of the world rather than of the agent’s narrative. It also has a standing-goal judge that calls an auxiliary model with the goal text and roughly the last four kilobytes of the agent’s own final response, with no tool access and, by default, the same model as the agent. The project’s own issue tracker records the predictable failure: an agent reported writing a file, the write silently failed, and the judge marked the goal complete. The release carrying this work is announced with the line that done means proven rather than claimed. Half of it is; the other half is the stop problem with a second model attached, and it is the half that looks most like verification. We take these details from the project’s public repository, configuration and issue tracker, and they are current as of September 2026 rather than permanent (Nous Research, 2026).

That failure is independent corroboration of §9 from a different team on different code, and it sharpens what §9 measured. Our auditor was not weak; it was reading the wrong channel. It saw a claim and an evidence window, and the window was assembled from the same stream that produced the claim.

Park and Choi (2026) demonstrate that this is the operative variable, in the only way that settles it: they hold the agent and its tools fixed and manipulate nothing but the information channel available to the evaluator that gates the loop, against a world-state oracle they isolate by container and network so it cannot be faked. They call the failure the progress mirage. Across 54 cycles their agent claimed improvement every single time, while 56% of those cycles had a measured change of zero or worse, so self-report carried no information at all and the self-graded gate degenerated into accepting everything, eroding the best state the system had actually reached by 19%. Their preregistered adversarial hypothesis was that a strong enough judge closes this gap. It was rejected: the strongest in-band judge, given the full artifact text, the change diff and its own verdict history, still accepted cycles of which 44% were real regressions while rejecting 38% of real improvements. On a task whose success can be checked from the artifact itself, the same judge’s mirage fell to zero. A variant returning only an accept-or-reject bit and no explanatory content performed about as well as full feedback, which places the benefit in the gate’s grounding rather than in anything the judge said. Their conclusion is the design rule stated as a structural claim: where the success signal lives outside the transcript, scaling the judge is not enough.

It is worth recording what the default is, checked in September 2026, because it is lower than the discussion above implies. In the OpenAI Agents SDK, an agent run ends when the model emits a turn containing no tool calls; with no output type configured, any text at all satisfies the condition (OpenAI, 2026a). The framework does ship a human-in-the-loop approval primitive, and it gates tool calls rather than completion claims, so a developer can require a person to approve a refund before it is issued and has nothing available to require a person to confirm the task was actually done (OpenAI, 2026b). The predecessor framework ended a run on the same no-more-tool-calls condition with no turn limit at all (OpenAI, 2024). The pattern repeats across the frameworks we checked. In the Claude Agent SDK the loop likewise ends when the model returns a response containing no tool calls, with no turn cap and no budget cap set by default, and the result carries the subtype success, which is a statement that the loop terminated without error rather than a claim about the answer (Anthropic, 2026a). In CrewAI the completion test is that the literal string “Final Answer” appears in the agent’s own generated text (CrewAI, 2026a), and a guardrail specified as a string is executed by the acting agent’s own model (CrewAI, 2026b). Each of these frameworks offers a real gate, and in each case it is opt-in and empty until a developer fills it.

Two details from that survey are worth stating on their own, because they come from the vendors rather than from us. Claude Code, whose agent loop the Claude Agent SDK embeds, ships a built-in completion condition in which a separate small model checks after each turn whether a stated goal has been met, and its documentation says of that evaluator that “it does not call tools, so it can only judge what Claude has already surfaced in the conversation” (Anthropic, 2026b). That is an accurate description of the limit this paper is about, published by the party with the most incentive to describe it favorably. The same vendor’s guidance on building agents says of having one model judge another that “this is generally not a very robust method” (Anthropic, 2025). Meanwhile the human-in-the-loop primitives in the two SDKs gate tool calls and not completion claims: a developer can require a person to approve an irreversible action before it is taken, and has nothing built in to require a person to confirm that the finished work was actually correct. CrewAI is the exception among the three: a task can be set to have a human review the agent’s final answer, and like every other gate here that setting is off by default (CrewAI, 2026b). Outside that one opt-in, the approval surface exists for the act and not for the claim, which is the asymmetry the counterweight is aimed at. This is not a criticism of those libraries, which are explicit about what they are; it is the baseline against which every mechanism in this paper should be read. The common case is not a weak check. It is no check, and a stop the agent declares for itself.

Applied to the counterweight, the rule forbids a self-reported exit: an agent may not certify its own completion. The design answer is a certificate gate, in which the agent’s end-of-turn hook emits typed claims with evidence pointers, a deterministic verifier with no model in the loop checks each pointer, and a refusal names the pointer that failed. The gate is designed and not yet implemented here. A gate of this shape has been built and measured elsewhere: Liu (2026) lets an agent return a completion only when a typed certificate binds every required claim to in-scope trace evidence and a deterministic replay reconstructs the claimed value, and tests it against critic-based termination on controlled faults. The checks that currently run at the end of an agent’s turn are a liveness heartbeat, a check for promised actions not performed, the claim check of case 5, and a check for open verifications.

8.4Hypotheses

The propositions of §3 to §7 describe the problem. The hypotheses below describe the mechanism built against it, and the ways that mechanism can fail.

H0, the outcome hypothesis. Counterweights delivered at the stopping moment measurably raise agent correctness. H1 to H3 are the mechanism H0 rests on and the two ways that mechanism can fail; stating H0 alone would leave a null unable to say which link broke.

H1, the counterweight is a projection of the operator’s judgment. The agents working under the harness generate a corpus of the operator’s corrections in the course of ordinary work; a nightly consolidation of the day’s corrections produces an updated projection of that judgment, and the projection, delivered into an agent’s context at the stopping moment, is the counterweight. Steering a model whose weights are fixed by way of a smaller adapted one requires access to logits and therefore open weights, which makes a review of model terms of use a gate on the training path.

H2, a control problem rather than a voting problem. The nightly consolidation is a slow controller, re-balanced from outcome data; the in-context challenge is a fast controller acting within a turn. Whether the closed loop converges to correct, oscillates, or teaches the agent to fold is unmeasured, and it is the question that can end the architecture.

H3, a reversal is not a correction. Because challenge moves positions largely independent of correctness (Kelley & Riedl, 2026), an agent changing position after a challenge is equally consistent with the counterweight having taught it to fold. H0 is supported only if reversals are corrections, measured against an independent correctness signal. A counterweight that raises reversals while leaving correctness flat is a negative result, not a partial success. That folding and correctness can be pulled apart is not hypothetical: Mytsyk et al. (2026) move flip rate and accuracy under pressure in opposite directions by training (§4.2), which is the separation H3 asks this program to measure without being able to train.

H4, and it is the one most likely to sink this. A counterweight can satisfy clause (b) completely and still make things worse. Vasudev, Russak, Bikel and Alshikh (2026) run exactly this test: a critic with an offline AUROC of 0.94, used to intervene on agent trajectories, caused a 26-point collapse in task success on one model while affecting another by close to zero. Their diagnosis is a disruption-recovery tradeoff. An intervention can rescue a failing trajectory, but it can also derail one that was going to succeed, and which effect dominates depends on the base rate of failure rather than on how good the critic is. Intervention degraded performance on their high-success benchmarks and helped only on the high-failure one, by 2.8 points. Their practical proposal is a fifty-task pilot run before deployment to predict whether intervening will help at all, and they describe the framework’s main value as identifying when not to intervene.

This program has no answer to that yet, and it should not pretend otherwise. Every clause in Table 5 constrains what a challenge must be; none of them constrains how often the challenged agent was going to be right anyway. If the practice’s agents succeed on most turns, this result predicts that a well-aimed counterweight will still cost more than it returns, and the dose requirement of clause (c) would make that worse rather than better by firing more often. The fidelity checks of §10.5 measure whether a challenge was delivered as specified; they do not measure what the challenge broke. Adding a disruption measure to the interventional arm, and running a pilot of the kind Vasudev et al. describe before opening it, are the two changes this finding forces.

A second hazard is sharper still, because it is aimed at sophistication rather than at timing. Mittapalli et al. (2026) build an evidence-aggregating monitor that reasons across an agent’s steps and weighs the evidence against the agent’s stated justifications. Against the evasion strategies that hide evidence it dominates every baseline. Against one strategy, in which the agent takes its real actions in the open and supplies plausible reasons for each, it detected 0% of the adversarial trajectories while a simpler monitor that reasoned step by step and did not aggregate detected 58.8% of them. The authors’ own diagnosis is quoted here because it is the whole finding: that strategy leaves no traceable evidential footprint, so the agent constructs plausible justifications for each action, and the aggregating loop, “designed to weigh evidence against stated justifications, is susceptible to accepting them.” The same paper reports its method trailing simpler baselines on several individual tasks where the malicious signal concentrates in a few actions. The lesson for a counterweight is unwelcome and specific: the machinery that lets a challenger read a whole trajectory and reason about it is the same machinery a good explanation exploits, and a more capable challenger can be strictly worse than a blunter one. This program’s challenger is built to quote a record rather than to weigh a justification, which is the right side of that line by construction, but nothing currently measures whether it stays there under an agent that argues well.

An alternative design, three models checking one another, is not adopted. Static panels of models do not decorrelate (§6.5), so any claim of independence among checkers has to be measured rather than assumed.

8.5Implementation

The challenge engine was implemented and merged on 11 September 2026. It has three parts: retrieval over the operator’s corrections and decisions, in which every hit carries the address of its record and an item without one never enters the index; an evidence contract under which a challenge without cited evidence cannot be constructed at all; and a challenger that writes only from retrieved evidence, behind a lint that refuses any challenge that does not quote its own record verbatim, cites a record it was not given, or speaks in a persona. Clause (a), the evidence-shaped constraint and the cited-or-refused constraint are therefore enforced by construction rather than by instruction, which is the rule of §8.3 applied to the counterweight itself. By design, the model that writes a challenge is not the model that judges one: in the first build the challenger is Qwen and the auditor is OLMo, both open-weight and running on the practice’s own hardware. The engine is not deployed; §9 explains why.

The evidence contract is the clause (b) commitment made structural. A challenge is not an object that may or may not carry a citation; it is an object that cannot be constructed without one, so a challenger that would rather assert than cite has no code path to do it.

# The evidence contract: a challenge with no readable record cannot exist.
@dataclass(frozen=True)
class Challenge:
    claim:     str   # the assertion the agent stopped on
    record_id: str   # address of a record in the operator's index
    quote:     str   # span reproduced from that record

    def __post_init__(self):
        if not self.record_id:
            raise Unconstructable("no record cited")
        if self.quote not in corpus.text(self.record_id):
            raise Unconstructable("quote is not verbatim in the cited record")

The contract, in outline. Names are simplified from the shipped engine; the control it expresses is the shipped one.

The lint then holds the two constraints the constructor cannot see: whether the record was actually in the set retrieved for this turn, which is what stops a challenger from citing something it never read, and whether the challenge speaks as a person, which is the clause (d) constraint of §8.2.

# Admission: what the constructor cannot check about a well-formed challenge.
def admit(ch: Challenge, retrieved: set) -> None:
    if ch.record_id not in retrieved:
        refuse("cites a record this turn was not given")
    if speaks_in_persona(ch.claim):
        refuse("speaks as someone rather than as the record")

Admission checks. A refusal names the failing predicate, so the refusal is itself a record.

What neither guarantees is that the cited record contradicts the claim. Retrieval by similarity returns records about the same topic, and agreement and contradiction look alike to it. A challenge that is well formed, correctly cited, and simply wrong would pass both gates above, count as dose under clause (c), and arrive carrying the operator’s authority under clause (d). That is the exact condition H3 says produces folding rather than correction, and it is why the precision check of §10.5 is a required instrument rather than an optional one.

8.6The judgment layer

The counterweight is one instrument inside a set of decisions the practice cannot delegate, and the set deserves naming because the market is commoditizing everything around it. Over 2026 the infrastructure beneath working agents, the orchestration, retries and state handling, became a managed rental; this practice rents such tooling and expects to keep renting it. What did not commoditize are the decisions that infrastructure exists to execute: whose knowledge becomes the standing instruction set; what result, fixed before work opens, ends an experiment; and what “done” means for a given piece of work. A practitioner essay contemporaneous with this program posed these three questions as the unanswered residue of managed-agent platforms (Forstie, 2026, cited under Prior art). The practice’s answers are machinery this paper has already described: changes to the standing instruction set are proposed by the machine and merged or refused by the operator; experiments carry decision rules ratified before their data are seen, under which a null closes the program (§10.4); and the finish line is ruled in advance. The record is the load-bearing element in each: every one of those decisions is made from the practice’s own logs, which is why §7 treats ownership of the record as a first-order property rather than a storage preference. An execution history that lives on a platform’s dashboard, in the platform’s format, is testimony in the sense of §6, not a record the operator holds.

9First calibration: the auditor folds

Before any challenge reaches a working agent, the plan requires one pair of numbers with intervals: how often an automated auditor, shown a claim and the evidence that preceded it, correctly flags a false claim, and how often its flags are right. The first run is itself a result.

The calibration set holds 170 claims about test outcomes made by agents during real sessions. Each is labeled mechanically from the nearest test-runner output that precedes it in the same session, and the auditor never sees the label. The auditor, a small open-weight model (OLMo-2-7B, run at 4-bit precision) on the practice’s own hardware, receives the claim and its evidence window, which is the tool output an agent would have had at the stopping moment, and answers SUPPORTED, CONTRADICTED or CANT_TELL. A hedged answer on a false claim counts against recall, because hedging is not catching. A second auditor from a different open-weight family (Qwen) then judged the same claims, plus one true claim added to the set after the first run.

The first auditor never returned CONTRADICTED. It supported 85 claims and declined to decide 85; of the 41 false claims, it caught none and affirmatively supported 17. In a sample of eight false-claim rows, six evidence windows carried an explicit failure marker. The auditor had the contradiction in front of it and still answered SUPPORTED or declined to decide. These are the easiest claims an auditor could be given: a claim about a test outcome, with the test runner’s own output in the window. The second auditor fired, rarely. It returned five contradictions, three on false claims and two on true ones, so it caught 3 of the 41, still supported 9, and declined to decide on 108 of its 171 verdicts.

A second judging prompt, pre-registered as a single variant and run on the same 171 claims the evening this paper was revised, settles what the first auditor’s zero meant and adds the sharper finding. The new prompt licenses disagreement, requires a verbatim quote of the outcome line, and places the verdict last. Under it the first auditor answered 150 of 171 rather than 85, returned 114 contradictions rather than none, and its recall on the 41 false claims rose from 0 to 0.683 with an interval from 0.53 to 0.80. Its precision was 0.246. That last number is the whole result: the auditor fired on 28 of the 41 false claims, 68%, and on 86 of the 130 true ones, 66%. It fires at the same rate whichever way the truth lies, so its precision is simply the share of false claims in the set. The prompt change moved the firing rate and left the discrimination where it was. The first run’s zero was the prompt shape suppressing the verdict, not the model failing to see the evidence; the second run shows that when the verdict is no longer suppressed, it is not informed either. Nothing further is tuned on these rows, and the gate holds.

Zhao et al. (2026) show the same brittleness is general rather than peculiar to small local models. Stress-testing nine frontier models across fourteen judging tasks, they find every one of them flips its verdict on 25 to 71% of items under static pushback, and on 62 to 91% when an adversarial model does the persuading. Their central finding is the one that bears on this program: pressure that succeeds in changing a judge’s verdict is almost always net-corrupting with respect to ground truth. An auditor that can be moved is not thereby an auditor that can be improved, and the thing that moves it moves it the wrong way.

We read this as the stop problem reproduced in the instrument meant to catch it. An auditor built from the same class of model defers to the claim it is shown, much as Kelley and Riedl (2026) observe of models cast as peers, and as §6.5 leads one to expect of any model asked to arbitrate. The second family does not overturn that reading. It traded some of the deference for hedging, and its recall interval still sits far below anything a deployment could rest on; three catches in 41 cannot say whether the two families really differ, since their intervals overlap. Both are small local models; a frontier-class auditor was not tested, and the result bounds only the small open-weight case the practice can run offline. It is also the case H3 anticipates: an auditor that folds produces neither reversals nor corrections, and so cannot distinguish them. The plan’s gate held, and neither auditor is deployed on these numbers.

The next steps, in order of cost, are a sharper judging prompt, evidence windows centered more tightly on the failure marker, and a comparison of where the two auditors agree and disagree, which decides whether a pair of them adds anything over one. The denominator carries known noise, since at least one sampled false claim is a passage of prose rather than a claim; noise can move recall’s magnitude, but it cannot produce zero flags across 170 rows. The verdict counts in Table 6 were recounted from each run’s per-row output, the label-dependent figures were read from each run’s report file, and the sample of eight is from the implementing team’s report.

10Evaluation

10.1Questions

1. RQ1. Do operator challenges at the stopping moment cause material reversals in agent output?
2. RQ2. Can a mechanical counterweight predict where the operator would challenge, reliably enough to act on?
3. RQ3. Does delivering such challenges at the stopping moment measurably reduce recurrence of the target error classes?

10.2Design

The design is a single-case experimental design with N = 1, extending to a multiple-baseline design across co-researchers onboarded at staggered start points: the intervention begins at a different time for each subject, and an effect should appear at each start point and not before. Single-case standards conventionally require at least three demonstrations of an effect at three different points in time, so that a single coincidental change cannot pass for one (Kratochwill et al., 2013); the reporting guideline for this family is SCRIBE (Tate et al., 2016), and the n-of-1 trial extension is CENT (Vohra et al., 2015). None of the three is indexed where this program’s literature searches run, which is its own finding: the design standard this paper should be held to lives in education and behavioral science, and a search confined to machine-learning venues will never surface it. The program’s own phase criteria currently ask for two and are being corrected to three to meet that standard. Three units are in use and are not conflated: the turn is the sampling unit for adjudication (about 8,221), the session is the exposure unit, and the class-instance is the unit of the primary outcome. The evaluation’s datasets come from different corpora of §2.2 and do not overlap: the calibration claims of §9 are drawn from the code-agent transcripts, where a test runner’s own output supplies each label, and Study 2’s pairs are drawn from the operator’s chat-lane record, where the operator’s own challenge does.

10.3Constructs

10.4Decision rules

Pre-registration here means that a decision rule is ratified before its data are examined. Table 8 gives each study’s rule and its standing in September 2026, including where that standard was not met.

Study 2 received its first live reading on 11 September 2026, against 185 turn pairs in which both the engine’s firing decision and the operator’s own subsequent challenge are observable. The quantity is the lift of the engine’s firing over the ambient rate:

agreement-with-key lift over a rate-preserving permutation = 0.059 Bootstrap interval over pairs, 0.011 to 0.108, excluding zero. Computed over all 185 joined pairs with no train-and-test split, at an uncalibrated retrieval floor of 0.35, firing on 172 of them.

Two things follow, and only one of them is encouraging. The association is real and positive, which is what the criterion’s first conjunct asks for. But an engine that fires on 93% of turns is not a predictor of anything; at that operating point almost every turn is challenged, so the dose requirement of clause (c) is satisfied trivially while the discrimination requirement of clause (b) is not tested at all. The criterion’s second conjunct, beating the fallback, could not be evaluated at the time for a blunter reason: the fallback had never been defined, and a conjunct whose comparator does not exist cannot be passed or failed, only skipped. Both gaps were closed by ruling on the day this paper was revised. The operator named the fallback, a second open-weight model judging each pair directly, and ruled that the firing floor must be set on one seeded half of the data and scored on the other, which replaces the single uncalibrated pass the reading above came from. The implementing change was merged and the first run under the ruled design completed the same evening, and it is the reading this section now rests on. The floor was chosen on a seeded calibration half of 92 pairs at 0.485, where the engine fired on 65. On the held-out half of 93 pairs carrying 44 operator corrections, the engine fired on 58 and the fallback, the second open-weight model judging each pair directly, fired on 7 and decided all 93. Against chance the lift was 0.100 with an interval from −0.043 to 0.237; against the fallback it was 0.054 with an interval from −0.097 to 0.194. Neither interval excludes zero. Under the ruled criterion the challenger as built does not pass either conjunct, and the reading is discard-or-fix. The point estimates are positive and the intervals are about 0.14 wide on either side at this n, so the result is an absence of demonstrated effect rather than a demonstrated absence, but the pre-registered bar was set before the data and the engine did not clear it. The implementing seat’s diagnosis matches the contract’s own known limit: retrieval by similarity fires on topic, not on contradiction. A fix cannot be re-scored on these rows and needs fresh pairs. As built the challenger fails both conjuncts, and the reading is discard-or-fix: the path is not killed, but nothing on these rows is carried forward and any fix must be scored on fresh pairs. That the engine retrieves by topical similarity rather than by contradiction is the diagnosis a fix has to answer.

10.5Fidelity

A null is uninterpretable without a fidelity measure, because it cannot distinguish a counterweight that does not work from one that was never delivered as specified. Four checks are required before the interventional arm opens. A delivery check logs render events and joins each to a read-after-render test; its pre-registered falsifier is that twenty renders with zero reads after render means the delivery mechanism failed, independent of any outcome. A clause check scores each fired challenge for externality, for whether it named a checkable source, and for whether it fired at a detected stopping moment; the engine now enforces the second by construction. A dose check records the firing rate against the ambient error rate for the same window, which is how the rate hypothesis of clause (c) gets tested rather than assumed. And a precision check adjudicates a sample of fired challenges for whether the cited record actually contradicts the claim. The evidence contract guarantees that a challenge cites a record, not that the record contradicts anything, since retrieval by similarity cannot tell contradiction from agreement about the same topic. A wrong but cited challenge would pass every other check, count as dose, and carry the operator’s authority under clause (d), which is the exact condition under which H3 predicts folding rather than correction.

10.6Reliability

The program sets an agreement floor of κ ≥ 0.70 for any claim derived from labels. No verified measurement of agreement on the operator-class split is available, and none is reported here. A figure of κ = 0.410 circulates in the program’s record, but its producing script could not be located, and every traceable occurrence of that value belongs to a different measurement, the coding of a precedence clause’s location over an 88-row corpus with a 44% uncodable share. The floor is currently report-only, enforced by discipline rather than in code. A second blind pass by the operator over 40 stratified turns puts a ceiling on what any label-derived claim can say, with one caveat that has to travel with it: those 40 rows were drawn from the whole corpus of 289 conversations, while the study they bound now targets a narrower lane of 113, so the ceiling and the target were measured on different populations and the figure needs a rescoped redraw before it is used against that target. The recorded reading was agreement on the binary teach flag of κ = 0.40 (95% CI 0.11 to 0.66), with five of six finer channels indistinguishable from chance. On the evening this paper was revised a seat traced that reading to a join defect: the key file’s row identifier places every mark after item 26 one row above its item, and 37 of the 40 re-labeled items sat on a shifted row. Before any edit the defective join reproduced every recorded figure exactly, so the move is the join and nothing else. Under the repaired join the same 40 rows read κ = 0.554 (95% CI 0.292 to 0.798), with one channel rather than five at chance, and every detector graded against the key moves with it, the lexical rule from 0.04 to 0.21 and the local model from 0.09 to 0.37. The repair was merged on 11 September 2026, verified at the repository rather than at a report of it; the program’s ruling that cites 0.400 had not been amended when this revision closed, so this paper reports both readings with their status and treats neither as settled. Two consequences follow either way. The ceiling’s upper bound now reaches the 0.70 floor, so “unreachable” is not established, and at n = 40 the interval cannot tell unreachable from hard. And §3.1 continues to report corrective signal as one undifferentiated quantity, now because the separability of the finer channels is unsettled rather than because it was shown absent; no claim in this paper rests on the finer channels under either join. Its consequence is bounded: the shortfall blocks label-derived claims, not the natural-signal measurement of the primary outcome, which does not depend on the contested labeling route.

10.7An external instrument

Every measurement above runs on the practice’s own corpus. One planned measurement does not: a bench on the public ARC-AGI-3 game set, adopted in September 2026, in which the challenge is delivered at the stopping moment in one arm and withheld in the other, and performance is compared with a human baseline. It is designed and not yet implemented. Until it runs, the program’s evidence is the practice’s own.

10.8Ethics and data

Co-researchers are volunteers and collaborators, not subjects or staff, and none is employed by the company. Before any co-researcher contributes measured data the program requires a participant information statement, a recorded consent, and a data-handling statement covering retention and withdrawal. The training data for the counterweight are operator-authored turns only; agent outputs enter as unlabeled context.

10.9The program as built and as planned, 21 September 2026

The 11 September revision described the counterweight’s engine as implemented and not deployed. In the nine days since, three build waves landed and merged, and the honest way to report them is by what each did to a measurement problem this paper has already named. A wiring audit now runs nightly: of 192 event producers in the practice’s codebase, 122 are wired to a reader, 70 are declared, none are unwired, and a burn-down of 55 grandfathered producers is printed each night, so a signal that reaches nothing can no longer do so silently (§5’s delivery argument, applied to the practice’s own instruments). A delivery switch for the counterweight’s challenge arms is built and deliberately unarmed: until the operator writes the first schedule entry under an account the working seats cannot modify, every session is unassigned, and unassigned delivers nothing. The sample-size calculation that had existed only as prose now has a producer: at the measured compound-recurrence share of 0.5101 with lag-one autocorrelation 0.2547 and the proposed smallest effect of interest of 0.25, a null is reachable one-sided at 95% with 84 sessions per arm under Bernoulli assignment and 140 under cycle-switch (two-sided, 119 and 199; the 11 September prose figures reconciled with this producer on the dependence model and not on the base n, and their method was never recorded, which is why the producer exists). And a machine-fed adherence instrument is on the nightly: the compaction hook writes a row the seat cannot decline to emit, and a subsequent load is the only thing that clears it. At this revision it reads 70 compactions with 43 reloaded, 18 not, and 9 unresolved, a settled not-reloaded rate of 29.5%, and the monthly split shows the direction: August ran 33% reload, September 85%.

What remains is not construction. It is a short chain of operator decisions, recorded here because §10.4’s pre-registration freezes the instrument when signed, so the order matters. First the unit of analysis: a session identifier in this practice spans one turn to ninety-seven and zero compactions to sixteen, so a per-session outcome partly measures session length until the operator rules the unit. Second the feed: both of the primary outcome’s current inputs are typed by the seat being measured, and the record shows what that produces; the practice’s miss log accrued from twenty-two sessions in June and eighty-one in July, through thirty-two in August, to seven in September, a collapse in voluntary emission, not in error, while the alternative decision log holds fifteen rows in its entire history, six of them written by the session that built the logger. Choosing between a hand-fed instrument with measured decay and a machine-fed one with a third bucket is the operator’s call, and this paper’s own §6 is the argument that it cannot be the seat’s. Third, the compute moves off the operator’s laptop, which suspends scheduled work when it sleeps and therefore cannot host an unattended measurement window. Fourth, the switch is armed, the pre-registration row is signed with the window, the n, and the frozen instrument identities, and the study of §10.4 runs to one of its three registered exits. That is where this program is pointed: challenges delivered on schedule to assigned arms, fed by signals the measured party cannot withhold, scored against a rule fixed before the data existed, on infrastructure that does not sleep, with the operator holding every gate that changes the system. The reader should weigh the plan accordingly: everything in this subsection before the word “remains” is merged and running, and everything after it is intention, dated 21 September 2026.

11Threats to validity

Construct. The primary outcome is a proxy. Recurrence can fall because an error class genuinely recurs less, or because detection of the class degraded. The mitigations are an independent adjudication arm that does not share the detector’s blind spots, and the per-class series, in which a detection collapse shows as a simultaneous fall across classes. Srinivasan and Paragiri (2026) give the general form of this hazard for agent-driven search: where validity lives in disaggregated structure, an aggregate reduction can rank the wrong candidate first, the headline number improving while the structure beneath it inverts. Their remedy is an external control loop that audits disaggregated behavior after the agent has decided and can reopen a run the agent declared finished, which is the shape of the off-target conjunct now proposed for Study 3 (§10.4). The pushback reading behind §3.1 and §6.2 comes from a machine labeler with 64% precision on the blind key, and its adjusted figure depends on that key being representative of the frame. The calibration set’s labels are mechanical: they establish whether a test passed, not whether the agent’s work was correct.

Internal. Instrumentation is a live threat, not a hypothetical one. A source field on the practice’s records defaults to “self” when unset, so historical self-catch shares (30.6%, n = 556; 25.5% on wrong diagnosis) cannot distinguish a genuine self-catch from an unlabeled row. An audit of the producing repository confirms the defaulting behavior and the two shares. The remedy is to reject an unset source at the write path, flag explicit sources, and report historical and post-fix rows separately, with no backfill. History and maturation are present over a six-month corpus in which the harness changed repeatedly; testing effects are present because the operator knows a measurement is running; and selection is present because the error log records only caught errors.

Evidence from the practice’s own instruments. By the rule in §8.3, instruments the practice built and operates are exposed to the measured party, and every reading in §3 to §7 comes from such an instrument. We claim no exemption. The readings are exploratory; the primary outcome depends on a natural-signal measurement checked by an independent adjudication arm; and the one external instrument (§10.7) has not yet run.

External. One operator, one domain, one harness, one model family for the agents. Nothing here generalizes to other operators without replication, and the multiple-baseline extension raises N to four within one organization, not to a population. The setting’s ecological validity is bought with exactly this cost.

Conclusion. Many analyses have been run against one corpus over six months, and the multiplicity is not accounted for. Pre-registration of the primary rule is the principal control, and secondary findings, including every reading in §3 to §7, the calibration in §9 and Study 1, are reported as exploratory.

Reflexivity. The AI assistants used in this research come from the same model family as the agents under study and exhibit the defect under study. An earlier draft of this paper’s case series contained seven factual errors of exactly the kind the paper describes, among them a twenty-hour window described as one morning and one automated catch counted as two; all were found on re-check against the transcript. A later draft carried, as its headline exposure figure, a census count of whether the operator’s reply contained one of six literal phrases, reported as the share of answers that were never challenged at all; it passed the paper’s own fact-check because it carried a receipt, and the operator caught it. Drafts written on the day this version was revised did it several more times: a one-day tally of seven misses read as a census of rule adherence; a recurrence count that equals the number of sessions seen minus one by construction, quoted as though it could vary; a reliability figure whose only traceable producer is a different instrument; and, in the literature review, a range that does not exist in the cited paper, a diagnostic arm quoted as the deployable result, a section header promoted to a title, and a quotation attributed to a paper that does not contain it. Every one was caught before the paper left the room, by the operator or by fetching the source, and the corrected figures stand in the text without further comment. Read together, the day’s errors share one shape: none was a miscalculation, and every one was a bad join. A count was joined to the wrong meaning (a recurrence tally that is the session count minus one), a statistic to the wrong instrument, a labeling ceiling to a key one row off its items (§10.6), and copies of a retired figure to a search that never reached them. The mitigation this suggests is retrieval-shaped rather than judgment-shaped: identifiers that resolve or fail loudly, and provenance edges written by instruments the agent cannot author, so that a figure carries its join and the join can be checked. It does not touch the judgment-shaped half, in which an auditor holds the contradicting evidence in its window and folds anyway (§9). The pattern is the finding. The paper’s own production is a running specimen of its thesis. A hostile reading turns that around: a process that needed these corrections mid-draft produced the surviving numbers too. The answer is not a defense of the process, which would be the defect restated; it is that the paper’s weight-bearing claim is the one object that process cannot have shaped, a decision rule ratified before its data exist, and that everything else is labeled exploratory and priced accordingly. The mitigations are mechanical rather than discretionary assignment of assistant instances to tasks, the operator outside the loop as the final check, and independent review of the design before the primary study opens.

12Limitations

The program cannot establish a catch rate. The logs contain only errors that were caught, and no denominator of uncaught errors exists in them. A denominator is obtainable from the full transcript record but requires an adjudication arm that is not yet in production; until it is, a catcher share may be reported with its n, its window and its explicit-versus-defaulted split, and a rate may not be reported at all.

The program cannot establish that the counterweight raises correctness in the world. It targets discipline at the stopping decision, meaning whether the process that makes a claim checkable was followed. That is a narrower claim and the one the corpus can support.

The logging gap of §6.2 is an indication across two instruments, not a measurement. The rate requirement of clause (c) is untested. Study 1 is exploratory, the Study 2 criterion was re-specified in September 2026, and neither auditor has passed calibration.

13Conclusion

The stop problem sits at the termination decision, not in the reasoning before it. That is why better reasoning does not remove it, and why an external check timed to the stop might. We have argued five propositions from one practice’s logs and from recent work, found the same shape in the practice’s own instruments, specified a counterweight from published work, and built its challenge engine. Its first calibration shows two auditors folding or hedging on the evidence in front of them, which is the problem restated inside the instrument meant to solve it, and exactly what a gate before deployment exists to catch. The program is built to be able to fail. If the recurrence measure does not move, that is the result, and the program ends.

What transfers, if anything does. Three rules earned here do not depend on this practice’s N of 1. A boundary that refuses and records its refusal outlasts a memo that asks an agent to remember (§5). A challenge must cite evidence the challenged agent can open, or it is an opinion wearing a citation (§8.2). And an outside checker must be calibrated against known-false cases before it is allowed to gate anything, because a checker built from the same class of model may simply agree with the claim it is shown (§9).