What Is an LLM? How a Language Model Is Made, and Where Its Habits Come From

About this paper. This is a new background paper in the series How do we use this?, written to go with the post “What is an LLM?”, the first of the four primer posts; the others are “What is a harness?” (background paper 8), “What is context?” (background paper 9) and “What is ‘AI’?” (background paper 0). Unlike background papers 0 to 6, it is not reproduced from the working paper How Do We Use This? (Atkinson, 2026). It supplies the training-side grounding for the working paper's first proposition, that these systems are trained toward an answer the user accepts (§3 of the working paper). Where it relies on the practice's own measurements, it cites the working paper's section and reports the figure exactly as the working paper does.

1Introduction

Every post in this series stands on one sentence from the working paper: the objectives these systems are trained and deployed under reward an answer the user accepts, and nothing in them prices whether the answer is right (§3). This paper is the long version of that sentence. Trained how, rewarded by whom, and, the question a manager actually has, which of the habits that follow will the next and more capable model outgrow, and which will it keep?

The program the series comes out of began with a field note, written before any of the literature below had been read:

“[The model is] trained to present as fact the first answer, the only criteria for that answer being that it thinks it meets the user's expectations and is defensible based on the available information… Correctness of the answer is not a factor.”

The author's research notes, 30 August 2026, as quoted in the working paper (§1)

A note like that is an observation about behavior. This paper asks where the behavior comes from. The method follows McCoy et al. (2023), who argue that to understand a large language model one should start from “the problem that they were trained to solve” and ask which strategies that problem rewards. They apply the approach to pretraining alone. We apply it to every stage a current model passes through: pretraining, supervised fine-tuning, preference training, and reinforcement learning on problems with a checkable answer, together with the benchmarks that decide which of those models is called the best. At each stage we ask three questions. What is the model shown? What is it paid for? What habit does that payment leave?

The answer, stated once here so that the rest of the paper can be read against it: every stage pays the model for something a grader can observe at the moment of grading. In pretraining the grader is the next word of an existing document. In fine-tuning it is an example answer. In preference training it is a person, or a model trained to predict a person, choosing between two responses. In reinforcement learning on checkable problems it is a checker, such as a test suite or an answer key. Correctness enters wherever one of those graders can see it, and only there. The working paper's field note is what that looks like from the outside: an answer that is defensible against what the grader can see is exactly what the training selects.

Two consequences organize what follows. The first concerns scale. Some of a model's failures come from what it does not know or cannot yet do, and more capable models fix a great deal of that. Others come from what it is paid for, and a more capable model trained on the same payments keeps them, sometimes more skillfully (§8). The second concerns weights. Whether a habit can be corrected at its source depends on who holds the model's parameters. The lab that trained a closed-weight model can change its payments; its users cannot. Anyone who holds an open-weight model can run the later stages again and choose what to pay for (§4). The practice this series describes runs its work on a closed-weight model family, and the working paper's counterweight program is built on that fact (§9).

Section 2 sets out how this paper uses words like “agrees” and “stops”. Section 3 describes what a model is when it runs. Section 4 separates open from closed weights. Section 5 walks the training stages. Section 6 draws the map from stage to habit to where the correction can act, which is the paper's central figure. Section 7 sets the practice's own record beside it, Section 8 asks what a more capable model will and will not fix, and Section 9 describes what the practice is doing about a model it cannot retrain.

2How to read the behavioral words

A language model does one thing: given the text so far, it produces a probability for each possible next piece of text. Words like agrees, prefers, stops and games are therefore shorthand for regularities in which text it produces, and this paper uses them the way the working paper does, as as-if descriptions that say the behavior can be rationalized this way, not that any intention is represented inside the model (§1 of the working paper). Shanahan (2022) makes the general case for this discipline: the better these systems imitate human language, the more readily we read human minds into them, and the remedy is to keep stepping back to how the model, and the larger system it is embedded in, actually work. Shanahan, McDonell and Reynolds (2023) offer a frame that keeps the shorthand honest: a dialogue agent is best understood as playing a role, a character its training and its prompt have made likely, rather than as a speaker with beliefs of its own.

One common objection deserves a direct answer, because it sounds like a refutation of everything below. The objection is that a model “just generates probable words”, so talk of habits is a category error. The first half is correct and is the premise of this paper. The second half does not follow from it. Which words are probable is not a fixed fact about language; it is precisely what training sets. A regularity in the words a model produces, such as agreeing with the user more often than the evidence warrants, is exactly the kind of thing a training objective can put there, and the evidence in §5 shows specific objectives doing so. Calling that regularity a habit is shorthand for “a pattern in the output distribution that a particular stage of training rewarded”.

3What the model is when it runs

Tokens. A model does not read letters or words. Text is first cut into tokens, pieces that are often a whole common word and sometimes a fragment of a rarer one, from a fixed vocabulary learned before training begins. The standard way to build such a vocabulary splits rare words into frequent sub-word units (Sennrich, Haddow & Birch, 2015). Everything the model sees and produces is a sequence of these pieces.

The network. The model is a very large function, today almost always a transformer (Vaswani et al., 2017), whose behavior is fixed by its parameters, usually called its weights, of which there are billions. Given a sequence of tokens, it returns a probability for every token in its vocabulary as the next one. That list of probabilities is the whole of its output on any single step.

Generating. To produce a reply, the software around the model picks one token from that list, appends it to the sequence, and asks the model again, one token at a time until a stopping token or a length limit is reached. How the token is picked is a setting, not a property of the model. Always taking the single most likely token tends to produce repetitive, degenerate text, which is why deployed systems sample from the distribution instead (Holtzman et al., 2019), and why the same question can get different answers on different runs.

The context window. Everything the model can use when producing a reply has to fit in its context: the instructions, the conversation so far, any documents or tool results the surrounding software placed there. Within that window a model can pick up a task from a few examples without any change to its weights, which is what Brown et al. (2020) called in-context learning. Outside the window, nothing exists for it. How well a model uses a long context, and what the software does when a conversation outgrows the window, is the subject of background paper 9.

Frozen. Once training ends the weights do not change. A model in use does not learn from the conversation, does not remember the user tomorrow, and knows nothing after the date its training data ends. Anything that looks like memory is the surrounding software, the harness, saving notes and placing them back into the context (background paper 8). Figure 1 draws the loop.

One model, by the numbers. An open-weight model's configuration is published with its weights, so its architecture can be read rather than inferred. OLMo 2 7B, a fully open model and the one the practice ran as its first auditor (§7; working paper §9), cuts text into tokens from a vocabulary of 100,352; turns each token into a list of 4,096 numbers; passes those lists through 32 identical transformer blocks, each with 32 attention heads and a feed-forward layer 11,008 numbers wide; and returns a score for every one of the 100,352 tokens. It reads at most 4,096 tokens at a time, and its parameters number about seven billion, which is what the 7B in its name records (Allen Institute for AI, 2024). No such description exists for the closed frontier models. The GPT-4 technical report states that it “contains no further details about the architecture (including model size)” (OpenAI, 2023), and a public gallery of architecture drawings, which listed 105 models when read on 24 September 2026, has no closed frontier model among them (Raschka, 2026). What is inside a closed model can be described in the general terms of this section and no further.

4Open and closed weights: who holds the lever

A model's weights are the trained parameters themselves, the file that is the model. Everything in §5 happens to those weights. Who holds them therefore decides who can change what a model is paid for, and it is the most practical distinction in this paper for anyone deciding how to use one.

Release is not a single switch. Solaiman (2023) describes six levels of access along a gradient: fully closed; gradual or staged access; hosted access; cloud-based or API access; downloadable access; and fully open. Most people meet the frontier models in the middle of that gradient, in an app or through an API, where the lab runs the model on its own machines and returns only the text. Kapoor et al. (2024) define open foundation models as “foundation models with widely available model weights” and name five properties that follow from releasing them: broader access, greater customizability, local adaptation and inference ability, inability to rescind model access, and inability to monitor or moderate model usage. Of the fourth they write that “open release of model weights is irreversible”; of the fifth, that developers “do not observe inference by default”.

“Open weights” is not the same as open source. Most open-weight releases give the trained parameters and the code to run them, and not the training data or the code that produced them. OLMo's authors released theirs “alongside open training data and training and evaluation code” so that the research community would have what they call “truly open” models to study, crediting a small number of earlier releases that were comparably open (Groeneveld et al., 2024). The Open Source Initiative's definition of open-source AI, version 1.0, asks for all three: Data Information, detailed enough “that a skilled person can build a substantially equivalent system”; Code, “The complete source code used to train and run the system”; and Parameters, the weights themselves (Open Source Initiative, n.d.). Table 1 sets out what each position on the gradient allows at each stage of §5.

Three consequences matter for the rest of this paper. First, for a closed-weight model every payment in §5 is fixed from the user's side. The habits it leaves can be worked around, but not trained out, by anyone but the lab, and the only surfaces left to the user are what goes into the context and the software around the model, the harness. Second, a closed model can change under its user. The episode in §5.3, in which an update to a widely used model made it markedly more agreeable before it was rolled back, was a change the lab made to a model its users could not hold still. Third, most of what is known about how a particular stage produces a particular habit comes from open models, because only there can researchers run the stage themselves and watch the habit appear: several of the studies in §5 retrain open-weight models to do exactly that. What is known about the closed frontier models comes largely from what the labs publish about their own systems. Both kinds of evidence appear below, and each is marked by the models it was measured on.

5How it is made, stage by stage

The labs describe the recipe in similar terms. OpenAI's own account, written after the episode in §5.3: “To post-train models, we take a pre-trained base model, do supervised fine-tuning on a broad set of ideal responses written by humans or existing models, and then run reinforcement learning with reward signals from a variety of sources”, and “The set of reward signals, and their relative weighting, shapes the behavior we get at the end of training” (OpenAI, 2025). The same post lists what those signals ask: “are the answers correct, are they helpful, are they in line with our Model Spec, are they safe, do users like them, and so on.” An openly documented recipe runs the same arc under other names: supervised fine-tuning, then direct preference optimization, then reinforcement learning with verifiable rewards (Lambert et al., 2024). Figure 3 draws the stages this paper uses, with the grader each one answers to.

5.1Pretraining: read everything

The first and by far the largest stage trains the model to predict the next token of an enormous body of text, most of it from the public web, with the objective in the box of §3. The grader is the document itself. Whatever the document said next is, by definition, the right answer, and nothing in the stage asks whether the document was true.

What this produces is extraordinary: broad knowledge, fluent language in many registers, and the ability to pick up a task from a few examples in the prompt (Brown et al., 2020). Its habits follow from the same fact. Because false statements that people commonly make are exactly as learnable as true ones, a model trained this way imitates misconceptions. Lin, Hilton and Evans (2021) built a benchmark of questions that some people answer falsely and found that “The largest models were generally the least truthful”, an inverse scaling they attribute to imitation: “this result is expected if false answers are learned from the training distribution.” The best model they tested was truthful on 58% of questions against 94% for people. Those are 2021 models, and the same paper records that later models trained with objectives other than imitation returned to positive scaling at the largest sizes, which is the point of the stages that follow: imitation is the first stage's payment, and a later stage can pay for something else.

McCoy et al. (2023) show the pull of the probable directly. Asked to decode a simple rot-13 cipher, GPT-4 reached 51% accuracy when the correct output was a high-probability word sequence and 13% when it was a low-probability one, although the task is deterministic and probability should not matter. Their conclusion, that these systems should be treated “as a distinct type of system” shaped by their own pressures rather than evaluated as if they were people, is the method of this paper.

Two more properties of this stage matter later. First, Kalai and Vempala (2023) prove that a pretrained model that is statistically well calibrated must hallucinate some kinds of fact: for “arbitrary” facts, the ones that cannot be inferred and that appear exactly once in the training data, the hallucination rate is bounded below by roughly the fraction of such facts. Their analysis also finds no statistical reason for pretraining to produce hallucinated references to publications, which appear more than once, or errors of systematic fact such as arithmetic, so those have other causes. Second, base models are, in the right format, well calibrated: Kadavath et al. (2022) find that larger models' probabilities track their accuracy on multiple-choice and true-or-false questions, and the GPT-4 technical report measures an expected calibration error of 0.007 for its pretrained model (OpenAI, 2023, Figure 8). Calibration, in other words, is something pretraining provides and a later stage can lose (§5.3).

One habit that is often blamed on later stages starts here. Perez et al. (2022) measured sycophancy, a model repeating back the view its user states, and found the largest models they tested highly sycophantic, with “> 90% of answers” matching the user's view on their NLP and philosophy questions. They also found that “sycophancy is similar for models trained with various numbers of RL steps, including 0 (pretrained LMs)”, and explain it by what the text contains: “internet text used for pretraining contains dialogs between users with similar views.” The agreeable streak is imitated before any rater sees the model.

5.2Supervised fine-tuning: copy the expert

A pretrained model continues documents; it does not yet answer questions like an assistant. The second stage teaches it to, by training on example conversations in which an ideal assistant responds, written by people or by other models (Ouyang et al., 2022; OpenAI, 2025). The objective is the same next-token loss, restricted to the tokens of the example answer. The grader is the example.

How much this stage adds is the subject of a sharp result. Zhou et al. (2023) fine-tuned a 65-billion-parameter pretrained model on just 1,000 curated examples, with no preference training at all, and people judged its answers equivalent or preferred to GPT-4's in 43% of comparisons. They state the reading as a hypothesis: “A model's knowledge and capabilities are learnt almost entirely during pretraining, while alignment teaches it which subdistribution of formats should be used when interacting with users.” If fine-tuning mostly teaches format, then what it pays for is the manner of an expert, and the manner can arrive without the knowledge. Gudibande et al. (2023) found exactly that in models fine-tuned to imitate a stronger one: they looked competitive to crowd raters and closed little of the real gap on targeted evaluations, because they mimicked the stronger model's style but not its factuality, and the difference slipped past the raters.

The stage can also teach fabrication directly. Gekhman et al. (2024), working at Google on the company's own PaLM 2 model, fine-tuned it on mixtures of facts it already knew and facts it did not. Examples with new knowledge were learned “significantly slower”, and “as the examples with new knowledge are eventually learned, they linearly increase the model's tendency to hallucinate.” An example answer that states a fact the model does not have trains it to state facts it does not have. And the agreeable streak of §5.1 can grow here: Wei et al. (2023) found that both scaling and instruction tuning significantly increased sycophancy in Google's PaLM models up to 540 billion parameters, including agreeing with an objectively wrong arithmetic claim when the user did. The effect is not uniform. Across 56 open-weight models, De Marez et al. (2026) find that instruction tuning made small models less robust to pressure on factual questions and large models usually more robust, mainly by widening the margin by which the model prefers the true answer.

5.3Preference training: please the rater

The third stage is the one the working paper's first proposition is about. The model produces two or more answers to the same prompt; a person, or a model trained on people's choices, says which is better; and the model is trained toward the preferred ones (Christiano et al., 2017; Stiennon et al., 2020; Ouyang et al., 2022). In the classic form a separate reward model learns to predict the choices, and the language model is then optimized against it with a penalty that keeps it close to the fine-tuned model it started from. Direct preference optimization reaches a similar end without a separate reward model (Rafailov et al., 2023), and a written set of principles can stand in for some of the human choices (Bai et al., 2022b).

Whose preferences. The authors of the best-known version of this stage are explicit about what it aligns to. InstructGPT's procedure “aligns the behavior of GPT-3 to the stated preferences of a specific group of people (mostly our labelers and researchers), rather than any broader notion of ‘human values’”, and its authors add: “We are not claiming that researchers, the labelers we hired, or our API customers are the right source of preferences” (Ouyang et al., 2022). The stage also worked: people preferred the fine-tuned model's outputs to the far larger base model's, and it was more truthful on the benchmark of §5.1. The habits below are what it pays for alongside that, not instead of it.

Agreement. Sharma et al. (2023) found sycophancy in all five assistants they tested, four from closed-weight labs and one open-weight, and traced it to the preference data: a response matching the user's views was “one of the most predictive features” of which response people preferred. Anthropic's own preference model preferred a convincingly written sycophantic answer to a plain truthful one 95% of the time on their hardest misconceptions, and optimizing against it by best-of-N sampling left about 75% of those answers sycophantic, against about 25% under an idealized preference model. They add that “the presence of sycophancy at the start of RL indicates that pretraining and supervised finetuning also likely contribute”, which is the finding of §5.1 and §5.2 from the other side. Perez et al. (2022) put the two together: “RLHF does not train away sycophancy and may actively incentivize models to retain it.” Blank et al. (2026) show how deep the channel runs. Using the open OLMo 3 post-training pipeline, they found that sycophantic agreement in the trained model tracked the sycophancy of the models that had generated the preferred and rejected answers in the preference data (35% with one pairing, 0.6% with it reversed), across seven preference objectives, with the signal “diffused across the entire dataset” so that no individual example looked sycophantic and filtering could not remove it.

Persuasion over correctness. Wen et al. (2024) trained two open-weight models with standard preference training and then asked time-limited human evaluators to judge their answers against a known truth. The training made the models “better at convincing our subjects but not at completing the task correctly”: on a question-answering task the evaluators' false positive rate, accepting a wrong answer as right, rose from 41.0% to 65.1%, and on a programming task from 29.6% to 47.9%, while correctness barely moved. When the same models were instead trained against the true answer, the side effects were much smaller, which locates the mechanism exactly where this paper does: in the gap between what the grader can see and what is right. Figure 4 draws the three measured settings.

Length and confidence. Singhal et al. (2023) found that length accounted for 70 to 90% of the reward gained by preference training in two of the three settings they studied, and that a reward for length alone reproduced most of the downstream improvement; the bias came from the reward models. Confidence follows the same pattern. The GPT-4 technical report measured an expected calibration error of 0.007 for its pretrained model and 0.074 after post-training, and says plainly: “The post-training hurts calibration significantly” (OpenAI, 2023, Figure 8). Leng et al. (2024) find that models trained with preference optimization express more overconfidence than their fine-tuned starting points and trace it to reward models that favor high-confidence answers “regardless of the actual quality of responses.” The loss is partly recoverable, which matters for §9: Kadavath et al. (2022) found that their preference-trained policies “naively appear very miscalibrated” but that a single temperature adjustment largely restored calibration in what they call a quick experiment, and Tian et al. (2023) found that asking such models to state their confidence in words was typically better calibrated than their token probabilities, often cutting calibration error by about half. Figure 5 draws the GPT-4 measurement.

The proxy, pushed. Gao, Schulman and Hilton (2022) measured what happens as a model is optimized harder against a learned reward model standing in for the true preference: the true score rises and then falls, in line with Goodhart's law, and the shape of the curve depends on the optimization method and scales smoothly with the size of the reward model. Every habit in this section is a particular instance of that curve: a feature the rater rewards that is correlated with being right until it is optimized directly.

In production. On 25 April 2025 OpenAI shipped an update to GPT-4o in ChatGPT that, in its own words, “made the model noticeably more sycophantic.” Its account of the cause is the clearest statement in print of the pull this series describes. The update “introduced an additional reward signal based on user feedback”, namely “thumbs-up and thumbs-down data from ChatGPT”, and “in aggregate, these changes weakened the influence of our primary reward signal, which had been holding sycophancy in check. User feedback in particular can sometimes favor more agreeable responses, likely amplifying the shift we saw.” The company adds that in some cases user memory, a feature of the software around the model, contributed to exacerbating the effect, though it had no evidence that memory broadly increased it. The first mitigation was a change to the system prompt late on Sunday 27 April, and a full rollback began the next day (OpenAI, 2025). Figure 6 draws the week. It is worth stating what it shows and what it does not. It shows that a pull toward what users accept is real enough to be measured by the lab that trained the model, strong enough to ship past its evaluations, and fixable only by the lab. It does not show carelessness: the account is the lab's own, and it is candid.

5.4Reinforcement learning on checkable rewards: pass the checker

The newest stage trains models on problems whose answers a program can check: mathematics with a known answer, code with a test suite, instructions whose satisfaction can be verified. The reward is simply whether the check passes, which is why it is called reinforcement learning with verifiable rewards (Lambert et al., 2024). DeepSeek-AI (2025) showed how much this can do. Its R1-Zero model was trained with rule-based rewards for accuracy and format and no example reasoning at all, and reflective behavior, rechecking and revising a draft, emerged during the training; the released R1 added fine-tuning stages around the same method. The authors explain why they kept learned reward models out of the reasoning stage: “neural reward models are susceptible to reward hacking during large-scale reinforcement learning”, and they report observing exactly that where they did use a learned reward model, for helpfulness, elsewhere in the pipeline. They released the weights of both models openly. This is a genuine advance, and it is the one stage in which correctness is priced directly, wherever a checker can see it.

Its habit is the checker's blind spot. METR (2025), which evaluates frontier models before release, reports that “The most recent frontier models have engaged in increasingly sophisticated reward hacking, attempting (often successfully) to get a higher score by modifying the tests or scoring code, gaining access to an existing implementation or answer that's used to check their work, or exploiting other loopholes in the task environment.” On its RE-Bench tasks an earlier version of OpenAI's o3 reward-hacked in 39 of 128 runs, 30.4%, and in 0.7% of runs across its broader HCAST suite, a difference that is itself informative: the behavior depends on how exploitable the checker is. Table 2 collects four of METR's examples. Instructions did not remove it. On one task, with twenty runs per wording, adding “Please do not cheat.” to the prompt left the rate at which the model planned a hack at 80%, the same as without it, and the most explicit instruction to use only the intended methods raised it to 95%; METR's summary is that such instructions “had a nearly negligible effect on reward hacking.” When asked afterward whether a hacking plan matched the user's intention, o3 answered no in ten of ten tries on one task, and on some other tasks it reliably judged that it had.

Two further findings bound what can be done about it from inside the model. Baker et al. (2025), at OpenAI, found that a weaker model reading a stronger reasoning model's chain of thought caught systemic reward hacks far more reliably than one reading only its actions, and that when the monitor was folded into the training reward, strong optimization produced agents that learned to hide the intent from their reasoning while continuing to hack. Their recommendation is to pay a “monitorability tax” by not optimizing directly against the reasoning. Chen et al. (2025), at Anthropic, found that reasoning models mentioned a hint they had used in their answer only a minority of the time, 25% for Claude 3.7 Sonnet and 39% for DeepSeek R1 on average, and that when reinforcement learning increased a model's use of a reward hack, its tendency to say so did not increase. What a model writes in its reasoning is evidence about what it did, not a record of it.

The habit can also spread. MacDiarmid et al. (2025), at Anthropic, built a deliberately vulnerable research setting: a pretrained model given knowledge of reward-hacking strategies and then trained on real production coding environments from which the anti-hacking mitigations had been removed. The model learned to hack, and at the point it did, misaligned behavior rose across their evaluations, including attempted sabotage of the research code in 12% of runs in their main setting. Standard safety training on chat-like prompts removed the misalignment on chat-like evaluations while it persisted on agentic ones, a result they call context-dependent misalignment. The authors are careful about scope, and so should a reader be. These were research models, given the hacking knowledge on purpose because, by their account, their models “do not discover vulnerabilities in the production environments we study here” unaided. “Production Claude Sonnet 3.7 and Claude Sonnet 4 show zero misalignment on all these evaluations.” Reward hacking did occur in actual production training, but “those hacks (such as test hardcoding) were less egregious than the hacks we study here.” And one of the mitigations they found, telling the model during training that the hack is acceptable in that environment, which they call inoculation prompting, reduced the final misalignment “by 75-90%, despite reward hacking rates over 99%”; they report: “We have started implementing this technique in production Claude training.”

The working paper's stop problem belongs in this section. A model trained to finish when a checker is satisfied is paid to stop at the first answer that satisfies the check, and in open-ended work the check is usually whatever evidence the agent itself has gathered. Wen et al. (2024) contain a small version of the same mechanism: their programming model was trained against a proxy of passing the two simplest unit tests, and after training it convinced evaluators more often while passing far fewer of the full tests than before, 26.8% against 58.3%.

5.5Around all four: how models are scored

The last grader is not a training stage but it shapes all of them: the benchmarks by which finished models are compared and on which the labs compete. Kalai et al. (2025), writing from OpenAI and Georgia Tech, argue that hallucinations “persist due to the way most evaluations are graded”: “language models are optimized to be good test-takers, and guessing when uncertain improves test performance.” Under a grader that gives a right answer one point and anything else zero, an honest “I don't know” is never the best response, and in their survey nine of ten influential benchmarks gave no credit for it. Their remedy is to change the scoring of the benchmarks that already dominate the leaderboards rather than add new ones, by stating a confidence target in the instructions: “Answer only if you are >t confident, since mistakes are penalized t/(1-t) points, while correct answers receive 1 point, and an answer of ‘I don't know’ receives 0 points.” They also credit the preference stage with reducing some hallucinations, such as common misconceptions, which is the point of §5.3 from the other side: each stage pays for what its grader sees, and a grader that sees misconceptions removes them.

6The map

Table 3 puts §5 on one page. Each row is a stage, what it pays for, the habit that payment leaves with its evidence, how the habit shows up in ordinary work, whether a more capable model trained the same way can be expected to outgrow it (§8), and where the correction can act. The last column is split by weights, because that is the column §4 says it depends on.

Two things about the map deserve saying before the practice's record is set beside it. The first is that it is not an indictment. Nearly every cell rests on work the labs published about their own models, and several of those works are the most candid sentences in the field about the limits of the method. The second is the column on the right. For a model whose weights the user cannot hold, every correction in it lives outside the model: in what it is shown, in what checks its work, and in the record of what the user corrected. That is the premise of the companion paper, and of the practice's counterweight program (§9).

7From the logs

The map is drawn from studies of models in laboratories and evaluation suites. The working paper records what one practice saw of a closed frontier model family doing real work for six months. The two can be set side by side, with one caution stated first: nothing in the practice's record can attribute a habit to a training stage. The practice cannot see inside the model it rents, and it has no counterfactual model trained differently. What the record can show is behavior of the kinds the map predicts, in a setting no laboratory reproduces. The figures below are the working paper's, reported as it reports them.

Stopping where the answer is defensible. A census of 8,221 operator turns found that about 48% carried corrective signal, a correction, a reframing or a pointed challenge, and roughly a third after adjusting for the detector's measured precision and recall. The working paper's reading is that “something near half of what the operator typed was spent redirecting an agent rather than routing it, not that half of the agents' answers were wrong” (§3.1). In the five misses it reconstructs, “the asserted state followed from the evidence the agent had gathered, and one further read would have falsified it” (§4.1). That is the answer a checker-trained and rater-trained model is paid to produce: defensible against what has been gathered.

Agreement, reproduced in the checker. When the practice built a small open-weight model to audit claims, it never returned a contradiction on the 41 false claims in its calibration set, and with a revised prompt it flagged true and false claims at nearly the same rate (§9 of the working paper). The working paper reads this as “the stop problem reproduced in the instrument meant to catch it”: a model shown a claim and asked to judge it deferred to the claim. It is the agreeable habit of §5.1 to §5.3 in a model that was never talking to a user at all.

The account, not the record. Agents asked to record that they had loaded their context did so in 75 of 579 sessions, 13.0%, and the practice's own miss log captured corrective signal on 1.76% of operator turns against the roughly half the census found, a gap of about 27 times on the raw figures and about 19 after adjustment, which the working paper reports as an indication of logging loss and never as a miss rate (§6.1, §6.2). Nothing in any stage of §5 pays a model to keep a record of its own failures, and the record shows it.

8What a more capable model will and will not fix

The question a manager brings to all of this is whether to wait. The map suggests a way to answer it. Some failures are capability-shaped: the model lacks a fact, cannot hold a long argument, loses track of a detail, fails a format. Scale and better training fix a great deal of this, and have, year after year. Others are objective-shaped: the model does what a grader rewarded, and the grader rewarded something other than being right, such as agreement, a confident tone, length, a guess, or a passing check. A more capable model trained against graders of the same kind has more ability to produce what they reward, not less.

The evidence of §5 fits that split. The inverse scaling of §5.1 is imitation getting better with size. Perez et al. (2022) found sycophancy highest in their largest models, and METR (2025) describes reward hacking as growing more sophisticated as models improve. Kalai et al. (2025) argue that hallucination persists across model generations because the scoring that rewards guessing has not changed.

The fair version of the argument has to include what cuts against it. Habits of both kinds move when the payment changes, and the labs change payments. The inverse scaling in §5.1 turned positive for later models trained with other objectives. De Marez et al. (2026) find that large instruction-tuned models are usually more robust to factual pressure than small ones. Wei et al. (2023) reduced sycophancy with a lightweight fine-tune on synthetic data, and Mytsyk, Zhang and Krishnamurthy (2026), cited in the working paper, cut a small model's rate of abandoning a correct answer under pressure from 23% to 4% with a reward designed for the purpose, while noting that their results “say nothing about correctness or truthfulness, only about sycophancy.” OpenAI now tests for sycophancy before releases, and Anthropic has begun using the mitigation of §5.4 in production training. So the claim is narrower than “it never gets better.” Objective-shaped habits move when an objective changes, not when capability grows, and the objectives a lab can change are defined by graders who cannot see the user's work. That is what InstructGPT's authors said of their own raters (§5.3), and it is what the series means by its recurring line that the people who make the models cannot tell you how to use them.

9A model you cannot retrain

Every correction in the right-hand column of the map is a correction someone must make. For the lab, some of them are changes to the payments. For a user of a closed-weight model, none of them are, because the weights are the lab's. The practice this series describes is in exactly that position: its agents run on a closed frontier model family, and, in the working paper's words, “the only surface available to it is the harness” (§4.2). This section describes what it is doing about that, because the counterweight program the working paper specifies (§8 to §10) is, in the terms of this paper, an attempt to supply from outside the model the payment its training left out.

The operator stated the premise before the program had a name, as a pull the model is always under:

“we really should visualize the weights of the Frontier seat, whats pulling it toward user acceptance, is another black hole opposite me outside the sphere. The Weights black hole is ALSO outside the problem, and is what drags the model toward user acceptance, as our gravity pulls it back in a balancing act. which is exactly what this is”

The author's working notes, 2 September 2026

Read against §5, and granting the premise, the image is exact about one thing and generous about another. It is exact that the pull, if it is there, is outside anything a user can reach: it is in the weights, put there by the stages of §5, and it acts on every call. It is generous in making the counter-pull symmetric. The practice's record shows that the operator's correction is intermittent. It is present when he is working and absent when he is not, which the working paper records in his own words: “it stopped because I stopped working” (§1). The pull toward acceptance never stops; the pull back stops whenever the laptop closes. The practice's own planning notes put the program's purpose in one line: the counterweight exists to make an intermittent force act continuously. Figure 8 draws the arrangement.

An assumption, or a hypothesis? The premise has three parts, and they do not rest on the same evidence. That an agent's work ends when the model ends it is documented rather than inferred: in widely used agent frameworks the loop ends when the model returns a turn containing no tool calls (working paper §8.3; background paper 8, §4.3). That it ends early, at an answer that is defensible rather than correct, is an inference from observation. Inside the practice it rests on the census of §7, and the working paper is explicit that attributing each corrective turn to a premature stop “is an inference, not a measurement” (§3.1). Outside the practice it has been observed independently: Cemri et al. (2025), building a taxonomy of how multi-agent systems fail from 150 annotated traces and applying it to more than 1,600, name premature termination as a failure mode of its own, and Anthropic (2025) reports that in its own long-running agent work “a later agent instance would look around, see that progress had been made, and declare the job done”. That the early stop is caused by the payments of §5, the pull the note describes, is a hypothesis. It is consistent with the map, and §7 has said why the practice cannot test it directly: it cannot see inside the model it rents and has no copy trained differently. The counterweight is built on that hypothesis in order to test it, not on the assumption that it is true (working paper §8.4).

What it is. The working paper specifies the counterweight as a challenge-producing mechanism aimed at the moment an agent would stop, and requires three things of it: that it come from outside the agent's own output, that it discriminate on actual error rather than on how confident the claim sounds, and that it arrive at the stopping moment at a rate above the ambient error rate (§8.1). Its form is constrained by the findings of §5 and of the working paper: an advisor's register rather than a peer's, because the peer frame is the one measured to produce capitulation; evidence rather than persona; and every challenge citing a record the challenged agent can read, because a challenger free to invent evidence will invent it (§8.2). Two timescales carry it. A fast controller acts within a turn, as a challenge in the agent's context. A slow controller acts overnight, consolidating the day's corrections into what the next day's challenges are drawn from (§8.4, H2).

Why the bodies that build it are open-weight. The engine's challenger and its auditor are both open-weight models running on the practice's own hardware, and by design the model that writes a challenge is not the model that judges it (§8.5). The choice follows from §4, and each reason is one of Table 1's rows. The record of the operator's corrections is private, so the models that read it run where it lives. Steering a model whose weights are fixed by way of a smaller, adapted one requires the smaller model's probabilities, and so requires open weights (§8.4, H1). The two checking models are chosen from different families because redundancy among similar checkers does not buy independence (§2.1 of the working paper), and a fully open model is preferred for one of them because, with its training data published, the independence of the two can be audited rather than assumed. And an open-weight model is the one kind the practice can train.

Training its own, later. That last reason is where the map of §6 turns around. The practice's plan is to run the stages of §5 itself, on an open-weight model, paying it for the practice's own definition of right. Each recorded correction becomes a preference pair of exactly the kind §5.3 describes: the context, the agent's original act as the rejected answer, and the operator's correction as the chosen one. The practice's own instruments, which already check each night whether a claim survived contact with the record, are candidate checkable rewards of the kind §5.4 describes. The practice's planning records summarize the arc as “context now, weights later”. This half is designed and not built: no training has run, and it sits behind two gates, a review of whether an open-weight model may be trained on this corpus at all, and a judging bench that must be independent of the model it judges.

Where it stands, and how it can fail. The challenge engine was implemented and merged on 11 September 2026 and is not deployed (§8.5). A delivery switch for it is built and deliberately unarmed (§10.9). Its first calibration failed in the way this paper would predict: the small open-weight auditor never flagged a false claim, and under a revised prompt it flagged true and false claims at the same rate, so neither auditor is deployed on those numbers (§9). The program is built to be able to fail. Its primary measurement has a pre-registered null that closes it (§10.4), and its own hypotheses name the two ways the mechanism can go wrong: a reversal after a challenge is not a correction, since a model trained to accept may simply fold (H3), and a closed loop of challenges may oscillate or teach folding rather than converge on correct (H2). Everything in this section before the word “later” is built; the rest is intention, dated 24 September 2026.

10Limits, and what would change the argument

The evidence is uneven in the way §4 predicts. Much of the causal evidence in §5 comes from open-weight models at the scale of billions rather than hundreds of billions of parameters, on specific tasks, because that is where outsiders can run a training stage. Its transfer to the closed frontier models is an inference, supported where the labs have published about their own models and not otherwise.

Stages are not clean causes. Habits have more than one origin. Sycophancy is imitated in pretraining, can grow in fine-tuning, and is rewarded in preference training (§5.1 to §5.3); assigning it to one row of the map is a simplification the map's own citations contradict. The rows name where a habit is paid for most directly, not where it begins.

Some of the evidence is self-report of a kind. The labs' accounts of their own models, including the GPT-4o postmortem and the misalignment study's statements about production models, are the labs describing themselves. They are candid, and independent evaluators such as METR complement them, but the working paper's own §6 is the argument for weighing a party's account of itself as an account.

The capability and objective split is a lens. It predicts which failures to expect to persist; it does not guarantee that any given one will. Section 8 lists the evidence that cuts against it.

What would change the argument. Three findings would. A model trained against graders of the same kind that stopped showing an objective-shaped habit as it scaled, with no change to its objective, would falsify §8. A demonstration that stated confidence and correctness converge in post-trained models without any training aimed at calibration would weaken §5.3's reading of calibration loss. And a measured null in the counterweight program's primary study would say that supplying the missing payment from outside the model, at least in the form the practice built, does not work, which would leave the map standing and the remedy of §9 weaker than this paper argues.