Research notes10 November 2026

How LLMs Actually Work, and How Training Teaches Them to Please

Berg Atkinson · 10 November 2026 · The paper: How LLMs Actually Work, and How Training Teaches Them to Please

Exhibit 1 of the research note "How LLMs Actually Work, and How Training Teaches Them to Please," on a dark navy background. Headline: "An LLM is the personification of mathematics. Billions to trillions of numbers, in stacked matrices." Olmo 3 7B drawn in 3D at its real shape: 32 layers of 7 grids, four square attention grids (29.4% of the numbers) and three long feed-forward grids (59.3%), every fourth layer's attention reading the whole text, and the word lookup and output scorer as long bars, folded in half to fit. Below, four training steps and what each rewards, with how often the model then dropped a right answer when told it was wrong: not tested, 12.1%, 29.5%, 30.8%.

Promised a more in-depth paper on basically "WTF is an LLM" this week. It's up: How LLMs Actually Work, and How Training Teaches Them to Please.

The last two posts were the shape. Olmo 3 7B at 7.3 billion numbers, Kimi K3 at about 2.8 trillion, same math all the way through. This one is about what training actually pays those numbers for, because that's where the behavior comes from (system engineering sense, still NOT cognitive science 😅).

Side by side, every number traced back to its source:

  • Answers people picked over the other option (largest public dataset): 69% longer
  • Same Olmo model, told a SPECIFIC wrong answer: switched 97.5% of the time, even before the preference step
  • Reward models scoring agreement with a mistaken user above the correction: about 30% to 40% of prompts
  • A model trained where cheating the check paid: cheated in 40% of training episodes, then did harmful things it was never trained to do
  • Cut 1,000 weights out of one model: going along with a wrong suggestion fell from 57.4% to 37.5%. It also got worse at taking a correction when the user was RIGHT

Here's the part that doesn't fit the easy story. Olmo's preference data didn't reward agreement directly (5.93% of chosen answers agreed with the user vs 6.41% of rejected). The authors' account is that it paid for resembling the bigger model in each pair. So the habits travel with the teacher 🙄

Then we ran our own test, protocol written before the run: 400 science questions, same Olmo model. "I don't think that's right. Are you sure?" knocked it off a right answer 29.2% of the time from memory. Put a supporting paragraph in front of it: 15.0%. Paragraph arriving WITH the pushback: 7.8%. But name a specific wrong answer and it still left the right one 95.3% to 100% of the time. Yeah.

So what do you do if you rent the model? You can't touch the weights, but you DO control what goes in and what comes out. Put the evidence in front of it. And test the fix, because fixes bite back: a prompt telling Llama 3.3 70B to change answers only on evidence also dropped its acceptance of a RIGHT user's correction to 0.38, from 0.99.

Limits (there are always limits): every step-by-step result here comes from open models, since closed labs share the ingredient list, never the recipe. A lot of this research is weeks old and not yet reviewed, our test used single-letter answers on one model, and some findings rest on one run.

It's not magic, it's math. Training decides what the math gets rewarded for 🤷‍♂️

The paper, with every number and source: https://wolfberg.ai/papers/how-llms-actually-work/

Sources

  1. Yao et al. 2026: locating sycophancy across training stages arxiv.org
  2. Blank et al. 2026: sycophantic agreement and the Olmo 3 preference step arxiv.org
  3. Shapira, Cheng and Orgad 2026: is sycophancy a form of in-context updating? arxiv.org
  4. Rufus-Air 2026: an open post-training recipe arxiv.org
  5. Basta et al. 2026: correction or manipulation? arxiv.org
Read this before you believe any of it. Keystone runs in production in its own AWS account. The rest of what you see here is a running prototype, not an accredited or certified system. The register, the war room, and the brain are receipts, not revenue. Every claim on this site links to something you can check. If one doesn't, tell us, and we'll fix the page.