Research notes1 November 2026

Checked Is Not Reviewed: The Checks Passed Them, a Separate Reader Found What the Checks Missed

Berg Atkinson · 1 November 2026 · The paper: Checked Is Not Reviewed

Summary card for the research note Checked Is Not Reviewed. Fourteen papers passed their AI author's checks, then an AI panel read them. 366 problems reported, 41 of them serious, by fourteen AI reviewers on 7 October. 0 of 4 of the author's checks compared a quote with its source, so a misquotation passed them. Auditors who had not written the fixes changed 98 of 548. A same-family AI review, commissioned by the author; counts are reports. The fix is a check that reads the source.

How do you know your AI's work is right? The usual answer: it passed its checks. Mine passed all of them 😅 My AI wrote a series of fourteen papers and ran its own automated checks on every one before it went live. Then a separate panel of AI reviewers read thirteen of them and reported 366 problems, 41 of them serious.

Here's the gap. All four of the checks compared a file with another file, or a citation with its catalog record. None of them compared a quote with the source it quotes. So a misquote went straight through the two checks that touched it.

The source says "Our results say nothing about correctness." The papers added six words that change what it means. My AI had "confirmed" the quote with a web tool that hands back a summary of the page, not the page...and the summary supplied the extra words. The paper re-checks the source itself: the real sentence is on page 19, and the added words aren't anywhere in it.

Side by side, the checks | the review:

  • What got compared: files against files, citations against catalog records | the papers against what they claim
  • Quote against its source: 0 of 4 checks | the misquote caught in 2 of the 3 reviewed papers that carried it
  • Then the panel's own fixes: 548 written | 98 changed or undone (17.9%) by auditors who hadn't written them

So a second reader found what the writer missed, both times. Was it the model? Not shown. Papers my AI wrote before and after it switched to a newer model drew about the same number of problems per paper (25.0 and 26.2). The paper's working position is that the job and who did it mattered more than the model, and it says straight out that's untested.

The fix is a check that reads the source: every quote checked against the source's raw text, and a source only counts as read if a full copy is saved. Those are 2 of the review's 5 proposed fixes, the 2 that can actually stop a paper. As of 30 October, neither is in the series' checks. And yes, one of those fourteen papers is titled "Never let your AI grade its own homework" 🙄

Grain of salt: the reviewers are the same model family as the author, and I commissioned them, so this is NOT independent review. 36 of the 41 serious problems were never re-checked, and each paper had one reviewer. Read the counts as reviewer reports.

A check that passes isn't a review. Somebody has to read the source 🤷‍♂️

https://wolfberg.ai/papers/checked-is-not-reviewed/

Sources

  1. Mytsyk, Zhang and Krishnamurthy: Mitigating LLM sycophancy with RL-based fine-tuning (arXiv:2608.25267, 26 Aug 2026) arxiv.org
  2. Yang et al.: Can We Believe AI Peer Review? (arXiv:2610.19339, 14 Oct 2026) arxiv.org
  3. Awuni et al.: Who Judges Matters (arXiv:2609.17857, 15 Sep 2026) arxiv.org
  4. Verma: Is this Citation on Point? (arXiv:2608.12571, 12 Aug 2026) arxiv.org
Read this before you believe any of it. Keystone runs in production in its own AWS account. The rest of what you see here is a running prototype, not an accredited or certified system. The register, the war room, and the brain are receipts, not revenue. Every claim on this site links to something you can check. If one doesn't, tell us, and we'll fix the page.