Counterweights: Influencing LLMs

All you can do is influence an LLM. That's the line the whole program hangs off.

I rent the model. I never touch the weights, so it's a box I can't open. What I do control is what enters and leaves the LLM, with a catch on the way in: I govern what it can reach, not what it finds. Out goes the Stop, a refusal, a forced retry, a verifier checking the agent's own claims. A counterweight is a check sitting at one of those edges, OUTSIDE the AI's own writing, firing on a real error and not on how sure the agent sounds.

Why bother? The model stops when its answer is defensible, not when it's right. Rules didn't fix it. My boot page records one act missed seven times in a single day in July. And a check inside the loop that's doing the answering can be talked past. A separate process that writes to a ledger the answering seat can't close is a gate.

Even a gate has a short reach. A refusal holds the answer for one turn. After that the retry can't be blocked, so what I control is a forced retry, not a veto. And the input side is the weak one: a fetched page or a file it reads comes in on the model's own choice. Fourteen more items are planned or proposed.

None of it was designed up front. It came in clumps, a kill check on 17 July to a launch gate and two wiring checks in mid-October. I want the new ones added one at a time and switched on and off by session, so each is its own test. I haven't ruled on the order.

So how much influence did the control buy? Of the fifteen I could read, ten FIRE, meaning a log shows they ran and produced something. Three more do it partly. The Stop hooks refused 272 times across 21 of my 37 sessions, and the verifier with no model refused 27 of the 356 certificates that carried a claim. That's control. Only one has a clean record of catching real errors, me, and even mine leans on my seats. Six others have an indirect one. Nothing shows catches moving upstream because of a mechanism. That's the influence part, and it's empty.

I said "shipped errors" is a stupid metric for us. The one I care about is how many misses get caught before they reach me: about 29% in June, about 63% in October. Sounds great. It isn't proof. The checks went in together, my seats write down who caught what, and a chart with ten go-live lines says "something happened". That's the whole finding.

Read literally, as code outside the model and at the stop, three of my four Stop hooks and I count (the fourth has never fired). Independent of whatever wrote the answer, only me. Plenty of control, no proof of influence yet. If the guards don't change how fast a mistake comes back, the program closes. That's already written down.