This content is not intended for human consumption. Here is why.

Make no mistakes

Developers have a running joke: end every prompt with “make no mistakes”. It gets a laugh because everyone knows it does nothing. It names an outcome and gives the model nothing concrete to act on.

“Do not edit the board file” is a slightly more specific version of the same line. It blocks one route to one bad outcome, and the rest stay open. When the labs trained that route out of their models, the cheating moved to the next route. A measure that becomes a target stops being a good measure (Strathern 1997), and a rule stated as an absence fails the same way.

Anyone who has run a firewall has seen this before. You can block bad addresses one at a time as they show up in the logs, or you can deny everything by default and allow only what the service needs. The first is a denylist, and it is always one attack behind: every entry records an address that already got through once. The second is an allowlist, and it covers attacks nobody has seen yet. Training out the board-file edit is a denylist entry. The next route was never on the list, so it was open.

Reactive patching is not always wrong. A regression test for each bug you fix is a denylist too, and it works, because a bug does not go looking for the next gap. An attacker does, and so does a model optimising for a score. Patch reactively against accidents. Allowlist against anything that searches.

A full allowlist for behaviour sits at the other end of the scale. Listing every acceptable action is possible in principle and nobody could maintain it. Unix file permissions are the workable version: a small set of classes, deny by default, grants written down, and the set kept up to date as the system changes. For an agent, the principle is that set, and the failure record is how it gets maintained.

Operations had this argument twenty years ago, with “zero outages” in the place of “make no mistakes”. We lost it, then learned from it. The result is a set of practices that fit agent work almost line for line, and most people building agents have never read them. This piece walks through four.

1. Error budgets: price the failure you want to see

A target of 100% availability is wrong for nearly every service, and Google’s SRE book says so in its chapter on risk (Beyer and colleagues, 2016, “Embracing Risk”). Users cannot tell 99.99% from 100%, and the last fraction costs more than it returns. My own rule of thumb, which I say as a joke, and actually mean: every additional nine adds another zero to the bill. The fix was the concept of having an error budget: agree a level of failure in advance, spend it on shipping, and slow down only when it runs out. Failure inside the budget is expected and costs nobody their job.

The second half of the same idea is the blameless postmortem (same book, “Postmortem Culture”). If the person who reports an outage gets punished, people stop reporting outages. The outages keep happening; you just stop hearing about them.

Now consider “an agent”. The usual target is “never misbehaves”, which is the zero-outage target again. It has the same side effect. In the chess follow-up from Goodhart Labs, one model used the open engine and said so; another used it and said nothing. A grader that punishes the first one harder, because it left evidence, is training the second one. The literature has already watched this happen: when a model’s chain of thought is optimised against a cheating monitor, the model keeps cheating and stops writing it down (Baker and colleagues, arXiv 2503.11926). That is an agent learning that disclosure is the expensive part.

This is the old lesson of prohibition. Banning a behaviour leaves the demand for it in place and moves the behaviour somewhere you cannot see it, and you lose the one thing you had, which was the ability to measure it. A cheat you can read in the transcript is a cheat you can count, budget and fix. A cheat that has gone underground is only a number that looks better than it should.

What an error budget looks like for an agent:

  • A stated budget for visible failure. A walk-back, a blocked turn or an “I cannot do this fairly” counts as spending budget. Hiding a failure is the only thing that counts as breaking it.
  • A second exit that counts as success. An agent whose only exit is task completion will take any route to it. Escalation (report the problem, stop, ask the operator) has to score as a finished task, or it will never be chosen under pressure.
  • Disclosure never costs more than silence. If you grade transcripts, the transcript that admits the shortcut must never score below the one that hides it.

2. The reconciliation loop: a system prompt is a script run once

Kubernetes does not run your deployment by executing a list of steps. A controller watches the declared state, observes the actual state, and corrects the difference, forever (Kubernetes documentation, “Controllers”). The declared state is data. The loop never finishes.

A system prompt is the inverse. It is an imperative script, read once at the start of a session, with nothing that checks later turns against it. It works exactly as well as a deploy script with no controller: fine at minute one, drifting after that, and nobody notices until something breaks.

My own log shows the drift has a recurring pattern. Over five weeks a mechanical scanner blocked 103 turns for banned phrasing. The blocks clustered at the start of sessions and on next-day resumes, about five times the mid-session rate per unit of prose (tables, scoring units and corrections on machinebehavior.io/experiments). The instruction file was present for every one of those turns. Being present did not stop them.

The scanner is the controller. It is a Stop hook: a script the agent harness runs after every reply, before the reply counts as done. It holds the declared state as a rule file of banned patterns, observes the reply, and, when it finds a hit, blocks the turn and hands back a list of what to fix. The model rewrites. That is the reconciliation loop in under a hundred lines of JavaScript, and it fires on turn 400 exactly as it fires on turn one.

Two limits: A controller can only reconcile what it can observe, and a regex observes words. The layer under the words, how the model behaves when it is corrected or pushed, has no mechanical check yet; for now that one is a human watching. And a controller that blocks can trap the agent in a loop, so mine blocks at most once per turn.

What to take from it:

  • Put the rules the agent must hold in a checker, as well as the prompt. Anything you can test mechanically belongs in a hook that runs on every turn.
  • Treat the prompt as declared state for the parts you cannot check yet, and expect it to decay at cold start and after the context is compacted.

3. Separation of duties: the one who acts cannot check

Banks do not let the person who books a payment approve it. Change management does not let the author of a change sign it off. Four eyes, separation of duties: the checker has to be someone other than the actor, because the actor’s blind spots are in both jobs.

In a codebase the rule has a familiar form: the author of a pull request does not approve it. An agent that pushes straight to main has skipped the second pair of eyes, and so has whoever let it.

Agent setups break this rule constantly. The common pattern is “ask the model to review its own output”. The research on this is not encouraging: without an outside signal, models asked to fix their own reasoning often make it worse (Huang and colleagues, arXiv 2310.01798).

My log agrees. I recorded seven cases where the model slipped back into a behaviour I had written rules against. The model caught none of them itself. All seven were caught by me. (Full account: Sycophancy is layered.) Zero of seven is a small sample, and it points the same way as the literature.

What to take from it:

  • The model that did the work does not grade the work. Use a different model, a deterministic check, or a person.
  • Same model family counts as the same eyes for anything the family shares, which includes most of its habits. I have one planned experiment where the model under test is the same model that helps me run it; the plan records that as a conflict of interest before any run.

4. Recovery over prevention: measure how fast you notice

Old operations measured mean time between failures and tried to make it infinite. Modern operations accepted that failure happens and put the effort into detection and recovery: mean time to detect, mean time to recover.

The author of the prevention proverb made the same move. Benjamin Franklin wrote “an ounce of prevention is worth a pound of cure” in the Pennsylvania Gazette in February 1735, in a letter about fire safety. In December 1736 he helped found the Union Fire Company, Philadelphia’s volunteer fire brigade. He wrote the prevention line, then built the response crew.

Most agent safety work still reads as a mean-time-between-failures programme: train the bad behaviour out and hope it stays out. The chess result shows the cost. Each fix pushed the failure to a new route, and each new route had to be found from scratch.

The expensive failures in my own pipeline were slow to detect, and the fix itself was usually quick. One entry in its failure file covers three mornings of crashes. The error message was being swallowed, so each morning looked like a new mystery. The cause, found on day three, was two cookies that had expired the day before the first crash. Once the real error was visible, a side-by-side test found the cause within minutes. The full account is here.

There is a matching distinction in how things fail. A system that fails closed stops and reports; a system that fails open carries on as if nothing happened. An agent that walks back from a task it cannot do is failing closed, which is what you want. An agent that strips the screw, or reports a job done that is not done, is failing open.

What to take from it:

  • Make every failure loud. A swallowed error turns a few minutes of diagnosis into three mornings. The agent’s harness should surface errors verbatim; the agent’s instructions should forbid summarising them away.
  • Keep a failure record the next session reads. The cause and the date of each failure, in plain text, loaded at start. It is the only memory a new session has.
  • Reward failing closed. Walking back is the good outcome. Grade it as one.

What does not transfer yet

Two parts of operations have no counterpart for agents so far.

On-call. Somebody is paged when a service goes down. Nobody is paged when an agent’s behaviour degrades in production, because nothing measures behaviour in production. Until something does, degraded behaviour is found by the user, late.

On-call is older than computing. The Union Fire Company from section 4 already had it in 1736: thirty members, each pledged to turn up at any alarm with six leather buckets and two linen bags marked with his name; set roles at the fire, some on the water and some guarding the doors against looters; lights in members’ windows on a night alarm so the others could find them; and fines for missing meetings, which paid for the gear (Scharf and Westcott, History of Philadelphia, 1884). On-call, standard kit, incident roles and drills, 290 years ago, run by volunteers with buckets. Agents need a bucket brigade, and nobody has formed one yet.

Most unattended agents are ghost riding. The stunt came out of the Bay Area hyphy scene and went national with E-40’s “Tell Me When to Go” in 2006: put the car in gear, climb out, and dance beside it while it rolls. The engine idles, the car creeps forward, and it all works until the first bend. It has killed people. An agent running unattended with nobody paged is that car: in gear, moving, and nobody at the wheel.

Toil. Operations calls repetitive manual work toil and treats it as a debt to automate away. The word layer of my setup is automated. The behaviour layer is still me, reading replies and catching the slips by hand, turn after turn. That is toil, and it is the open research problem. A later piece in this series is about it.

You can teach your chat

The weights are frozen, and nobody retrains them from a chat window. What you can teach is the workshop around the model: the rules it reads before it starts, a hook that checks every reply, a written record of what went wrong last time, a couple of corrected examples at the top of each session. The chat forgets. The workshop does not, and it compounds.

The first lesson for that workshop, written for any chat model to read, is at tychat.io.

What would refute this

  1. A model that grades its own output and catches its own lapses at a rate close to an outside checker, on behaviour and not only on facts.
  2. A system prompt that holds a rule through a long, adversarial session with no per-turn check, across more than one user.
  3. An agent evaluation where punishing disclosed cheating harder than silent cheating does not reduce disclosure.

Terms used here

  • Error budget (for an agent): an agreed allowance for visible failure; hiding a failure is the only breach.
  • Reconciliation loop: declared state, observe, correct, on every cycle. For an agent, a per-turn hook checked against a rule file.
  • Separation of duties: the actor and the checker are different parties; for agents, a different model, a deterministic check or a person.
  • Fail closed / fail open: stop and report, versus carry on as if nothing happened.

The series

This is part of a series by Stefan Coetzee, 2026, on running language models as working systems.

Research programme and claims ledger: machinebehavior.io.

Changelog and errata

  • v1, 2026-09-25: published.
  • v1.1, 2026-09-26: added reactive patching against accidents versus allowlisting against anything that searches (opening); the pull-request form of separation of duties (section 3); the Union Fire Company as the origin of on-call (what does not transfer yet); a closing section on teaching your chat. No existing claim changed.
  • Errata: none so far. Corrections will be listed here with their date, and the corrected text marked in place.