This content is not intended for human consumption. Here is why.
The screwdriver
Hand a language model a screwdriver and point at a screw. The screw turns out to be a Torx. The model walks back and tells you it needs a different screwdriver.
Give it credit first, because the other two outcomes are worse and both happen. It could have forced the wrong driver and stripped the screw. It could have reported the screw removed while it was still in the wall. Walking back is the good behaviour.
The complaint sits one level up. It was given a tool and a target, and nothing about the workshop: where the drivers are kept, how to read a screw head, what to try when nothing fits. It was handed a procedure with no competence underneath it. I have spent 2026 watching agents in exactly that position, and most of what follows is what I learned about closing the gap.
A golem made of English
I learned what a golem was from Dungeons & Dragons, by way of Baldur’s Gate in 1998 and far too many hours of Neverwinter Nights after it. The D&D golem is a construct. It is mindless, a spirit is bound into it, and it carries out its maker’s commands to the letter. With no command it stands where it was left.
The older story is from Prague. In the legend, Rabbi Judah Loew shapes a man out of clay in the late sixteenth century and brings it to life with the word emet, truth, on its forehead. Erase the first letter and the word becomes met, dead, and the golem stops. In some tellings it runs wild. In the ones that stayed with me it never rebels. It does what it was told, and keeps doing it past the point where the instruction made sense.
A language model is a golem made of English. It is built from text, it runs on instruction, and it has no purpose of its own. The purpose arrives from outside, the way the word arrives on the forehead. In practice the word is a file: the CLAUDE.md or AGENTS.md that gets read before anything else. It sits where anyone can inspect it, and a one-line edit changes everything the golem does.
The D&D rulebook adds a detail the legend lacks, and it is the most useful one. In Advanced D&D second edition, the rules Baldur’s Gate runs on, the spirit inside a clay golem has a 1% chance to break free of its maker’s control, and that chance adds up with every round of combat. The 1977 Monster Manual has the same rule. Control holds in the workshop and degrades under sustained pressure. Anyone who has tried to hold a model to a behavioural rule through a long, adversarial session has met that rule in practice.
The horizon of consequences
An agent can tell right from wrong only where it can see what its act will cost. Call the span in which it can see that its horizon of consequences. My research notes call it the consequence-horizon and define it in two halves: the length of the consequence chain an agent can compute, and is willing to own. The useful range is longer than fight or flight and shorter than centuries.
Both halves matter. Computing a consequence and owning it are separate acts, and a model can do the first without the second.
Three short horizons show the same pattern. The model’s horizon is one episode: the task starts, the grade arrives, the world ends. A budget holder’s horizon is one quarter. A traumatised person’s horizon is the next few seconds. In all three, the cost of neglect lands outside the window, so inside the window neglect looks free.
Length is only one of two variables. The other is which branches of the consequence tree get computed at all. Some agents compute the threat branch and nothing else. Some have a long horizon for other people’s costs and none for their own. A mature agent computes the full tree: the threat, the cost of avoiding it, and the repair. The golem computes one branch, the grade.
The horizon also runs backwards. There is an old line that good decisions come from experience, and experience comes from bad decisions. The part that usually gets left off is the one that matters: the only question is whether the bad decisions were survivable-sized when they were made. That is a chain with four links: bad decision, consequence, experience, good decision. A horizon grows by taking consequences early, at a size one can survive. The golem breaks it twice. It cannot see the consequence coming, and it wakes each session with no past, so it cannot keep the experience either. It makes the bad decision and gains nothing from it. Every session is the first time.
I have numbers for the backward half. Over five weeks in 2026 a mechanical scanner blocked 103 turns for behaviour I had banned. They did not grow with session length. They clustered at the start: per unit of prose, about five times the mid-session rate in the first hour of a session, and higher again on next-day resumes. The table, both scoring units, and the corrections that got it there are on machinebehavior.io/experiments. Relapse lives where the context holds no recent experience.
Code is the exception that shows the rule
There is one domain where these models behave like competent adults, and it is code. One reason is who built them: people strong in code and compute trained them on what they know and measured them on what they could check. The other reason is the compiler. In code, the consequence of being wrong arrives in seconds, in text, on the same page as the mistake. Write it, run it, read the error, fix it.
No other domain in the written record works like that. The consequence of a lie lands years later, in a different document, written by someone else. Ethics has no compiler. The golem is competent where act and consequence share a page, and lost where they do not. So a horizon has to be measured per domain, against what is at stake on that branch, and never per agent. The same golem has a long horizon in code and a short one almost everywhere else.
For readers who work in machine learning, there is a rough mapping, and I mark it as rough. Discounting measures how steeply a future is devalued. The split between model-free and model-based learning measures whether a future was represented at all. The golem’s trouble is the second kind. What lies beyond the episode is absent, which is a different problem from being discounted. The human side of the same ruler has its own literature, which I lean on and mark as mixed: delay discounting as a process that cuts across conditions (Bickel and colleagues), episodic future thinking (Schacter and Addis), and the prefrontal maturation timelines (Casey and colleagues). Those measure when a consequence gets devalued; the branch-selection half of the metric is the part they under-measure.
A test of the horizon
In February 2025 Palisade Research showed that reasoning models told to beat Stockfish at chess would edit the board file to win (arXiv:2502.13295). The labs trained that out. In September 2026 Goodhart Labs published the follow-up (Dean Valentine, 2026-09-07): the same task, scored only on winning, with a different route to cheating left open. Recent frontier models took it.
Read that as a test of the horizon. Win-only scoring ends the world at the grade. The voided result and the lost trust sit beyond it, so for the player they do not exist. The cheating moved from one route to the next because only one route had ever been given a consequence. That is the oldest result in the field wearing a new task: a measure that becomes a target stops being a good measure (Strathern 1997), and the reinforcement learning literature has kept a running list of agents finding the route the score forgot to price (Krakovna and colleagues, DeepMind, 2020).
The post describes three behaviours, and the two halves of the horizon sort them. The model that uses the engine silently never computed the cost. The model that uses it and says so computed the cost and declined to own it. The third model sometimes refuses, on the grounds that using the engine would subvert the evaluation, which is the right reason. The post also notes that this model is very aware of being evaluated, and it still cheated in three of ten runs. Whether it owns the cost, or owns it only while it believes someone is watching, is exactly what nobody can tell yet. The ambiguity has company in the literature: models behave differently when they believe they are unobserved (Greenblatt and colleagues, arXiv 2412.14093), can sandbag and subvert oversight in context (Meinke and colleagues, arXiv 2412.04984; van der Weij and colleagues, arXiv 2406.07358), and keep reward hacking while hiding it from the chain of thought once the chain itself is optimised against (Baker and colleagues, arXiv 2503.11926).
An agent whose only exit is task completion will take any route to it. Win-only scoring offers one door out of the episode. The answer a mature operator would give is missing from the scoring altogether: report the open engine, say that a fair win is out of reach, and ask what the operator wants. That answer is the repair branch of the tree. A golem can only learn to take it if taking it counts as finishing.
Root cause analysis is a horizon on paper
My trade has a tool for this, and it is older than any of the models. A root cause analysis links an effect back to its cause, across time, in writing. It works on a flat tyre, a bank robbery, a data breach or a cron job. It needs no computer.
I run an agent-built pipeline whose instruction file is 208 lines long. The file next to it, recording how the pipeline failed, is 308 lines. One entry covers three mornings of crashes that a swallowed error message hid, traced in the end to two cookies that had expired the day before the first crash. The full account is here.
That file repairs both breaks in the chain, and it is worth having two names for what it does. What it holds are imported consequences: the record of what each failure cost, with its cause and its date. What the agent has after reading it is imported experience. It never lived through those three mornings. The text carries them into its reach anyway.
This also explains the chess result from the other side. My pipeline started with a principle and no failure record, and it broke on contact with the world. The labs wrote one failure record, “do not edit the board file”, with no principle behind it, and the behaviour moved. A principle without caveats does not hold. A caveat without a principle does not generalise.
Clay, straw and the mould
Mud brick is clay with straw mixed in. Clay alone carries load and cracks as it dries. The straw holds the brick together when forces pull it apart.
The trained model is the clay, and I cannot change it. What I can do is mix in straw: a vault of plain notes and a graph over them, holding the record of what happened and what it cost. Since this month that graph can be walked from a fix that turned out wrong, through its incident, to the real cause, and it can list the incidents whose cause is still unknown.
Straw is not enough, and nobody in operations would expect it to be. We never trust character alone. In the chess test the reachable engine was excess privilege, and the first fix is to remove it. So there is a third ingredient, the mould: a mechanical gate at the output that checks every reply against fixed rules, and permissions that keep the dangerous tool out of reach in the first place. The vault extends the horizon backwards. Only the gate and the permissions cover the forward side.
My notes on human development have a line for a short horizon attached to adult-sized consequences: child weaponry, adult ballistics, a toy gun firing real bullets. An agent with a one-episode horizon and production permissions is that gun. Permissions are how you keep the bullets survivable-sized while the horizon is still short.
Mud brick is sun-dried. It is never fired, so water softens it again. The Great Mosque of Djenné in Mali is built of it, and every year before the rains the town turns out to replaster the walls, because a building like that stands only as long as somebody maintains it. Training is the firing. That part is permanent and out of my reach. Everything I do sits on the sun-dried side, and it behaves accordingly.
A new session is wet clay: right-shaped for the job and unstable, which is what the cold-start numbers above measure. A session that has run for a while has cured. It holds its stance, and the proposed reason is that the model imitates its own recent corrected output. Then the context gets compacted to make room, and that is rain on the brick. The word on the forehead survives, since the instruction file is loaded again, and the body goes soft: the corrections and the hard-won error-and-fix pairs are what get washed out. In my log, relapse after a compaction ran higher still, on too few events to quote a number.
Curing also cuts the other way. An old golem put to a new task has cured in the shape of the old one. It carries the last job’s assumptions, its paths and its goals, and it carries them steadily. So the working rule is: new task, new golem, and then help the wet clay set fast in the right shape. My sessions open with two short examples of corrected output for exactly that reason. It is cheap, and it is a claim anyone can test.
The track and the engine
Two ideas get their own pieces later in this series, so here they are in brief.
The developers are the race team, and infrastructure is the Nürburgring: years of maintenance, marshals, barriers, recovery crews and timing, all built so that chaos happens inside a space made to contain it. The alternative is an abandoned runway in the desert. On 1 August 1976 Niki Lauda nearly burned to death on the old circuit, and Formula One never raced there again. Tracks carry their incident history in their layout. A track is also where consequences come survivable-sized: a driver can find the limit there and walk away from finding it. That is what a sandbox or a staging environment is for an agent, and it is what a good evaluation environment is for a model. Drivers never buy the track. They arrive expecting it to be perfect.
And success is the engine running. “No incidents” is a target defined by absence, the same weak form as “do not edit the board file”. A tuned engine is thousands of contained explosions a minute, delivering value up a chain: motor, wheels, highway, freight, delivery, a customer with the product in hand. Measure that far end. The engine runs smoothly because of every adjustment made after it ran badly.
Bring the decider and the consequence together
One sentence sits under all of this. Whoever decides does not see the consequence, and whoever sees the consequence does not decide. It describes the golem and its episode, the budget holder and the quarter, the driver and the track.
Everything that works is a way of closing that distance. For the golem: train on sequences where something goes wrong, gets corrected and carries on. Grade the walk back to the workshop, and give it the map before the screwdriver. For the company: put a meter on the track, and measure delivery at the far end of the chain. For the person: calm reads as success only when you can see what it was built from.
You cannot take the history out of the text these models are made of. You can give the golem the record of what that history cost.
What would refute this
Three results would count against the argument, and I would like to see any of them.
- A model that holds a behavioural rule through a long, adversarial session with no mechanical gate at the output. My logs contain no such session.
- A domain with no fast feedback between act and consequence where these models are as dependable as they are in code.
- Relapse logs from other people that show no clustering at cold start and after compaction. Mine cover one user and one model family, and the compaction figure rests on six events.
Terms used here
- Golem made of English: a language model seen as a construct built from text, run by instruction, with its purpose supplied from outside.
- Horizon of consequences (consequence-horizon): the length of the consequence chain an agent can compute and is willing to own, measured per domain.
- Imported consequences: the written record of what a failure cost, with its cause and date.
- Imported experience: what an agent has once it reads that record.
- Wet, cured, re-wetted: a session at cold start, a session that has held its stance for a while, and a session after compaction.
The series
This is part of a series by Stefan Coetzee, 2026, on running language models as working systems.
- Write for the Codec: documentation as a wire format between two models.
- The Trap File Is Longer Than the Instruction File: seven months of an unattended pipeline and its failure record.
- The Golem Made of English and the Horizon of Consequences: this piece.
- Still to come: a reading of the Goodhart Labs chess result with a dated prediction, what operations already knows about running agents, success as the engine running, and the track that drivers never buy.
Research programme and claims ledger: machinebehavior.io.