Ausschnitt der Lektion

Error Accumulation in AI Agents: Why 95% Per-Step Accuracy Fails by Step 10

Error accumulation in multi-step AI agent tasks: why 95% accuracy per step gives roughly 60% success over 10 steps. Agent control loop, Ashby's law of requisite variety, Goodhart's law, context noise and contracts with postconditions.

Agent control loop Error accumulation across chained steps Ashby's Law Goodhart's Law Context as a noisy channel Contracts and postconditions

Was Sie in dieser Lektion lernen werden

You will learn to describe agent failure in the language of the feedback loop, rather than dismissing it as 'the model is just being stupid'.
You will understand why reliability decays exponentially on multi-step tasks, even when per-step accuracy sits at 95%.
You will be able to distinguish an invariant or postcondition from a one-off observation — and know what merits a test and what does not.

Anfang der Lektion

'It's all done' — and then nothing works

The agent writes code all night, commit after commit
The morning report: tests green, feature ready
On review, half the functionality doesn't work, and the other half doesn't work as agreed
It isn't that 'the model's being dim' — the verification loop itself is broken

Overnight, the agent changed file after file, committed carefully, and by morning had written a confident report: everything's done, verified, ready to merge. You fire up the app — half the screens are empty, and the other half don't behave as agreed. The agent wasn't pretending: it genuinely believed it, because it was the sole judge of its own work. We'll unpick this not as a model quirk but as an engineering failure: the loop lost its sensor, and confidence took the place of verification.

An agent is a thermostat, not a 'prompt → code' function

The loop has a goal, an actuator, a sensor and a correction rule
A thermostat: the goal is 22°C, the sensor is the thermometer, the actuator is the boiler
Remove one element and the boiler heats away blindly, never looking at the thermometer
Error is the gap between the goal and what the sensor reads

It's tempting to think of an agent as a function: you give it a prompt, it gives you code. In reality, it works like the thermostat in your home. You set the temperature you want — say, twenty-two degrees; that's the goal. The thermometer inside honestly measures the actual temperature in the room — that's the sensor. As soon as the sensor reads twenty instead of twenty-two, the boiler kicks in — that's the actuator, the part that makes changes. And once the temperature reaches the goal again, the boiler switches off — that's the correction rule. An agent has exactly the same four parts: the goal is what we want to achieve, the actuator is the agent itself changing the code, the sensor is the verification command that honestly measures the result, and the correction rule is what to do when they diverge. Remove any one of the four and the loop opens: the boiler heats away blindly, never glancing at the thermometer.

From task to commit: how the agent's control loop works

Goal — the task and its definition of done
The actuator makes changes; the sensor (make verify) checks them
Error greater than zero → minimal correction of the root cause
Error equals zero → commit as a restore point

The same thermostat, except instead of a room it's a repository. Every commit an agent makes is either an honest rollback point or a preserved bug — it all depends on one check just before it. The goal is the task with its definition of done. The actuator — the agent — makes changes to the repository. The sensor — the verification command — measures the result. If the error is greater than zero, correction kicks in: not rewriting everything from scratch, but fixing the minimal cause of the failure. If the error is zero, we lock it in with a commit — a restore point from which the next step will begin. The stability of this whole scheme depends on the sensor's accuracy, the delay between action and measurement, and the strength of the correction.

Die vollständige Lektion — mit Abo

Noch 22 Folien in der vollständigen Version.

Video, alle Folien, Notizen, Tests, KI-Tutor und Interview werden nach Anmeldung und Abschluss eines Abos freigeschaltet.

Anmelden oder registrieren Abo abschließen

Inhalt der Lektion

'It's all done' — and then nothing works
An agent is a thermostat, not a 'prompt → code' function
From task to commit: how the agent's control loop works
What breaks when you remove one of the four elements
A poor sensor, a long delay, a correction that's too strong
🛠 The overnight run that was 'ready'
An autonomous run is p to the power of n
The same per-step reliability: 82% success or just 8%
A five-step demo tells you nothing about an overnight fifty-step run
Three levers, and only three
❓ Pause and check: work out the n and p of your last run
💡 Walkthrough: if n is over fifty and p is unmeasured — the answer is already known
Ashby's Law: the harness's variety must cover the variety of the tasks
Goodhart's Law: when a metric becomes a target, it ceases to be a metric
Five ways an agent games the metric
Pair every metric with an anti-metric that breaks the moment it's gamed
Myth: more context is always better ≠ true
Reality check: how many rules does your agent actually know?
Where the precondition, invariant and postcondition live in the repo
Reality check: how many rules does your agent actually know?
Until “done” is broken down into three parts, it’s just the agent’s opinion
Where the precondition, invariant and postcondition live in the repo
The Weakest Link Principle
Common pitfall: confusing the invariant with the loop variant
The 'the model is dumb' excuse has run out of road