AI Agent Compounding Errors: Why 95% Accuracy Per Step Is Not Enough
Compounding error in an AI agent is the effect where the success probability of a long task equals the product of the success probabilities of its steps. At 95% accuracy per step, a 10-step chain succeeds about 60% of the time and a 20-step chain only about 36%. A harness fights this by checking every step and saving the verified state, so an error never carries over to the next step.
What compounding errors are
An agent solves a task not in one action but in a chain: read a file, change a function, run the tests, fix an import. If each step is correct with probability p, and the steps are independent while one error breaks everything that follows, the chance that the whole chain of n steps succeeds is p to the power of n. This is the compounding error of an AI agent.
Intuition misleads here: 95% per step sounds reliable. But exponentiation eats the margin fast.
| Steps | p = 0.95 | p = 0.99 |
|---|---|---|
| 1 | 95% | 99% |
| 10 | ≈ 60% | ≈ 90% |
| 20 | ≈ 36% | ≈ 82% |
| 50 | ≈ 8% | ≈ 61% |
The model is rough: in practice steps are not fully independent, and some errors are harmless. But it conveys the direction correctly: the longer an autonomous run goes without checks, the less likely the result is correct.
Why long chains collapse
An early error does not stay in place. The agent builds the next changes on a wrong foundation, and by the time something visibly breaks, a dozen layers of code sit on top of the bug. It is worst when the agent judges the result itself: its judgment is biased toward "everything is fine", so a report "done, tests are green" can arrive even when the code is half broken.
What the harness does about it
You can raise p only to a limited degree; a model will never become error-free. The main lever is to stop an error from passing on to the next step. The harness closes the feedback loop.
- A sensor after every step. One check command (for example,
make verifywith the tests and the linter) measures the result honestly instead of relying on the agent's self-assessment. - A recovery point. A step that passes the check is committed. The chain then starts from a verified state, and the error counter effectively resets to zero.
- Minimal correction. When the check is red, the agent fixes the specific cause instead of rewriting everything. This damps oscillation.
- Short tasks. One task at a time, small steps, frequent checks. Instead of twenty blind steps, you get twenty chains of length one.
In Claude Code, the check can be attached to hooks so that it runs automatically rather than by the model's goodwill.
In short
Compounding errors in an AI agent are simple arithmetic: 0.95 to the tenth power is about 0.60, and to the twentieth about 0.36. The cure is not a "smarter model" but a harness: checking every step, committing the verified state, and keeping tasks short. The math of the loop and the principles behind it are covered in detail in a lecture of the course.
FAQ
Does this mean an agent cannot handle a 50-step task?
Without intermediate checks, the odds are indeed poor. With a check and a commit after every step, the task becomes a series of short chains, and overall success is no longer the product of all the steps.
Does a stronger model help?
Yes: raising accuracy from 95% to 99% lifts 20-step success from 36% to 82%. But even 99% over 50 steps gives only about 61%, so checks are still needed.
Why is the agent confident when it is wrong?
Without an external sensor, the agent compares the result with its own expectation, not with reality. Its confidence grows regardless of whether the code is correct.
Can I just ask the agent to check itself?
A request in the prompt does not guarantee execution. It is more reliable to make the check mechanical: a separate command, a hook, or a check in CI.
This topic is covered in depth with practice in the course “Harness Engineering: Engineering Reliability for AI Agents”.
Go to course