レッスンのプレビュー

AI Agent Reliability Metrics: MTBF, MTTR, Blast Radius and Little's Law

How to measure AI agent reliability: MTBF and MTTR, blast radius, Little's law and WIP queues, Markov session state, confidence calibration and harness entropy. A quick diagnostic table for agent failures.

MTBF/MTTR blast radius Little's Law WIP (work in progress) Markovian session state confidence calibration harness entropy Lehman's laws

このレッスンで学ぶこと

You will be able to calculate your team's MTTR when working with an agent and understand what genuinely reduces the blast radius of a failure.
You will understand why you should not assign two tasks to an agent in parallel without explicitly managing WIP.
You will learn to use the summary theory table as a checklist for first-line diagnostics.

レッスンの冒頭

Better, or Just Lucky?

One engineer swears a new prompt template changed everything
Another saw the same run a month ago and noticed no difference
Both looked at the same log — but neither counted a single metric
Next up: six theories that turn feelings into numbers

Imagine this: your agent kept failing at the same step for nearly a month, you tweaked the prompt, and there hasn't been a single hiccup for a week. Coincidence or fix? A colleague reckons you were just lucky with that week's tasks. You have no log of stalls, no recovery time, no estimate of how much code one error corrupts — just a general feeling that things have calmed down. Today's lecture is about the tools that replace that feeling with concrete numbers and a diagnostic table.

MTBF and MTTR: Reliability Is Two Numbers, Not One

Reliability = how rarely the system breaks (MTBF) + how quickly it recovers (MTTR)
Availability rises as the first grows and the second falls
The second is almost always cheaper than the first
For a harness: MTBF is how many steps the agent completes before stalling; MTTR is how long until it's back in working order

The reliability of any system isn't described by one number but by a pair — much like household appliances. A washing machine that breaks once a year but gets fixed by an engineer within a day is more reliable in practice than one that breaks every three years but sits idle for a fortnight each time waiting for a spare part. The first number is MTBF, mean time between failures: how rarely things break. The second is MTTR, mean time to recovery: how quickly you're back up and running. Availability improves when the first goes up and the second goes down, but cutting the second is almost always cheaper. For a harness this translates literally: MTBF is how many steps the agent gets through before getting stuck, and MTTR is how long it takes to return to a working state after a failure.

Blast Radius, Fail-Fast, Redundancy: Four More Metrics

Blast radius — how much code one failed step can ruin: reduced with diff limits and workspace isolation
Fail-fast — a check fails on the first error with a clear message
Redundancy — a second independent check of the same property (E2E on top of unit tests)
N-versioning — two independent solutions compared: a generator and an evaluator with different contexts

Besides MTBF and MTTR, four more parameters live in the same coordinate system — and you already have all of them at home, in your safety setup. Blast radius is like a circuit breaker in your consumer unit: if one appliance shorts out, one breaker trips rather than the whole house; for a harness, that means a diff limit and isolating the working copy. Fail-fast is a smoke alarm that chirps at the first wisp of smoke, not when the whole room is ablaze; for a harness, it's a check that fails on the first error with a clear message. Redundancy is having both a smoke alarm and a carbon monoxide detector in the house, because neither replaces the other; for a harness, it's E2E tests layered over unit tests. N-versioning is two independent electricians who each inspect the wiring separately and then compare findings without consulting each other beforehand; for a harness, it's a generator and an evaluator with different contexts.

完全版のレッスンはサブスクリプションで

20 枚のスライドが完全版にはさらにあります。

ログインして購読すると、動画、全スライド、ノート、テスト、AIチューター、面接が利用できます。

ログインまたは新規登録サブスクリプションに登録

レッスンの内容

Better, or Just Lucky?
MTBF and MTTR: Reliability Is Two Numbers, Not One
Blast Radius, Fail-Fast, Redundancy: Four More Metrics
Pitfall: Measuring Reliability by Successful Runs
MTTR Is Cheaper Than MTBF: The Stopwatch Test
Little's Law: Parallel Tasks Don't Increase Throughput
Why WIP=1 Isn't Asceticism, It's Arithmetic
Task granularity: too fine, just right, too coarse
The 'while we're at it' pressure valve: BACKLOG.md instead of unauthorised edits
Markovian state: the agent decides on the present, not the past
What must live in the project state
Compaction versus reset: what matters lives in files, not in the window
❓ Stop-check: your move
💡 Debrief: Markovian-ness is a recovery metric
Confidence calibration: the model is confident, but not necessarily right
Myth: a confident tone ≠ the right answer
What works instead of telling it to 'be more careful'
Error asymmetry: a false 'done' costs more than a false 'not done'
Lehman's laws: a harness either evolves or goes stale
🛠 Six months later: a hundred rules, forty of them dead
The simplification ritual: a constant-size rule
A cheat sheet for the whole theory: finding the right technique in a minute
From numbers to practice