강의 미리보기

AI Agent Reliability Metrics: MTBF, MTTR, Blast Radius and Little's Law

How to measure AI agent reliability: MTBF and MTTR, blast radius, Little's law and WIP queues, Markov session state, confidence calibration and harness entropy. A quick diagnostic table for agent failures.

MTBF/MTTR blast radius Little's Law WIP (work in progress) Markovian session state confidence calibration harness entropy Lehman's laws

이 강의에서 배울 내용

You will be able to calculate your team's MTTR when working with an agent and understand what genuinely reduces the blast radius of a failure.
You will understand why you should not assign two tasks to an agent in parallel without explicitly managing WIP.
You will learn to use the summary theory table as a checklist for first-line diagnostics.

강의 시작 부분

Better, or Just Lucky?

One engineer swears a new prompt template changed everything
Another saw the same run a month ago and noticed no difference
Both looked at the same log — but neither counted a single metric
Next up: six theories that turn feelings into numbers

Imagine this: your agent kept failing at the same step for nearly a month, you tweaked the prompt, and there hasn't been a single hiccup for a week. Coincidence or fix? A colleague reckons you were just lucky with that week's tasks. You have no log of stalls, no recovery time, no estimate of how much code one error corrupts — just a general feeling that things have calmed down. Today's lecture is about the tools that replace that feeling with concrete numbers and a diagnostic table.

MTBF and MTTR: Reliability Is Two Numbers, Not One

Reliability = how rarely the system breaks (MTBF) + how quickly it recovers (MTTR)
Availability rises as the first grows and the second falls
The second is almost always cheaper than the first
For a harness: MTBF is how many steps the agent completes before stalling; MTTR is how long until it's back in working order

The reliability of any system isn't described by one number but by a pair — much like household appliances. A washing machine that breaks once a year but gets fixed by an engineer within a day is more reliable in practice than one that breaks every three years but sits idle for a fortnight each time waiting for a spare part. The first number is MTBF, mean time between failures: how rarely things break. The second is MTTR, mean time to recovery: how quickly you're back up and running. Availability improves when the first goes up and the second goes down, but cutting the second is almost always cheaper. For a harness this translates literally: MTBF is how many steps the agent gets through before getting stuck, and MTTR is how long it takes to return to a working state after a failure.

Blast Radius, Fail-Fast, Redundancy: Four More Metrics

Blast radius — how much code one failed step can ruin: reduced with diff limits and workspace isolation
Fail-fast — a check fails on the first error with a clear message
Redundancy — a second independent check of the same property (E2E on top of unit tests)
N-versioning — two independent solutions compared: a generator and an evaluator with different contexts

Besides MTBF and MTTR, four more parameters live in the same coordinate system — and you already have all of them at home, in your safety setup. Blast radius is like a circuit breaker in your consumer unit: if one appliance shorts out, one breaker trips rather than the whole house; for a harness, that means a diff limit and isolating the working copy. Fail-fast is a smoke alarm that chirps at the first wisp of smoke, not when the whole room is ablaze; for a harness, it's a check that fails on the first error with a clear message. Redundancy is having both a smoke alarm and a carbon monoxide detector in the house, because neither replaces the other; for a harness, it's E2E tests layered over unit tests. N-versioning is two independent electricians who each inspect the wiring separately and then compare findings without consulting each other beforehand; for a harness, it's a generator and an evaluator with different contexts.

전체 강의는 구독 후 이용 가능

전체 버전에는 20개의 슬라이드가 더 있습니다.

로그인하고 구독하면 영상, 모든 슬라이드, 노트, 테스트, AI 튜터, 면접을 이용할 수 있습니다.

로그인 또는 회원가입 구독하기

강의 목차

Better, or Just Lucky?
MTBF and MTTR: Reliability Is Two Numbers, Not One
Blast Radius, Fail-Fast, Redundancy: Four More Metrics
Pitfall: Measuring Reliability by Successful Runs
MTTR Is Cheaper Than MTBF: The Stopwatch Test
Little's Law: Parallel Tasks Don't Increase Throughput
Why WIP=1 Isn't Asceticism, It's Arithmetic
Task granularity: too fine, just right, too coarse
The 'while we're at it' pressure valve: BACKLOG.md instead of unauthorised edits
Markovian state: the agent decides on the present, not the past
What must live in the project state
Compaction versus reset: what matters lives in files, not in the window
❓ Stop-check: your move
💡 Debrief: Markovian-ness is a recovery metric
Confidence calibration: the model is confident, but not necessarily right
Myth: a confident tone ≠ the right answer
What works instead of telling it to 'be more careful'
Error asymmetry: a false 'done' costs more than a false 'not done'
Lehman's laws: a harness either evolves or goes stale
🛠 Six months later: a hundred rules, forty of them dead
The simplification ritual: a constant-size rule
A cheat sheet for the whole theory: finding the right technique in a minute
From numbers to practice