Harness Engineering: Engineering Reliability for AI Agents

Designing instructions, state and verification around Claude Code, Codex and Cursor — from diagnostics to autonomous graphs

15 bài giảng 5g 57phút Trong đăng ký

Khóa học này dành cho ai

Developers and technical leads who already work with an AI coding agent (Claude Code, OpenAI Codex, Cursor, Gemini CLI, Aider) but get inconsistent results. Ideal for those who want their agent to run productively for hours and across sessions rather than ten minutes under close supervision, and for anyone building internal tooling or a platform around agents.

Yêu cầu

Git, the command line, and hands-on experience with at least one AI coding agent. No machine learning background is required — this course is not about how the model works under the bonnet, but about how to engineer a dependable environment around it.

Chương trình học

15 bài giảng

Giới thiệu Your Agent Isn't Broken — What's Around It Is

2 Works fast, breaks quietly 30.864g

3 If it isn't the model weights, it's the harness 37.392g

4 The horse and the harness 35.088g

5 Rule No. 1: harness first, model second 32.376g

6 Writing prompts, or building a system? 33.624g

7 By the end of the course you'll hold a working repository, not a set of notes 33.744g

8 Fifteen lectures — not a list of topics, but the story of one agent 37.152g

9 What you'll be able to do after just the first lectures 31.224g

10 What you'll be able to do after just the first lectures 31.752g

11 The question that started it all 31.488g

7 phút

Miễn phí Xem

Why 95% per-step accuracy kills an agent by the tenth step

We lay a formal foundation for harness engineering: the agent as a feedback system (perception → decision → action → evaluation), the mathematics of error accumulation over multi-step runs, Ashby's Law (requisite variety) and Goodhart's Law (a metric that becomes a target ceases to be a metric), context as a noisy channel, and contracts, invariants and postconditions as the only objective criterion of readiness.

Agent control loop Error accumulation across chained steps Ashby's Law Goodhart's Law Context as a noisy channel Contracts and postconditions

26 phút

Sau đăng ký Đăng ký

What Actually Measures Agent Reliability, Rather Than 'Vibes'

We continue building the theoretical foundations: MTBF/MTTR and blast radius as metrics of harness reliability; queues/WIP and Little's Law as a model for why an agent's parallel tasks hold each other up; the Markovian nature of session state (the future depends only on the current state, not on history); confidence calibration (when an agent lies 'confidently'); entropy and Lehman's laws as applied to harness degradation over time; and a consolidation of all this theory into a single table for rapid diagnosis.

MTBF/MTTR blast radius Little's Law WIP (work in progress) Markovian session state confidence calibration harness entropy Lehman's laws

2 Better, or Just Lucky? 28.056g

3 MTBF and MTTR: Reliability Is Two Numbers, Not One 43.296g

4 Blast Radius, Fail-Fast, Redundancy: Four More Metrics 56.136g

5 Pitfall: Measuring Reliability by Successful Runs 34.464g

6 MTTR Is Cheaper Than MTBF: The Stopwatch Test 39g

7 Little's Law: Parallel Tasks Don't Increase Throughput 28.632g

8 Why WIP=1 Isn't Asceticism, It's Arithmetic 53.568g

9 Task granularity: too fine, just right, too coarse 44.328g

10 The 'while we're at it' pressure valve: BACKLOG.md instead of unauthorised edits 34.752g

11 Markovian state: the agent decides on the present, not the past 48.36g

12 What must live in the project state 41.832g

13 Compaction versus reset: what matters lives in files, not in the window 33.552g

14 ❓ Stop-check: your move 30.648g

15 💡 Debrief: Markovian-ness is a recovery metric 30.72g

16 Confidence calibration: the model is confident, but not necessarily right 51.48g

17 Myth: a confident tone ≠ the right answer 36.624g

18 What works instead of telling it to 'be more careful' 41.856g

19 Error asymmetry: a false 'done' costs more than a false 'not done' 35.208g

20 Lehman's laws: a harness either evolves or goes stale 46.512g

21 🛠 Six months later: a hundred rules, forty of them dead 31.896g

22 The simplification ritual: a constant-size rule 35.376g

23 A cheat sheet for the whole theory: finding the right technique in a minute 37.848g

24 From numbers to practice 40.368g

24 phút

Trong đăng ký Đăng ký ngay

The Most Powerful Model on the Market — and It Still Makes Cock-Ups in Production

We unpack why even flagship models routinely fail at seemingly simple tasks in real-world repositories. We introduce the five layers where an agent actually breaks down (task, instructions, state, tools, verification), together with a diagnostic loop that pinpoints the failing layer in a single pass.

the five layers of failure the diagnostic loop model capability versus execution reliability

2 A familiar sight: the agent says 'done', but the tests are red 27.36g

3 🤔 Myth: you just need a more powerful model 32.904g

4 Model capability and result reliability are not the same thing 32.376g

5 🛠 The Anthropic experiment: same model, different harness 33.048g

6 📖 The OpenAI case study: a million lines without a single line from a human 32.664g

7 The five layers where an agent actually breaks 40.632g

8 Layers 1 and 2: is the task clear, and does the agent know the conventions? 49.272g

9 Layers 3 and 4: is the environment reproducible, and is there an honest check? 47.4g

10 Layer 5: does knowledge survive the session, or is it back to square one every time? 45.72g

11 Classic pitfall: fixing the model, not the layer 38.304g

12 The course's running project: a FastAPI payment backend 25.488g

13 The diagnostic loop — the core of the whole methodology 35.28g

14 Applying the loop to our payment backend 36.912g

15 ❓ Pause check: which layer is the problem in 23.64g

16 💡 Breakdown: two layers at once 30.6g

17 Step 1: write your Definition of Done explicitly 30.48g

18 Step 2: set up an AGENTS.md 29.616g

19 Step 3: keep a failure log 23.832g

20 Homework experiment: two runs of one task, one unexpected takeaway 33g

21 The model is rarely to blame — the harness around it usually is 29.688g

22 phút

Trong đăng ký Đăng ký ngay

The five parts that make up an agent which actually works

Five harness subsystems — instructions, tools, environment, state and feedback loops — and how they interact. We examine the typical adoption curve (what to fix first), a method for measuring the value of each component via ablation with a fixed model, and the resulting diagram of flows between subsystems.

the five harness subsystems ablation measurement adoption curve

2 'Just write a longer prompt' 36.912g

3 Myth: a harness = a prompt file 33.384g

4 A kitchen with five stations 27.168g

5 Stations 1–2: instructions and tools 32.472g

6 Stations 3–4: environment and state 33.312g

7 Station 5: feedback — the pass 30.072g

8 Where to start: 5 → 1 → 4 → 3 → 2 46.632g

9 ❓ Quick check: which station is your weakest? 24.624g

10 💡 Diagnosis: how to tell what's broken 32.64g

11 Why different teams all follow the same path 37.176g

12 🛠 What this looks like in real teams 34.464g

13 How to measure a component's value 22.992g

14 A zero drop ≠ 'the component is useless' 49.488g

15 Ablation pitfalls: it's not the place to start 31.896g

16 Constrain, don't micromanage 28.8g

17 A map, not a textbook 28.584g

18 Your harness rots just like code 34.536g

19 Where an agent's work leaks when one link breaks 32.112g

20 Read the diagram as a loop, not a list 31.848g

21 30 minutes on the weakest station of your harness 32.208g

22 Next up: your first AGENTS.md for a payments backend 32.088g

23 A harness is a loop, not a file 32.088g

24 phút

Trong đăng ký Đăng ký ngay

Your Agent Is Making Up the Project Structure Because You Never Left It a Map

The repository is the single source of truth for an agent with no memory between sessions. We examine what to measure when mapping your repository, four principles of a good map, a workable file structure, and ACID-like guarantees for agent state.

the repository as the single source of truth the repository map ACID for agent state

2 📖 The agent reinvents the architecture from scratch — and here's why 34.368g

3 Three sources in total — beyond them, the agent knows nothing 30.384g

4 The OpenAI principle: the repository is the specification 30.504g

5 🤔 Myth: if it's in the repository, the agent will see it 34.896g

6 What to measure: knowledge visibility gap 29.4g

7 What to measure: knowledge decay rate 34.056g

8 ❓ Stop-check: how many questions can the agent answer? 26.136g

9 Principle 1: knowledge lives next to the code 36.648g

10 Principle 2: one standard entry point 23.208g

11 Principle 3: minimal, but complete 30.816g

12 Principle 4: updated alongside the code 28.464g

13 💡 Debrief: four principles close the gap 36.576g

14 A working repository structure for your agent 39.024g

15 Top level: entry point, progress, decisions 39.504g

16 The service layer: .agent/ and scripts/ 42.408g

17 Pitfalls: documentation that lies 37.032g

18 ACID for agent state 38.184g

19 Consistency: a green check after every step 31.344g

20 Consistency: a green check after every step 31.608g

21 Isolation: two transfers from one account don't clash 41.16g

22 Durability: context doesn't count 39.144g

23 Homework: close your knowledge gap 36.048g

24 The repository as a contract, not an archive 39.24g

24 phút

Trong đăng ký Đăng ký ngay

The rule is written right there in the file. The agent doesn't read it. Why?

Why one giant instructions file is worse than having none at all. Four independent causes of instruction degradation, the SNR (signal-to-noise ratio) metric for instructions, a three-tier architecture of router → documents → executable checks, the rule passport, and practical guidelines for keeping instructions in good working order.

router architecture instruction SNR three-tier architecture rule passport

26 phút

Trong đăng ký Đăng ký ngay

Your Agent Has Amnesia Every Session — Here's the Treatment Plan

An agent is like a brilliant engineer with complete amnesia every morning: why does each new session re-ask questions that were settled yesterday and contradict its own earlier decisions? We examine compaction versus context reset, four continuity artefacts (PROGRESS.md, session handoff, git checkpoints and a decision log), plus the recovery cost metric. We also cover initialisation as a phase in its own right: a bootstrap contract built on four conditions, an acceptance checklist, warm start versus cold start, and a minimal scripts/init.sh.

cross-session continuity session handoff bootstrap contract warm start

28 phút

Trong đăng ký Đăng ký ngay

Three features at once, none finished: the story of one agent

Why the agent has no brakes, and what happens when it grabs hold of three tasks at once — and how a hard WIP=1 limit and clear task boundaries put things right. Then, the feature list (feature_list.) as a genuine harness primitive: the three-part structure of each behavior/verification/state record, a four-state finite state machine, scripts/verify.sh as the only party entitled to set a status to passing, and the difference between strict and weak verification.

WIP=1 task granularity feature_list. feature finite state machine gatekeeper

28 phút

Trong đăng ký Đăng ký ngay

The agent wrote 'Done ✅'. The feature doesn't work. Let's work out why

The verification gap: the divide between 'the agent said it's done' and 'the feature actually works'. Three-layer completion validation, a Definition of Done in AGENTS.md, priority constraints (no refactoring before verification), red flags for the agent — self-healing error messages, and a three-role architecture (implementer / verifier / arbiter).

verification gap Definition of Done three-layer validation separation of roles

1 The anatomy of one 'Done' 38.904g

2 Three failures the tests never caught 31.992g

3 Why AI is far more confident than it is right 34.128g

4 The model is the best advocate for its own output 48.216g

5 The only real fix: separate the doer from the checker 34.272g

6 10 tasks that will show you how much your agent lies about being done 29.256g

7 ❓ Stop-check: what's your verification gap? 31.032g

8 💡 Breakdown: most teams land at 30–50% 29.952g

9 Three filters every feature must pass before it can be called done 53.472g

10 The progression rule: no skipping a layer 32.952g

11 Definition of Done in AGENTS.md 35.136g

12 🤔 Myth: unit tests pass, so it’s done 31.512g

13 Pitfall: refactoring before the feature has been verified 44.04g

14 The order is fixed: correctness → performance → style 32.016g

15 Red-pen feedback: an error should teach, not scold 33.312g

16 An error message the agent fixes itself — without you lifting a finger 38.136g

17 Planner, generator, assessor: three roles against agent self-deception 32.304g

18 Why the assessor must not see a single line of the generator’s reasoning 26.4g

19 🛠 The same prompt: broken one run, working the next 33.048g

20 Homework: measure your verification gap twice 28.392g

21 Done is what’s been verified, not what’s been said 29.712g

24 phút

Trong đăng ký Đăng ký ngay

All Tests Green — Yet the Feature Is Broken. Where Is the Bug Hiding?

Blind spots in unit testing: an agent passes every module test, yet the feature fails to work as a whole. Why E2E tests alter the agent's behaviour before they even catch the bug. How to turn an architectural rule ('the service must not access the database directly') into an executable check rather than a paragraph of text. The verification hierarchy, from linting all the way up to E2E.

Isolation blind spots End-to-end (E2E) tests Executable architectural rules Verification hierarchy

2 Everything's green. Nothing works. 40.44g

3 Myth: 100% unit coverage ≠ the feature works 36.024g

4 Unit tests aren't blind by accident — they were designed that way 44.424g

5 Four types of bug a unit test can't see in principle 44.352g

6 Units see strictly fewer bugs than E2E — and that's not a metaphor 47.88g

7 E2E isn't just a filter on the way out 50.232g

8 Three ways E2E changes the agent's code before the first run 35.952g

9 Architectural boundaries first, E2E second 44.184g

10 📖 An OpenAI observation: agents copy patterns 37.704g

11 Dependency arrows only point upwards — and a machine checks it 41.448g

12 Enforce invariants, don't micromanage the implementation 36.696g

13 ❓ Checkpoint: over to you 31.584g

14 Three levels of automating an architectural rule 34.896g

15 💡 Deep dive: level 2 — linter rules 35.64g

16 💡 Deep dive: level 2 — linter rules 32.76g

17 Level 3: architectural tests in CI 31.104g

18 Writing the same code review comment twice? That's a bug in your harness 38.16g

19 How a reviewer's complaint becomes an automated check: steps 1–3 28.944g

20 Steps 4–6: lock the check in for good — and don't forget the main thing 34.464g

21 Project: E2E for checkout and arch-check.sh for payments 44.424g

22 The Inverted Verification Pyramid 41.04g

23 Pitfall: calling it “done” at the third level from the bottom 38.064g

24 Project: E2E for checkout and arch-check.sh for payments 40.416g

25 Unit tests prove the details, not that the feature works as a whole 36.312g

26 phút

Trong đăng ký Đăng ký ngay

The Agent Did the Work — Yet Nobody Can Explain What It Did

Everything appears to work, yet no one — neither a human nor the agent's next session — can explain what exactly was done and why. We break down the two layers of observability, the sprint contract (in scope / out of scope / open questions settled before work begins) and an evaluator rubric in place of general opinion. Then comes clean handover and the battle against entropy: five dimensions of clean state, a session exit checklist, two-mode cleanup and idempotent cleanup operations.

harness observability sprint contract evaluator rubric clean handover harness entropy

28 phút

Trong đăng ký Đăng ký ngay

How to let an agent work for hours without your supervision

A shift from one-off prompting to an autonomous loop. The simplest possible loop; four types of loops that must not be confused; six loop primitives; the separation of generator and evaluator as the only non-negotiable guarantee; an example of a well-designed loop (goal, metric, constraints, order of operations, stopping conditions, escalation); four quiet costs of autonomy; and the maturity ladder.

Loop Engineering generator and evaluator stopping conditions escalation to a human

26 phút

Trong đăng ký Đăng ký ngay

When a Single Loop Is No Longer Enough for an Agent

Where the limits of a single-loop agent lie: three structural failures (no rollbacks, no shared state, no anchors to reality). A four-level framework and four elements of a graph. Anchors — fixed points that cannot be bypassed by local optimisation. The difference between a graph and a workflow. How to build your first graph in six steps. The orchestration tax, and five criteria for when a graph is genuinely needed.

Graph Engineering anchors orchestration tax graph versus workflow

2 The metric grows, the product dies 40.296g

3 +18% accuracy, −85% cost — figures you shouldn't trust 38.304g

4 Which of the four levels you're actually losing control on 38.904g

5 Node and edge: the only real boundary between a graph and an ordinary workflow 49.224g

6 What happens when two agents write to the same field at the same time 57.384g

7 📖 Judge and defendant in one head — that's what breaks a single loop 41.688g

8 The satnav won't ask if you're going the right way — and the loop won't ask if it's the right goal 42.84g

9 If you can't answer these four questions, don't draw the graph 36.336g

10 The anchor: what must never be optimised 52.536g

11 Where should a rollback lead when the final check fails 47.952g

12 ❓ Your turn: where is the anchor in your system 31.128g

13 💡 Walkthrough: an anchor is what you leave alone 39.768g

14 Graphs and workflows: the same skeleton, different contents 39.96g

15 A graph is a generalisation of the workflow, not a replacement 38.952g

16 🤔 Myth: drawing a graph is an engineering achievement 38.304g

17 Step 1: what can be made shared, and what must stay private forever 40.968g

18 Step 2: why verify must remember nothing about implement's work 51.096g

19 Steps 3–4: after the third failure, the rollback goes somewhere you wouldn't expect 51.768g

20 Steps 5–6: the checkpoint that gives you human approval for free 39.576g

21 Loop: decide later. Graph: decide up front 48.216g

22 Spinning up an agent is cheap — closing the loop is expensive 36.792g

23 Five criteria: when a graph is genuinely needed 43.584g

24 A graph isn't an upgrade to the loop — it's a different level of the system 31.416g

24 phút

Trong đăng ký Đăng ký ngay

Assembling the Harness in Full: From an Empty Repository to an Autonomous Agent

Course finale: we take everything we built step by step across the end-to-end project — a payments FastAPI backend — and bring it together into one working system. We walk through every harness artefact from start to finish: the AGENTS.md router with its topic documents, feature_list. with its finite state machine, PROGRESS.md and the shift-handover protocol, scripts/verify.sh and scripts/arch-check.sh as verification gates, the sprint contract and assessor rubric, the clean session-exit checklist, program.md for the autonomous loop and graph.md for the multi-role process. To finish, we compare the agent's metrics before and after adopting the harness.

assembling the harness as a whole end-to-end project before/after metrics capstone

2 We've spent twelve lectures building the harness piece by piece. Does it actually work as one system? 37.488g

3 This is what the repository looks like once the harness is finally finished 38.568g

4 AGENTS.md isn't a tips file — it's an 80-line router 30.864g

5 'Done' means the checklist has been passed — not that it merely looks done 27.888g

6 feature_list.: a task list that knows how to say 'no' 32.784g

7 Who gets to write the word 'passing'? Only this script 26.16g

8 A new session doesn't ask 'what were we doing yesterday?' — it reads 33.384g

9 One script answers the question “is everything all right?” before the agent says a word 22.8g

10 One architectural boundary that is now physically impossible to violate 26.496g

11 The feature contract is written BEFORE the code — not as an explanation afterwards 27.864g

12 “Looks fine” is banned here — grading runs on a rubric backed by evidence 30.864g

13 How an agent drives the feature queue all night and decides for itself when to stop 32.112g

14 A single agent can no longer cope alone — meet the four-role graph 32.712g

15 The graph knows how to roll itself back — here’s where a feature goes when review objects 35.328g

16 ❓ Checkpoint: the assembled harness hides one conflict — find it 37.992g

17 💡 Breakdown: a passing result from verify.sh doesn't mean 'the full Definition of Done has been met' 42.576g

18 MTTR down sevenfold, verification gap from 38% to 4%: here's the full report 43.2g

19 The harness is something you'll still be fixing long after the course ends 42.72g

20 phút

Trong đăng ký Đăng ký ngay