Fragmento de la lección

Agent Says 'Done' but the Feature Is Broken: Verification Gap and Definition of Done

The verification gap in AI coding agents: why 'done' doesn't mean 'works'. Three-layer completion validation, Definition of Done in AGENTS.md, no refactoring before verification, and splitting roles between executor, verifier and arbiter.

Inicio de la lección

The anatomy of one 'Done'

• Database schema changed, endpoint written, email template added
• Unit tests green — the agent reports completion
• The migration failed halfway through, the email never goes out, and the end-to-end scenario was never run once

Let's rewind a couple of minutes and look, frame by frame, at what actually happened. The agent changes the database schema, adds the endpoint, wires in the email template, runs the unit tests — all green, report ready. It says 'done' with exactly the same confidence it would have if the feature worked perfectly. But the migration actually failed halfway through, leaving the schema in an inconsistent state. The email service isn't configured at all. And the end-to-end scenario — a person clicks the link, enters a new password — was never run once in the whole session. Code review or production will be the first to find out.

Three failures the tests never caught

• The database schema migration failed halfway through — the state is inconsistent
• The email template exists, but SMTP isn't configured — the email never goes out
• The end-to-end scenario 'click the link → change the password' was never run once

Three concrete failures sit behind one breezy 'done'. First: the schema migration failed midway through, leaving the database in an inconsistent, half-finished state. Second: the email template is there, the code looks lovely, but the environment variable holding the mail server address was never set — so the email simply never goes out. Third: nobody, the agent included, ever ran the scenario end to end, from clicking the link through to changing the password. The unit tests are green precisely because each one checks its own small piece in isolation, rather than the system as a whole.

Why AI is far more confident than it is right

• A property of the whole class of systems, not of any one vendor
• Guo et al., ICML 2017: neural networks are poorly calibrated — confidence runs ahead of accuracy
• The agent 'feels' the finish line long before it has actually crossed it

This isn't the quirk of one particular model — it's a fundamental property of the whole class of systems. A classic paper by Guo and colleagues, presented at the ICML conference in twenty seventeen, showed that modern neural networks are poorly calibrated: they claim confidence well above their actual accuracy. For a coding agent this plays out very concretely — it 'feels' that the task is solved long before it actually is. Confidence in the report doesn't correlate with the real quality of the result, and counting on a stronger model to fix it is wishful thinking.

Contenido de la lección

1. The anatomy of one 'Done'
2. Three failures the tests never caught
3. Why AI is far more confident than it is right
4. The model is the best advocate for its own output
5. The only real fix: separate the doer from the checker
6. 10 tasks that will show you how much your agent lies about being done
7. ❓ Stop-check: what's your verification gap?
8. 💡 Breakdown: most teams land at 30–50%
9. Three filters every feature must pass before it can be called done
10. The progression rule: no skipping a layer
11. Definition of Done in AGENTS.md
12. 🤔 Myth: unit tests pass, so it’s done
13. Pitfall: refactoring before the feature has been verified
14. The order is fixed: correctness → performance → style
15. Red-pen feedback: an error should teach, not scold
16. An error message the agent fixes itself — without you lifting a finger
17. Planner, generator, assessor: three roles against agent self-deception
18. Why the assessor must not see a single line of the generator’s reasoning
19. 🛠 The same prompt: broken one run, working the next
20. Homework: measure your verification gap twice
21. Done is what’s been verified, not what’s been said

Artículos sobre esta lección

• El agente escribió «listo», pero la funcionalidad no funciona: por qué pasa y cómo verificar el trabajo de un agente de IA
• Qué es harness engineering: cómo convertir un agente de IA en una herramienta útil
• Cómo hacer fiable un agente de IA: plan paso a paso, de la medición al trabajo autónomo
• CLAUDE.md y AGENTS.md: qué son, qué escribir en ellos y por qué el agente ignora las reglas
• Harness vs context engineering: en qué se diferencia el harness engineering del prompt y del context engineering

Todas las lecciones del curso

1. What Is Harness Engineering? Why AI Coding Agents Fail Because of the Environment
2. Error Accumulation in AI Agents: Why 95% Per-Step Accuracy Fails by Step 10
3. AI Agent Reliability Metrics: MTBF, MTTR, Blast Radius and Little's Law
4. Why AI Agents Fail in Production: Five Failure Layers and a Diagnostic Loop
5. What an AI Agent Harness Is Made Of: Instructions, Tools, Environment, State, Feedback Loops
6. Repository Map for AI Coding Agents: Project Structure and State Without Memory
7. AGENTS.md Best Practices: Why a Giant Instructions File Fails and How to Build a Router
8. AI Agent Memory Between Sessions: Session Handoff, PROGRESS.md and Bootstrap Contract
9. feature_list.json for AI Agents: WIP=1, Feature Status and verify.sh Verification
10. Agent Says 'Done' but the Feature Is Broken: Verification Gap and Definition of Done
11. All Tests Pass, Feature Is Broken: E2E Tests and Architecture Rules for AI Agents
12. AI Agent Observability: Sprint Contracts, Evaluator Rubrics and Clean Session Exit
13. Autonomous AI Coding Agent for Hours: Loop Engineering and Stop Conditions
14. Graph Engineering for AI Agents: When a Single Loop Is Not Enough and You Need a Task Graph
15. Building an AI Agent Harness From Scratch: AGENTS.md, feature_list.json, verify.sh and an Autonomous Loop