Đoạn xem trước bài giảng

Agent Says 'Done' but the Feature Is Broken: Verification Gap and Definition of Done

The verification gap in AI coding agents: why 'done' doesn't mean 'works'. Three-layer completion validation, Definition of Done in AGENTS.md, no refactoring before verification, and splitting roles between executor, verifier and arbiter.

verification gap Definition of Done three-layer validation separation of roles

Bạn sẽ học gì trong bài giảng này

Calculate the verification gap in your process and pinpoint where the agent is overstating its readiness.
Write a Definition of Done that is verified by a script rather than by the agent's own judgement.
Separate the implementer and verifier roles so that neither can override the other.

Phần đầu bài giảng

The anatomy of one 'Done'

Database schema changed, endpoint written, email template added
Unit tests green — the agent reports completion
The migration failed halfway through, the email never goes out, and the end-to-end scenario was never run once

Let's rewind a couple of minutes and look, frame by frame, at what actually happened. The agent changes the database schema, adds the endpoint, wires in the email template, runs the unit tests — all green, report ready. It says 'done' with exactly the same confidence it would have if the feature worked perfectly. But the migration actually failed halfway through, leaving the schema in an inconsistent state. The email service isn't configured at all. And the end-to-end scenario — a person clicks the link, enters a new password — was never run once in the whole session. Code review or production will be the first to find out.

Three failures the tests never caught

The database schema migration failed halfway through — the state is inconsistent
The email template exists, but SMTP isn't configured — the email never goes out
The end-to-end scenario 'click the link → change the password' was never run once

Three concrete failures sit behind one breezy 'done'. First: the schema migration failed midway through, leaving the database in an inconsistent, half-finished state. Second: the email template is there, the code looks lovely, but the environment variable holding the mail server address was never set — so the email simply never goes out. Third: nobody, the agent included, ever ran the scenario end to end, from clicking the link through to changing the password. The unit tests are green precisely because each one checks its own small piece in isolation, rather than the system as a whole.

Why AI is far more confident than it is right

A property of the whole class of systems, not of any one vendor
Guo et al., ICML 2017: neural networks are poorly calibrated — confidence runs ahead of accuracy
The agent 'feels' the finish line long before it has actually crossed it

This isn't the quirk of one particular model — it's a fundamental property of the whole class of systems. A classic paper by Guo and colleagues, presented at the ICML conference in twenty seventeen, showed that modern neural networks are poorly calibrated: they claim confidence well above their actual accuracy. For a coding agent this plays out very concretely — it 'feels' that the task is solved long before it actually is. Confidence in the report doesn't correlate with the real quality of the result, and counting on a stronger model to fix it is wishful thinking.

Bài giảng đầy đủ dành cho thuê bao

Còn 18 slide nữa trong phiên bản đầy đủ.

Video, toàn bộ slide, ghi chú, bài kiểm tra, gia sư AI và phỏng vấn sẽ mở sau khi đăng nhập và đăng ký thuê bao.

Đăng nhập hoặc đăng ký Đăng ký thuê bao

Nội dung bài giảng

The anatomy of one 'Done'
Three failures the tests never caught
Why AI is far more confident than it is right
The model is the best advocate for its own output
The only real fix: separate the doer from the checker
10 tasks that will show you how much your agent lies about being done
❓ Stop-check: what's your verification gap?
💡 Breakdown: most teams land at 30–50%
Three filters every feature must pass before it can be called done
The progression rule: no skipping a layer
Definition of Done in AGENTS.md
🤔 Myth: unit tests pass, so it’s done
Pitfall: refactoring before the feature has been verified
The order is fixed: correctness → performance → style
Red-pen feedback: an error should teach, not scold
An error message the agent fixes itself — without you lifting a finger
Planner, generator, assessor: three roles against agent self-deception
Why the assessor must not see a single line of the generator’s reasoning
🛠 The same prompt: broken one run, working the next
Homework: measure your verification gap twice
Done is what’s been verified, not what’s been said