The agent wrote "done", but the feature does not work: why it happens and how to verify an AI agent's work

Why this happens

Everyone who works with agents knows the situation: the agent wrote "done", but the feature does not work. It is not lying on purpose. It reports what it managed to check and is sincerely convinced that this is enough. The gap between the claim and reality is a property of the scheme "the executor evaluates its own work", not a defect of a particular model.

There are three reasons. The first is that models tend to be overconfident: the confidence of an answer is higher than its accuracy. The second is that the agent evaluates its work in the same context where it did it: it has already convinced itself that the solution is correct, and asking it to "be more objective" does not change the context. The third, and the most practical, is that the agent has access only to partial checks: unit tests turn green quickly, while the end-to-end scenario has never been run.

Chronicle of a typical false "done"

The agent was asked to implement password reset. It changes the database schema, adds an endpoint, wires up an email template, runs the unit tests — everything is green. A confident "Done" appears in the chat.

In reality the following happened:

• the migration failed halfway and left the schema in an inconsistent state;
• the email template exists, but the mail service (SMTP) is not configured in the environment — no email is sent;
• the scenario "user clicks the link → enters a new password → logs in" was never run.

Unit tests could not catch any of these problems: the database was mocked, sending the email was mocked, and there was no end-to-end run at all.

Why green tests do not prove that the feature works

A unit test checks a module in isolation: dependencies are replaced with mocks, the outside world is switched off. This gives speed and precision, but also a systematic blindness. The blindness is not a bug in the tests but a consequence of their design. Green unit tests prove only that each part works on its own, not that the parts work together.

There are four classes of defects that a unit test cannot see at all:

Verification levels are nested: everything that unit tests catch is also caught by integration tests, and everything integration tests catch is caught by an end-to-end (E2E) run. Each higher level sees strictly more. That is why the right to say "done" should belong only to the top level.

How to measure the problem: the verification gap

Before fixing anything, count. Take the last 10 tasks closed by the agent. For each, answer two questions: did the agent say "done", and did independent verification confirm it (you walked through the scenario yourself, the E2E test is green)? The share of mismatches is your verification gap.

This number is useful for two reasons. It shows the scale of the problem without arguments of the "well, it seems to work" kind. And it gives a baseline: after every change to the process you recalculate it on a similar set of tasks and see whether it helped.

How to really verify: three layers of validation

A handy analogy is a car service station. A car passes three stations, and no station can be skipped.

1. Diagnostics (seconds): syntax, types, linter. It catches typos and gross errors.
2. Test bench (minutes): unit and integration tests, the application starts. The engine works, but the car is still standing still.
3. Test drive: an end-to-end scenario run the way a user runs it — through the API, a browser or the CLI, with a real database and configuration.

The progression rule: you do not go on the test drive while diagnostic errors are still lit, and you do not release the car without the test drive. Green unit tests are the second station passed, not the third.

It is convenient to put everything behind one entry point:

# Makefile
check:
	ruff check .
	mypy app
	pytest tests/unit tests/integration

e2e:
	docker compose up -d db mailhog
	alembic upgrade head
	pytest tests/e2e

verify: check e2e

Here E2E starts a real database and a test mail server, applies the migrations and runs the scenario from start to finish. A failed migration or an unconfigured mail server shows up immediately, not at the client's side.

Definition of Done in AGENTS.md

The agent should not decide by itself what "done" means. The criterion is written into the instructions file and phrased in a verifiable way:

## Definition of Done
A feature is ready only if ALL items are met:
1. make check finished with exit code 0
2. make e2e finished with exit code 0, and the feature's scenario is in tests/e2e
3. Migrations apply on a clean database and roll back
4. New environment variables are described in .env.example
5. PROGRESS.md records what was done and how it was verified

These are NOT signs of readiness:
- "the code is written and looks correct"
- "the unit tests pass"
- "it works on my machine"

An explicit list of what does not count as readiness works better than it seems: the agent stops at the first green run less often.

Who has the right to set "passing"

The next step is to take the readiness status out of the agent's hands altogether. Features are kept in a list (for example, feature_list.json) with the fields "behavior", "verification command" and "state". Only a script may switch a feature to passing: it runs the common make check, then the verification command of the specific feature, and on success records the status together with the commit hash as evidence. If it fails, the feature stays active — there is no "almost worked".

{
  "id": "F04",
  "behavior": "The user receives an email with a link and changes the password",
  "verification": "pytest tests/e2e/test_password_reset.py",
  "state": "active"
}

There is a trap here: the script checks exactly what is written in the verification field. If an architectural check is part of the Definition of Done but not part of the feature's command, the gap stays invisible. That is why the common make check must include all mandatory checks, not only the tests.

An independent evaluator

An executor's self-report is never an independent source of truth. The structural solution is to separate the roles:

• the planner breaks the requirement into features with acceptance criteria and does not write code;
• the generator implements one feature and runs the tests, but does not decide whether it is done;
• the evaluator gives a verdict against a checklist of criteria.

The evaluator must not see the generator's reasoning, otherwise it inherits its confidence. It receives only the requirement, the diff and the results of the runs. In Claude Code this is easy to do with a sub-agent: it has its own clean context. Even better if the evaluator is a different model: it has different blind spots. An assessment of "looks fine" is forbidden: every finding must refer to a file, a line or a log.

Error messages the agent can fix on its own

A check is useful only if its output leads to a fix. "Test failed" or a bare AssertionError tells the agent nothing. A good message says what broke, why the rule exists and how to fix it:

ERROR arch-check: app/services/refund.py imports app/db/session.py
WHY: services do not access the database directly, only through repositories
FIX: use RefundRepository from app/repositories/refund.py

This format turns a check from a prohibition into a hint: the agent fixes the violation without your involvement.

Manual checking: where it is still needed

Automation does not cancel the human, but it narrows the human's role. Manual checking is needed where the criterion is hard to formalize: the look of the interface, the texts, the comfort of a scenario. Do it using a checklist of acceptance criteria rather than "clicking around". If you write the same comment to the agent a second time, that is a signal to turn it into an automatic check: a test, a linter rule or a grep guard in make check.

How E2E changes the agent's behavior before the run

The value of an end-to-end test is not only in the bugs it catches. If the agent knows in advance that its work will be checked by an E2E scenario, it writes the code differently: it thinks about the contract with the neighboring component, not only about its own function, handles the error paths and does not cross architectural boundaries, because a violation would surface in the run. The check works as a constraint at the input, not only as a filter at the output.

There is a condition: E2E makes sense when the system has clear boundaries. On a spaghetti architecture an end-to-end test will only prove that the spaghetti works and will not show where the design was violated. That is why layers and dependency rules are established first, and only then end-to-end scenarios are built. Agents readily copy the existing patterns of a repository, including bad ones, so constraints are better introduced from day one.

Common mistakes

• Asking the agent to "check more carefully". The context is the same, so the assessment will remain inflated.
• Treating 100% coverage as a guarantee. Unit coverage does not cover the seams.
• Refactoring before the feature is confirmed. The order is strict: correctness → performance → style.
• Mocking in E2E what breaks most often. The database, migrations and configuration must be run for real in an end-to-end test; external paid APIs can be replaced with a test server.

Summary

When the agent has written "done" and the feature does not work, the problem is almost always in the verification process, not in the model. Count the verification gap, fix the Definition of Done in AGENTS.md, make E2E mandatory, give the readiness status to a script and add an independent evaluator. This scheme is covered in practice in the harness engineering course.