Ausschnitt der Lektion

All Tests Pass, Feature Is Broken: E2E Tests and Architecture Rules for AI Agents

Unit test blind spots when an AI agent works on a codebase: why tests pass while the feature fails end to end. E2E tests, turning architecture rules into executable checks (arch-check), and the verification hierarchy from linters to E2E.

Isolation blind spots End-to-end (E2E) tests Executable architectural rules Verification hierarchy

Was Sie in dieser Lektion lernen werden

Identify a blind spot in your own test coverage where unit tests give a false picture of feature readiness.
Write at least one architectural rule as an executable script rather than a paragraph buried in documentation.
Build a verification hierarchy that runs from the cheapest checks through to the most expensive ones.

Anfang der Lektion

Everything's green. Nothing works.

Three layers of unit tests — all passing
The 'Export' button: broken path, frozen progress, leaking memory
Five defects — and not one caught by the unit tests

Five bugs were hiding exactly where no test ever looked. The scene: a desktop app with a file export feature. Unit tests on all three layers — from path parsing to the write module — are green, and the agent reports 'done'. You run the app yourself and click the button. The path format is wrong — the component expected an absolute path and got a relative one. The progress bar doesn't move. On large files the app starts eating memory and eventually crashes. Five defects in total, and every single one sits right on the boundaries between components. Not one unit test noticed them, because each test exercised its own module in complete isolation from its neighbours.

Myth: 100% unit coverage ≠ the feature works

🤔 Myth / Often confused:
'Unit tests are green — so the feature is ready'
✅ In reality:
A unit test checks a module in isolation, not the joint between modules
📌 Remember: green units only prove that each part works on its own

A hundred per cent unit coverage doesn't mean the feature works — and this is the most common confusion of all. In fact, a unit test by definition checks a module in isolation — dependencies are mocked, the environment swapped out. It tells you nothing about what happens at the joint between two real components: whether data formats line up, whether resources get released, whether the logic survives a real environment. That's exactly why, in our file export example, the agent was confident everything was ready — and it was wrong in five places at once, none of which sits inside a single module.

Unit tests aren't blind by accident — they were designed that way

Mock the dependencies, test the module on its own
Fast and precise — but systematically blind
The blindness isn't a bug in the tests — it's the design

Picture a production line at the Mini plant in Oxford: the brake caliper is tested on its own rig, the steering wheel on another, and every sensor is taken off the car and bench-tested separately. Each check is fast and precise — you can see straight away which part is faulty. But not a single rig checks what happens when all the parts are assembled into one car and it drives off down the road. A unit test works the same way: it mocks everything around it and checks one module in pristine isolation. That gives you speed and diagnostic precision — but that same isolation makes the tests systematically blind to four classes of defect that live not inside a module but at the joint between modules — precisely where the parts are meant to come together into one system. The blindness isn't a bug in the tests; it's a direct consequence of their design.

Die vollständige Lektion — mit Abo

Noch 21 Folien in der vollständigen Version.

Video, alle Folien, Notizen, Tests, KI-Tutor und Interview werden nach Anmeldung und Abschluss eines Abos freigeschaltet.

Anmelden oder registrieren Abo abschließen

Inhalt der Lektion

Everything's green. Nothing works.
Myth: 100% unit coverage ≠ the feature works
Unit tests aren't blind by accident — they were designed that way
Four types of bug a unit test can't see in principle
Units see strictly fewer bugs than E2E — and that's not a metaphor
E2E isn't just a filter on the way out
Three ways E2E changes the agent's code before the first run
Architectural boundaries first, E2E second
📖 An OpenAI observation: agents copy patterns
Dependency arrows only point upwards — and a machine checks it
Enforce invariants, don't micromanage the implementation
❓ Checkpoint: over to you
Three levels of automating an architectural rule
💡 Deep dive: level 2 — linter rules
💡 Deep dive: level 2 — linter rules
Level 3: architectural tests in CI
Writing the same code review comment twice? That's a bug in your harness
How a reviewer's complaint becomes an automated check: steps 1–3
Steps 4–6: lock the check in for good — and don't forget the main thing
Project: E2E for checkout and arch-check.sh for payments
The Inverted Verification Pyramid
Pitfall: calling it “done” at the third level from the bottom
Project: E2E for checkout and arch-check.sh for payments
Unit tests prove the details, not that the feature works as a whole