Anteprima della lezione

AI Agent Observability: Sprint Contracts, Evaluator Rubrics and Clean Session Exit

How to know what an AI coding agent did: two observability layers, sprint contract (in scope / out of scope), an evaluator rubric instead of opinion, clean handoff, session exit checklist and code entropy.

harness observability sprint contract evaluator rubric clean handover harness entropy

Cosa imparerai in questa lezione

You will set up a sprint contract and an evaluator rubric instead of the vague 'looks fine'.
You will implement a clean session exit checklist covering five dimensions.
You will halt the growth of harness entropy — measured quantitatively, not by feel.

Inizio della lezione

Sound familiar?

The agent reported the job done — but nobody knows what was actually verified
The next session starts not with the task, but with excavating someone else's code
The longer a project lives, the more each subsequent agent run costs

This isn't a one-off glitch — it's a systemic hole in harness observability. Over the course of a week, the same agent can honestly say “done” ten times — and each time it will mean something different. And after a month of work with no explicit cleaning up after itself, every new session spends not five minutes warming up but half an hour — just to work out what's actually going on in the project. Today we'll tackle both ends of this problem: how to make the agent's work explainable, and how to hand over cleanly between sessions.

The agent isn't being dim — it simply has nothing to show

“Done, but two tests are failing” — “Why?” — “Could be a timing issue”
“Which critical paths did you touch?” — “Let me have a look at the code…”
The agent isn't stupid — it physically has nothing to present as evidence

Play this dialogue through in your head — it will surely be familiar. The agent spent twenty minutes changing files and came back with “done, but two tests are failing”. Ask “why”, and you get guesswork about timing. Ask “which critical paths did you touch”, and the agent has to dive back into the code to answer a question about its own work. It's not about the model's intelligence. The problem is that the harness gave it nothing to point to: no execution trace, no agreement on what the result was even supposed to look like. Nothing to observe means nothing to explain — even if an explanation exists.

Observability is really two different layers

Runtime: logs, traces, metrics — answers “what happened”
Process: sprint contract, rubric, acceptance criteria — answers “why it was done this way”
Runtime only = you can see what broke, but not what was wanted
Process only = you know what was wanted, but not why it didn't work out

A harness has two independent layers of observability, and people mix them up all the time. Picture a garage: first the mechanic signs off a work order with you — what's to be fixed, and by which signs the car will count as roadworthy. That's the process layer: the sprint contract, the evaluator rubric, the acceptance criteria — they explain why a change should be accepted at all. And when you come to collect the car, you're shown the service log — what was removed, what was replaced. That's the runtime layer: logs, traces, metrics — what the system actually did. Service log only, and you can see the mechanic was turning spanners, but it's unclear whether he fixed what was actually broken. Work order only, and you know the goal but can't fathom why the car still squeaks. You need both — and they reinforce each other.

La lezione completa è disponibile con l'abbonamento

23 altre slide nella versione completa.

Video, tutte le slide, appunti, test, tutor AI e intervista si sbloccano dopo l'accesso e l'abbonamento.

Accedi o registrati Attiva l'abbonamento

Contenuti della lezione

Sound familiar?
The agent isn't being dim — it simply has nothing to show
Observability is really two different layers
Without observability: “correct” turns into mysticism
❓ Your move: how to stop the grading lottery
Blind retries and a dropped handover between sessions
Why the agent won't patch this on its own
Sprint contract: the agreement before the code
A sprint contract in full: one page instead of guesswork
✅ The answer: a rubric instead of opinion
Common pitfall: scoring without evidence
Common pitfall: scoring without evidence
Task traces: not anecdotes, but aggregates
The agent can explain everything — and still leave a mess behind
Thirty minutes just to work out what the last session did
Why entropy grows on its own
Pitfall: 'we'll tidy it up later'
A session is only closed when all five dimensions are closed
Cleanup is two different modes, and you mustn't mix them up
Cleanup is two different modes, and you mustn't mix them up
The quality document: .agent/quality.md
The harness ages too: simplification isn't optional
The cleanup script must survive its own failure
One loop — either it closes cleanly, or it starts over from scratch
End-to-end project: step 6 for the payments backend
Summary: a harness that explains itself