AI agent observability: how to see what the agent did and what to log

What AI agent observability is

AI agent observability is the ability to answer three questions from artifacts, not from memory or from the agent's words: what it did, why it did it that way and what proves the result is correct. If after twenty minutes of work the agent says "done, but two tests fail, possibly because of timing", and when asked which paths were affected it replies "let me look at the code", there is no observability. The agent is not stupid: it simply has nothing to show you.

The problem is systemic. The same agent can honestly say "done" ten times in a week, and each time the word will mean something different: in one case all tests were run, in another only one, in another nothing. The longer a project lives, the more expensive every next session becomes, because it starts with digging.

Two layers: runtime and process

Observability is often confused with logs, but logs are only half of it.

Runtime alone: you can see what failed, but not what was being aimed for. Process alone: you know what was wanted, but not why it did not work out. You need both.

Why the agent will not do this on its own

Telling the agent "log better" does not work, for three reasons.

• It does not know what it does not know. The agent will not log a signal whose importance it does not suspect.
• The format drifts. In one session it writes in detail, in another in one line. Sessions cannot be compared and aggregates cannot be computed.
• Logs cannot close the process gap. The answer to "why this way" lives in documents written before the work, not in command output.

That is why observability is built into the harness — the agent's environment: templates, hooks, scripts, a mandatory report format.

What to log: the minimal set

A good journal is not a transcript of everything, but a set of facts from which you can reconstruct the course of work and compute statistics.

• Identifiers: the session ID, the task or feature ID, the agent's model.
• Actions with side effects: which commands were run, which files were changed. Reading files usually does not need to be logged.
• Check results: which verification command, the exit code, how many tests passed and failed.
• Retries: how many attempts a step took and how they differed.
• Consumption: tokens and time, if the tool reports them.
• Outcome: the task was accepted, rejected or abandoned, and why.

What not to write: secrets, access tokens, personal data from command output. The journal may be shared or fed back into the model's context.

An action journal through Claude Code hooks

In Claude Code, the journal is conveniently kept with hooks — commands that the tool itself runs, not the model, so they always fire. A hook receives a JSON description of the event on its input, including the session ID, the tool name and its parameters. Example configuration in .claude/settings.json that appends a line to the journal after every Bash, Edit or Write call:

{
  "hooks": {
    "PostToolUse": [
      {
        "matcher": "Bash|Edit|Write",
        "hooks": [
          { "type": "command", "command": ".claude/hooks/log-action.sh" }
        ]
      }
    ]
  }
}

#!/usr/bin/env bash
# .claude/hooks/log-action.sh — one JSON line per action
mkdir -p .agent/logs
jq -c '{ts: (now | todate), session: .session_id, tool: .tool_name,
        cmd: .tool_input.command, file: .tool_input.file_path}' \
  >> .agent/logs/actions.jsonl || true

The JSONL format (one JSON record per line) is convenient because it is easy to filter with jq and load into any analytics system. In addition, Claude Code itself keeps the full history of sessions locally in the ~/.claude/projects/ directory — you can open it if you want to see the reasoning and the output in full, but for statistics your own compact journal is more convenient.

Tracing: one trace per session, a span per task

When there are many agents or they work on a schedule, text journals are not enough: you need aggregates. Ordinary distributed tracing tools and the OpenTelemetry standard are useful here.

• Trace — one agent session.
• Span — one task within the session.
• Nested spans — verification steps: build, tests, end-to-end.

The minimum attributes per span: the feature ID and session ID, the model, the verification result, tokens, the number of retries. Such data can be read in Jaeger, Grafana Tempo or Honeycomb and answers questions that are not visible within a single session: which type of task needs retries most often, where the agent spends the most tokens, after which change to the harness the share of accepted tasks dropped. Claude Code has built-in export of metrics and events through OpenTelemetry — it is enabled with environment variables; the details are in the tool's documentation.

Commits as a journal

The cheapest and most reliable journal you already have is git. It only needs to be made readable.

1. One commit — one task. Then git log matches the list of what was done, and rolling back a task is one git revert.
2. The message says what was verified and how. Not "fix", but "what changed, why, and which command verified it".
3. Commit only after a green check. Then every point in history is a working state, and git bisect finds the culprit.

fix(payments): round discount to cents

The discount was computed in float, a cent was lost on a sum of 999.99.
Moved the calculation to int in cents.
Verified: make test (142 passed), make e2e checkout — ok.
Out of scope: rounding in reports, recorded in BACKLOG.md.

The agent's report: evidence instead of impressions

The agent's final message is also an observability artifact, if it has a fixed form. Put the template into AGENTS.md or CLAUDE.md:

## Report at the end of a task
- What was done: list of changes (file — essence)
- How it was verified: exact commands and their results
- What was NOT verified and why
- Deviations from the task agreement
- Open questions for a human

The main rule: every statement points to evidence — a file and line, test output, an entry in the journal. "Looks fine" and "I think it's ok" do not count as a report. The item "what was not verified" is especially valuable: it honestly shows the limits of confidence.

The agreement before the code and the rubric after

The process layer starts before the work. A short task agreement (a sprint contract) fixes the scope, the acceptance criteria, a section "out of scope" and the open questions that a human decides. The "out of scope" section prevents half of the disputes about the result.

After the work, the result is assessed by a rubric, not by impression. Otherwise two experienced reviewers will give different verdicts, and the assessment becomes a lottery.

Each dimension is scored A/B/C/D, and a score without a reference is invalid. A simple decision rule: one D or two Cs — reject, with a list of what to fix. The same rubric can be applied by a separate checking agent.

How to review what the agent did during a session

If there is no report, or it cannot be trusted, go through the artifacts in this order.

1. The state of the tree: git status — what is not committed, whether there are stray files.
2. The history: git log --stat from the start of the session — which commits and files.
3. The scale of changes: git diff --stat main...HEAD, then the full diff of the important files.
4. The action journal: which commands were run and what the checks returned.
5. A repeat check: run the build and the tests yourself. Trust the output of a command, not a retelling.

# which commands the agent ran in the last session
jq -r 'select(.tool=="Bash") | .cmd' .agent/logs/actions.jsonl | tail -n 30
# which files were changed most often
jq -r '.file // empty' .agent/logs/actions.jsonl | sort | uniq -c | sort -rn | head

Common mistakes

• Logging everything. Gigabytes of transcripts that nobody reads. Record actions with side effects and check results.
• A free-form format. If the agent sets the journal format itself, there will be no aggregates. The harness sets the format.
• Trusting a report without command output. "Tests are green" without the output is an opinion.
• Blind retries. Without the cause of the failure, the agent edits random places. Require a diagnosis with a reference to the log first, and only then a fix.
• "We'll clean it up later". Debug files and uncommitted changes confuse the next session: it cannot tell the temporary from the intentional.

Summary

AI agent observability is built from two layers: runtime (an action journal through hooks, traces, commits with a description of checks) and process (a task agreement before the code, a rubric and a report with links to evidence). Then "done" means the same thing in every session, and reviewing an agent's work takes minutes. The topic is covered in practice in the harness engineering course, on a cross-cutting payments backend project.