AI Agent Evals: How to Measure Reliability Instead of Vibes

Why evaluate AI agents if it is already clear they work

That is exactly what is hard to see. Over a week the agent never got stuck after a prompt change: is that a fix, or just a week of simple tasks? One engineer is sure the new template changed everything, a second one read the same log and noticed no difference. Both are right only as far as neither of them has counted anything.

AI agent evals replace the feeling with a number. You take a fixed set of tasks, run the agent, check the results automatically, and compare the metrics before and after a change. Then the argument "did it get better" is settled by a table, not by authority.

Without evals, any change to the harness (a prompt, AGENTS.md, the tool set, the model) is a blind bet. With evals, it is an experiment with a measurable result.

Why "vibes" mislead

The impression from an agent's work has three systematic errors, and all of them push toward false confidence.

• Survivorship bias. Successful runs are remembered and shown to colleagues; failed runs are restarted and forgotten. A reliability figure computed from successful runs says nothing about where the agent actually breaks.
• Small samples. Three successes in a row is a normal event even for an agent that succeeds in half of the cases. The probability is 0.5³ = 12.5%, meaning roughly every eighth streak.
• Confident tone. The agent writes "done, all tests pass" with the same confidence whether or not it is true. The model's confidence is not calibrated: the tone of an answer is not a signal of correctness.

Hence the rule: anything that is not recorded and recalculated counts as unmeasured.

pass@k and pass^k: two metrics people confuse

An agent is non-deterministic: the same task may pass on one repeat and fail on another. So one task is run several times, and one of two metrics is computed.

pass@k: at least one of k attempts succeeds

The metric comes from the evaluation of code generation. It answers the question "is the system capable of solving the task at all, given several attempts". It is an honest metric for a scenario where a human picks the best of several variants, or where there is a reliable automatic check that filters out the failures.

pass^k: all k attempts succeed

This metric answers a different question: "how consistently does the agent solve the task". It describes work without supervision, where the agent is launched many times and every failure costs money or time.

The difference between them is huge. Suppose an agent solves a task with 70% probability per attempt, and the attempts are independent:

By pass@5 the agent looks almost perfect; by pass^5 it looks unreliable. Both numbers are correct; they just describe different things. If you plan to let the agent work autonomously, look at pass^k: it is closer to what you will see in real work.

How to calculate it in practice

Run each task n times (for example, 5) and record the number of successes c. For pass^k, the value is approximately the share of runs where all k in a row succeed; more simply, estimate the success probability p = c / n and raise it to the power k. For a set of tasks, average over the tasks.

import json
from statistics import mean

# results.jsonl: {"task": "add-pagination", "runs": [true, true, false, true, true]}
rows = [json.loads(line) for line in open("results.jsonl")]

def pass_at_k(runs, k):
    p = sum(runs) / len(runs)
    return 1 - (1 - p) ** k

def pass_pow_k(runs, k):
    p = sum(runs) / len(runs)
    return p ** k

for k in (1, 3, 5):
    print(k,
          round(mean(pass_at_k(r["runs"], k) for r in rows), 3),
          round(mean(pass_pow_k(r["runs"], k) for r in rows), 3))

The estimate via p to the power k is rough when n is small, but for comparing versions of the harness with each other it is usually enough. What matters more is that n and the task set do not change between comparisons.

How to assemble a task set for evals

The task set is the main asset. A good metric on bad tasks measures the wrong thing.

Take tasks from your own reality

Public benchmarks such as SWE-bench are useful for comparing models, but their tasks are curated: clear descriptions, ready-made tests. In your repository the requirements are vague and tests are sometimes missing, so a benchmark number is almost always higher than what the agent will show on your code. The best source of tasks is closed tickets and real agent failures from the log.

Composition of the set

• 20–50 tasks to start. Fewer and the noise swallows the difference between versions. More and a run becomes expensive, and people stop running it.
• Different types: a new feature, a bug fix, a refactoring, a task in someone else's module, a trap task (where the right answer is "you must not do it this way" or "a clarification is needed").
• Every task that failed in real work becomes a test. This way the set grows together with the understanding of where the agent is weak, and regressions do not return unnoticed.
• A fixed starting state: a specific commit, pinned dependencies, an identical environment. Otherwise you measure environment drift, not the agent.

Task format

Each task is a description, a starting state and a success criterion that can be checked without a human.

id: add-pagination
repo_ref: 3f2c1ab
prompt: |
  Add pagination to GET /payments: the parameters limit (default 20, maximum 100)
  and cursor. The response contains next_cursor.
check:
  - pytest tests/test_payments_pagination.py
  - mypy app/
limits:
  max_changed_files: 6

Note that the success criterion is the Definition of Done, written down as commands. If you cannot write a criterion for a task, that is a sign the task is also ambiguous for the agent.

Automated checks or an LLM judge

The result can be checked by code or by another model. Each approach has its own zone.

Rule: code first, the judge on top

Whatever can be checked deterministically, check deterministically. Add an LLM judge only for properties that code cannot express, and never instead of tests.

How to make an LLM judge useful

• Separate context from the generator. The judge should not see the agent's reasoning, only the task, the criteria and the final diff. Otherwise it will repeat the same mistakes: this is N-version checking, two independent views instead of one.
• A rubric instead of "rate it on a ten-point scale". Concrete yes/no questions: "is there validation for limit > 100", "were files outside app/payments changed".
• Calibration on manual labels. Label 20–30 results by hand and compare them with the judge's verdicts. If agreement is low, change the rubric; do not trust the judge more.
• Asymmetric errors. A false "done" is more expensive than a false "not done": the first goes to production, the second costs one restart. Tune the judge to reject when in doubt.

Which metrics to track besides the share of successes

The share of solved tasks is only the top layer. To understand where exactly to fix things, you need reliability metrics from classical engineering.

• MTBF — how many steps or tasks the agent completes before getting stuck. It shows how rarely it breaks.
• MTTR — how much time it takes to return to a working state after a failure. Reducing it is almost always cheaper than raising MTBF: a good progress file and a reproducible environment shorten recovery several times over.
• Blast radius — how much code one failed step damages. It is limited by a diff size cap and by isolation of the working copy.
• Share of false "done" — tasks where the agent declared success but the check failed. The most dangerous metric: it shows whether the agent's report can be trusted at all.
• Cost and time — tokens and minutes per task. A harness that raised success by 5% but doubled the cost does not always win.

A failure journal: evals from everyday work

Not every team is ready to build a runner right away. The minimal version of evaluation is a failure journal with three columns.

| Task                    | Result                             | Layer         |
|-------------------------|------------------------------------|---------------|
| pagination /payments    | "done", 2 tests red                | verification  |
| webhook timeout fix     | 30 min on pip install              | environment   |
| refunds refactoring     | old ORM syntax                     | context       |

A layer is one of five places where an agent breaks: the task specification, the context, the execution environment, the verification feedback, and the state management. After twenty entries the real bottleneck becomes visible, and almost never where you expected it. Each row of the journal is a candidate for the evals task set.

Why a stronger model does not save you

The first reaction to agent failures is to pick a more expensive model. Sometimes this helps, but more often the gap between the benchmark and your reality is closed not by the model but by the environment around it.

If the agent spends half an hour setting up the environment, that is the environment layer, and a flagship model will spend the same half hour on it. If the agent says "done" with red tests, that is the verification layer: it has no command that tells it the truth. If it writes code in someone else's style, the project's conventions were never given to it. None of these problems is about the model's intelligence.

Evals let you check this honestly. Run the set with the current model and with a more expensive model, then with the current model and an added AGENTS.md with the check commands. Often the second change gives more gain than the first, at zero cost. But do not take this on trust: such comparisons are exactly what the set is for.

Step-by-step plan: evaluating AI agents in a week

1. Day 1. Start a failure journal and record every agent failure in it, with its layer.
2. Day 2. Pick 20 tasks from closed tickets and the journal. For each, fix the commit and write the success criterion as commands.
3. Day 3. Write a runner: a clean working copy, launching the agent with the task prompt, running the checks, writing the result to JSONL.
4. Day 4. Run each task 3–5 times. Calculate pass@1, pass^3, the share of false "done", and the average cost.
5. Day 5. Change one thing in the harness and run again. Compare only with the same set and the same number of repeats.

Change one variable at a time. If one experiment changes the model, the prompt and the tools together, you will not know what worked.

Common mistakes in evaluating agents

• One run per task. The agent's non-determinism makes such a result a lottery. Minimum three repeats.
• The set changes between comparisons. You added new tasks and changed the prompt at the same time: the numbers are not comparable.
• The agent sees the verification tests. If the tests live in the repository, the agent may tune the code to them or simply change them. Keep hidden checks outside the working copy, and verify that the agent did not touch the test files.
• The agent's report instead of a check. "All tests passed" in the model's answer is not a result. The result is the exit code of the command run by the runner.
• Only pass@k. A pretty number for a presentation and a poor forecast for autonomous work.

Summary

AI agent evals are a fixed set of real tasks, automatic checks, and metrics that are recalculated after every change to the harness. pass@k shows what the agent is capable of; pass^k shows whether it can be trusted without supervision. Automated checks give the truth; an LLM judge with a rubric and separate context supplements them where code is powerless. And a failure journal across the five layers shows what to fix first, and most often it is not the model. Metrics of reliability and the diagnostic loop are covered in practice in the harness engineering course.