Agent evals
An agent is a model that takes actions, like calling tools, over many steps. Agent evals check the end result, how reliably it is reached, and whether the steps followed the rules.
- One successful run says little, because an agent can succeed on Monday and fail on Tuesday.
- pass^k asks whether all k runs of a task succeed, which measures reliability.
- The same one-run score can hide very different reliability.
- Checking only the end state can pass a run that broke a rule on the way.
Try this: drag k to 8 and compare the three task sets.
Try it
What just happened
- Each tile is one task: the number of runs, out of , in which the agent succeeded. All three task sets total 50 successes in 80 runs, so all three score 62.5% on a single run.
- The orange bar, pass@k, asks whether any of k runs works. It climbs as k grows: with “Every task works 5 times in 8” it reaches by k = 4. That suits code with tests, where you can try again and check the answer.
- The grey bar, pass^k, asks whether all k runs work. Pick “Every task works 5 times in 8” and it falls to at k = 8, because every task fails about 3 runs in 8. Pick “Tasks either always work or never” and pass^8 stays at .
- The average one-run score hides this. What counts is how the failures are spread across tasks, which a single success rate does not show.
Show the formula
Run each task n times, count the c runs that succeed, and average over tasks. This is the estimator from the τ-bench paper.
pass^k is the chance that k runs drawn without replacement are all successes. It is the mirror of the pass@k formula on the pass@k page.
Where you meet it
A shop wants an agent that handles refunds. It scores 61% on a test of customer requests. The shop asks for the chance that it gets a given request right every time it is tried.
- τ-bench puts an agent in simulated conversations with a user played by a language model, gives it tools and policy rules, and compares the database at the end with the goal state. The paper reports gpt-4o succeeding on about % of the retail tasks and % of the airline tasks in a single run, and on only about % of retail tasks in all 8 runs.
- SWE-bench scores an agent that edits a real code base by whether the project’s tests pass afterwards.
Where it breaks
Checking the end state is cheap and objective, and it cannot see how the agent got there. Two runs end with the right refund in the database. In one the agent skipped an identity check. Tick the box to grade the steps too. These runs are made up to show the gap.
With only the end state checked, the lucky run is graded . With steps graded it is . A run that gets the right result by breaking a rule is a bug that has not yet cost anything. Add checks on the steps that matter, such as policy rules, or have a judge read the whole trajectory.
Check yourself
A task works in 6 of 8 runs. What is its pass^2 and its pass@2?
pass^2 = C(6,2)/C(8,2) = 15/28 = 0.54. pass@2 = 1 − C(2,2)/C(8,2) = 1 − 1/28 = 0.96. Two tries usually find a success, but two in a row work only about half the time.
An agent has a one-run success rate of 60% on every task, independently. What is pass^8?
About 0.6 to the power 8, which is 1.7%. In practice agents are rarely independent across runs of the same task, which is why real pass^8 is higher, but still far below pass^1.
Why use pass@k for code and pass^k for customer service?
Code can be checked by running tests, so a failed try costs little and one success is enough. A customer cannot be asked to retry: the agent has to be right every time.
Used in
- Loading…
Go deeper
- Yao, Shinn, Razavi and Narasimhan, “τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains” (2024). The pass^k metric and the numbers quoted here.
- Jimenez et al., “SWE-bench” (2023). Agents scored by tests on real repositories.
- pass@k on this site. The estimator this page mirrors.