pass@k
pass@k scores a code model by letting it try a problem k times and asking whether any try passes the tests. HumanEval and the Codex paper made it the standard code metric.
- A code eval runs the model’s program against unit tests, so a try either passes or fails.
- pass@k asks whether any of k tries passes, so more tries can only help.
- It is estimated from n samples with a count, not by running exactly k tries once.
- The reverse question, do all k tries pass, shows how reliable the model is.
Try this: drag k up to 20 and watch pass@k climb while “all pass” collapses.
Try it
What just happened
- The model wrote programs for one problem and passed the tests. pass@1 is the share that pass, .
- For pass@, pick of the samples at random without putting any back. The orange bar is the chance that at least one of them is a passing one: , even though each single try passes only of the time.
- The thin blue bar is the shortcut 1 − (1 − pass@1)k. It pretends the tries are independent, and it always reads lower, here . The paper calls it a consistent underestimate.
- The grey bar asks the opposite: do all k tries pass? At k = that is . As k grows pass@k heads to 100% and “all pass” heads to 0%.
Show the formula
Generate n samples per problem, count the c that pass the tests, then average this over problems.
C(a, b) is the number of ways to choose b items from a. The fraction is the chance that k samples drawn without replacement are all failures, so one minus it is the chance of at least one pass. When fewer than k samples fail, pass@k is 1. The paper uses n = 200 and k up to 100, and computes the fraction as a running product so the numbers stay small.
Where you meet it
A team says its code model “solves 70% of problems”. The first question to ask is how many tries it got.
- The Codex paper introduced HumanEval, 164 hand-written problems with an average of 7.7 unit tests each, and reports pass@k from n = 200 samples. Its abstract says the model solves 70.2% of problems with 100 samples per problem.
- SWE-bench scores a model by whether the tests pass after it edits a real code base, across 2,294 tasks from GitHub issues.
- τ-bench borrows the idea and flips it. Its pass^k counts a task only when all k runs succeed, which suits a customer-service agent that must work every time.
Where it breaks
pass@k trusts the tests. If the tests are weak, a wrong program passes and the score climbs for the wrong reason. Here are four solutions to “return the largest item in a list”. Only one is right. The functions run for real against the tests you tick.
With one test, of 4 solutions pass, so pass@1 reads . With all four tests it is . The model did not improve; the tests got stronger.
Check yourself
n = 10 samples, c = 3 pass. What is pass@2?
The chance that two samples drawn without replacement are both failures is C(7,2) / C(10,2) = 21/45. So pass@2 = 1 − 21/45 = 24/45, about 0.53. The naive shortcut gives 1 − 0.7² = 0.51.
A model solves one problem every time and never solves another. What is its average pass@5 over the two?
0.5. Each problem gets its own estimate (1 and 0) and the estimates are averaged. More tries cannot help a problem the model never solves.
Why not run exactly k tries once and report whether any passed?
That is one noisy draw. The paper notes this way of computing pass@k can have high variance. Drawing n ≥ k samples and counting uses all of them and gives an unbiased estimate with less noise.
Used in
- Loading…
Go deeper
- Chen et al., “Evaluating Large Language Models Trained on Code” (2021). Introduces HumanEval and the unbiased pass@k estimator; Appendix A shows why the naive shortcut is biased.
- Yao, Shinn, Razavi and Narasimhan, “τ-bench” (2024). Defines pass^k, the chance that all k runs succeed.
- Jimenez et al., “SWE-bench: Can Language Models Resolve Real-World GitHub Issues?” (2023). A code eval scored by running real tests on real repositories.