Is 82% better than 80%?
Error bars show how far an eval score could move if you had asked different questions. Good eval reports print them next to every score.
- An eval score comes from a sample of questions, so another sample would give another score.
- The range around a score shrinks with the square root of the number of questions.
- A gap between two models counts only when its own range stays away from zero.
- Grading both models on the same questions makes that range smaller.
Try this: drag n up to 1,000 and watch the gap’s range leave zero.
Try it
What just happened
- Each model card shows the share of questions answered right, and a bar for its 95% range: the score plus or minus 1.96 standard errors. With questions the standard error of model A is , so its score could sit anywhere from to .
- The gap card subtracts the two scores. Its range is wider than either model’s alone, because both scores are uncertain: it is points, give or take . The range runs from to and includes 0, so the page cannot say B is better.
- Tick “paired”. Both models answer the same questions, so a question both get right or both get wrong adds nothing to the gap. The more the models agree on which questions are hard, the narrower the range.
- At 1,000 questions the unpaired range is to , which is above 0. The last card shows how many questions this gap needs: about unpaired, or paired at agreement .
Show the formulas
Each question scores 1 (right) or 0 (wrong). The score is the mean, p. The formulas below are from Miller (2024).
With significance 5% and power 80%, z_α/2 + z_β is 1.96 + 0.84 = 2.80. This page assumes the models answer the same way every time, so the only noise is which questions were picked.
Where you meet it
A team reports that model B scores 82% and model A 78% on a 200-question eval, and ships B. The numbers look clear. The page above says the gap could be anything from to points.
- Miller recommends printing the standard error under each eval score, as other sciences do.
- When questions come in groups, such as several questions about one passage, he reports that clustered standard errors on two popular evals were over 3 times larger than the naive ones.
- A worked example in the paper needs about 969 questions to see a 3-point gap 8 times out of 10, and concludes that new evals should have at least 1,000 questions.
Where it breaks
The simple range assumes the score could land on either side. A model that answers every question right has nothing above it, so the formula gives it a range with no width, as if the test were perfect. Pick a perfect score and compare it with the Wilson range, which keeps the width.
A perfect score on a test this small means little. The Wilson range for 20 of 20 starts at . The fixes are more questions, or a range that does not assume a bell curve.
Check yourself
Model A scores 70% on 100 questions. About how wide is its 95% range?
The standard error is √(0.7 × 0.3 / 100) = 0.046, and 1.96 times that is about 0.09. The range is about 61% to 79%, nine points either side.
You double the number of questions. How much narrower does the range get?
The range shrinks with the square root of n, so it gets about 29% narrower (multiply by 1/√2, which is 0.71). To halve it you need four times the questions.
Two models, both near 80%, differ by 3 points on 500 questions. Is that enough?
No. Unpaired, the standard error of the gap is √(0.32 / 500) = 0.025, so the range is about ±5 points. Paired at agreement 0.5 it is √(0.16 / 500) = 0.018, about ±3.5 points. Both ranges include 0.
Used in
- Loading…
Go deeper
- Miller, “Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations” (2024). Every formula on this page. It also covers resampling answers and next-token probabilities to cut variance.
- Wikipedia: Binomial proportion confidence interval. The Wilson interval and why the simple one fails near 0% and 100%.