The map of evals
An eval is a test for an AI system. This is the map of the series: which eval fits which question.
- First decide what you are grading: a search step, an answer, or an agent’s whole run.
- Then decide who grades: a fixed rule, a program, a language model, people, or real users.
- Every eval has a blind spot, shown live on its page.
- Whatever you pick, ask how far to trust the number.
Try this: pick A language model and see which evals it can grade.
Try it
What just happened
- The three cards are the three things you can grade. Each lists the eval pages that apply, with a line on when to use each one. There are pages in all.
- The filter dims the evals that your choice of grader cannot do. A language model can grade of them, from RAG context to open answers. Pick “A program that runs” and only the code and agent checks stay lit.
- A fresh reader can take this path. Start with “Is 82% better than 80%?”, because every later number needs error bars. Then read the grading you need: exact match for short answers, pass@k for code, LLM as judge and pairwise votes for open answers, and the retrieval pages for search.
- Finish with the pages on agents, online tests and contamination. They are about trust in the whole eval rather than in one metric.
Where you meet it
A team ships a support bot. They check retrieval with recall@k, grounding with a RAG eval, wording with a judge model, and the whole thing with an A/B test. No one of these tells them the bot is good.
- Most real eval suites combine a cheap offline check, such as a rule, a program or a judge model, with a slower online check on real users.
- Each page here quotes the paper or reference it is checked against, so you can read further.
Where it breaks
No single eval is enough. This table lists what each one cannot see. The blind spot is shown live on the page it belongs to.
When two evals disagree, one of these blind spots is usually the reason. Look the disagreement up in this table first.
Check yourself
You have a code model and a test suite for each problem. Which eval fits, and what should you watch for?
pass@k, which runs the tests. Watch the tests themselves: weak tests let wrong code pass, so the score can rise without the model improving.
You want to compare two chat models on open questions, with no reference answers. What are your options?
Pairwise votes from people or a judge model, ranked with Elo or Bradley-Terry. Check a judge model for position and length bias, and check people for agreement with kappa.
Your offline eval says a change helped by 2 points on 300 questions. What do you do before you ship?
Ask whether 2 points is more than noise with 300 questions (it probably is not), then confirm on real users with an A/B test sized in advance.
Used in
- Loading…
Go deeper
- Is 82% better than 80%? The best first page: error bars for every score.
- Zheng et al., “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena” (2023). A paper that touches several of these evals: judges, pairwise votes and human agreement.
- Miller, “Adding Error Bars to Evals” (2024). Statistics for eval scores.