Precision, recall and reciprocal rank

These three scores grade a search result list when each result is simply relevant or not. They are the usual first checks for search and for the retrieval step of RAG.

Precision@k asks how many of the top k results are relevant.
Recall@k asks how many of all the relevant pages made it into the top k.
Reciprocal rank asks how early the first relevant page appears.
Each score ignores something the others see.

Try this: drag “Reset your password” up to rank 1 and see which numbers change.

Try it

What just happened

Look at the top rows above the dashed line. of them are marked relevant. Precision@ divides that by , which gives .
Recall@ divides the same count by R, the number of relevant pages in the whole collection, here, which gives . Recall needs R, and R is a number you must know or estimate.
Reciprocal rank is 1 divided by the rank of the first relevant page: . Moving the best page to rank 1 lifts it to , but precision@ stays at , because both pages are still inside the top .
Averaged over many queries, reciprocal rank becomes mean reciprocal rank (MRR). For a ranking where a result can be better or worse, not only relevant or not, see the nDCG@k page.

Show the formulas

precision@k = (relevant results in the top k) / k recall@k = (relevant results in the top k) / R reciprocal rank = 1 / (rank of the first relevant result), or 0 if none MRR = average of reciprocal rank over all queries

Example for MRR: three queries whose first correct answers come at ranks 3, 2 and 1 give (1/3 + 1/2 + 1) / 3 = 0.61.

Where you meet it

A help-centre search returns ten pages. Support wants to know whether the right page is near the top, and whether a customer who reads the first five has seen everything useful.

Precision@k suits a screen that shows only k results, such as the top five on a phone.
Recall@k suits a retrieval step that feeds a later step, such as the passages a RAG system hands to a language model: a page that is not retrieved cannot be used.
Reciprocal rank suits a question with one right answer, such as a lookup or a navigation search.

Where it breaks

Precision@k assumes there are at least k relevant pages. When the collection holds fewer, even a perfect system cannot score 1. Pick how many relevant pages exist, and look at a perfect list.

With one relevant page, a perfect list scores on precision@10. With three, . The system is perfect in both cases, so compare precision only between lists of the same query, or use a score that knows the best case, like nDCG.

Check yourself

A list has 3 relevant pages in the top 10, out of 12 in the collection. What are precision@10 and recall@10?

Precision@10 = 3/10 = 0.3. Recall@10 = 3/12 = 0.25.

You swap two relevant pages that are both inside the top k. Which of the three scores can change?

Only reciprocal rank, and only if one of them was first. Precision@k and recall@k count what is inside the top k and ignore the order.

Three queries have their first relevant result at ranks 1, 2 and 4. What is the MRR?

(1 + 1/2 + 1/4) / 3 = 1.75 / 3 = 0.58.

Used in

Loading…

Go deeper

Wikipedia: Evaluation measures (information retrieval). Precision, recall and their cut-off versions, with the known weaknesses of precision@k.
Wikipedia: Mean reciprocal rank. The definition and a worked example, and why only the first relevant result counts.
nDCG@k on this site. The score for graded relevance, which also rewards putting good results early.