nDCG@k
nDCG@k is a score from 0 to 1 for how good a ranked list of search results is. Good results near the top earn more.
- A good result near the top is worth more than the same result lower down.
- Each result earns its grade, made smaller by its rank.
- nDCG is that total divided by the best total the same results could earn.
- A score of 1 means no better order exists.
Try this: drag the grade 3 results to the top and watch nDCG climb. Then lower k.
Try it
What just happened
You searched for “fix a flat bike tire”, and people graded each result from 0 to 3. Three ideas turn those grades into one number between 0 and 1.
Show the formula
The best possible order puts the highest grades first. Some texts use 2grade − 1 in place of the grade, which makes a perfect result worth much more than a good one. This page uses the grade itself.
Where you meet it
A team changes how its search ranks results. They replay 500 saved searches, have people grade the top results, and compare the average nDCG@10 before and after the change. Because every score falls between 0 and 1, searches with many good results and searches with few can be averaged together.
- The scikit-learn library has an
ndcg_scorefunction. - Järvelin and Kekäläinen introduced the idea in 2002, as a way to score lists where results have more than two levels of “relevant”.
Where it breaks
Judges grade only some results. A result with no grade counts as 0, even when it is the best page on the web. Here a new system finds a new page nobody has graded.
The fix is more grading, or grading the new system's top results before you compare. A score from a list with many “?” marks tells you less than it seems to.
Check yourself
You move a perfect result from rank 5 to rank 1. Does the “best possible” number change?
No. The best possible score depends only on the grades and on k, not on the order you chose. Only your own DCG changes. Try it: the blue number stays put while the orange one moves.
What is nDCG when no result has a grade above 0?
The best possible score is 0, so the fraction is 0 divided by 0. Tools handle it in their own way. This page, and most libraries, return 0. Some evaluations skip that search instead.
Set k to 1. What does nDCG tell you?
It compares the grade of your first result with the highest grade anywhere in the list. With a top grade of 3, a first result graded 1 gives 1 divided by 3, about 0.33.
Used in
- Loading…
Go deeper
- Järvelin and Kekäläinen, “Cumulated gain-based evaluation of IR techniques” (ACM TOIS, 2002). The paper that introduced DCG and nDCG.
- Wang, Wang, Li, He, Chen and Liu, “A Theoretical Analysis of NDCG Type Ranking Measures” (2013). A study of how well nDCG can tell one ranking method from another.
- scikit-learn:
ndcg_score. A common implementation, with notes on ties and options.