RAG evals

RAG finds passages and asks a language model to answer from them. A RAG eval says which of the two steps went wrong.

A RAG answer can be wrong because retrieval found the wrong passages.
It can be wrong because the model ignored good passages and made things up.
Faithfulness asks whether the context supports each claim.
An answer can be faithful and still wrong.

Try this: untick “Old notice” and watch Answer B.

Try it

What just happened

The retriever returned the passages that are ticked. Context relevance is the share of them that are needed to answer the question: here, because the old notice is not needed. Context recall asks whether the needed passage was found at all: .
Faithfulness is the share of an answer’s claims that the retrieved context supports. Answer A makes one claim and the current notice supports it. Answer C adds an invented claim about coffee, so its faithfulness is .
Answer B is faithful, , to the old notice, and wrong. Faithfulness only checks the answer against the context it was given. It cannot tell you the context was stale.
Answer D is faithful to a passage about books, but answers a different question. That is a failure of relevance, which is a separate score.

Show how RAGAS computes these

faithfulness = supported statements / all statements in the answer context relevance = sentences needed to answer / all sentences in the context answer relevance = average similarity between the question and questions generated from the answer

RAGAS has a language model split the answer into statements and check each one against the context, and extract the needed sentences from the context. For answer relevance it generates questions from the answer and compares them to the real one with embeddings. This page uses one-sentence passages and labels done by hand, so the first two scores can be computed exactly. Answer relevance does not look at whether the answer is true, so an answer can be relevant and wrong.

Where you meet it

A support bot answers “When does the library reopen?” with a date from an old notice. A retrieval score says the search found something relevant. A faithfulness score says the answer matches what it found. Neither says the date is out of date.

The RAGAS paper names three things to grade: whether the retriever finds relevant and focused passages, whether the model uses them faithfully, and the quality of the generation itself. Its scores do not need human-written reference answers.
The retrieval half can also be graded with the usual ranking scores, such as recall@k on the retrieval metrics page.

Where it breaks

Faithfulness is measured against whatever the retriever returned. If only the old notice comes back, the right answer is not supported and the wrong one is. Here the retriever returned only the old notice.

The right answer scores for faithfulness and the wrong one . Context recall is : the needed passage was never retrieved. When faithfulness and correctness disagree, look at the retrieval step first.

Check yourself

An answer makes 4 claims and the context supports 3. What is its faithfulness?

3 / 4 = 0.75. One claim is not in the context, which means the model added it.

A model answers correctly from its own memory, without using the retrieved passages. What happens to faithfulness?

It drops, because the claims are not supported by the context. The answer may be right, but it is not grounded in what was retrieved, which matters when the context is what you trust.

Faithfulness is 1.00 and the answer is wrong. Where do you look first?

At the retrieval step. If the context was stale, off-topic or incomplete, a faithful model will repeat its mistakes. Check context relevance and recall.

Used in

Loading…

Go deeper

Es, James, Espinosa-Anke and Schockaert, “Ragas: Automated Evaluation of Retrieval Augmented Generation” (2023). Defines the three scores used here.
Precision, recall and reciprocal rank on this site. The retrieval half, as a ranked list.