Retrieval metrics
Lesson 8 of 10
8Retrieval metrics
The big pictureA RAG system has two stages that fail for different reasons: retrieval may not find the right document, and generation may ignore it or invent on top of it. Measure each alone. Retrieval metrics (recall, precision, MRR) tell you where the problem is before you blame the model, and faithfulness tells you whether the model stuck to what it found.
What the picture shows: Hybrid raised recall at 3 from 0.70 to 1.00 and MRR from 0.67 to 0.90. But look behind the average: on the fifth question the right one dropped from first to second, which is why you read the questions that got worse with every comparison.
Map of this part
The concepts in order, click any one to jump to it:
Two ways into the same idea
To know whether search is good you ask three questions: did it fetch the right passage at all? How much of what it fetched is useful? How high did the right one rank? Then the key question: is the final answer actually built on the source, or did the model add its own?
Recall@k is the share of questions whose source appears in the top k, Precision@k the share of useful results among them, and MRR the mean reciprocal rank of the first correct one. faithfulness checks that every claim in the answer is supported by the retrieved passages.
Before You Start
📏Metrics
You know precision and recall from classification, and here they are the same two ideas on a ranked list of results.
📐Evaluation first
You have a golden set, and all you add now is the right documents for each question.
The Core Idea
A RAG system has two stages that fail for different reasons. In this lesson you measure each alone: recall, precision and RR for one question, then MRR to compare two retrievers, then the faithfulness of the answer.
1Two stages, a number for each
Retrieval may not find the right document, and generation may ignore it or invent on top of it. If you measure only the final answer, you will not know which one failed.
Look at the picture: three numbers you measure in this order, moving on only once the previous one is good.
2Recall, precision and RR
Recall at k: how many of the correct documents appear in the top k. Precision at k: how many of the top k are correct. And RR is 1 / the rank of the first correct result.
Look at the picture: the question has two correct documents, the system returned five in order, and the correct two sit at ranks 2 and 4.
def recall_at_k(retrieved, relevant, k):
return len(set(retrieved[:k]) & relevant) / len(relevant)
def precision_at_k(retrieved, relevant, k):
return len(set(retrieved[:k]) & relevant) / k
def reciprocal_rank(retrieved, relevant):
for rank, doc in enumerate(retrieved, start=1):
if doc in relevant:
return 1 / rank
return 0.0
retrieved = ["d7", "d2", "d9", "d4", "d1"]
relevant = {"d2", "d4"}
print("recall@3: ", round(recall_at_k(retrieved, relevant, 3), 2))
print("recall@5: ", round(recall_at_k(retrieved, relevant, 5), 2))
print("precision@3:", round(precision_at_k(retrieved, relevant, 3), 2))
print("RR: ", reciprocal_rank(retrieved, relevant))
Two correct documents among five retrieved. In the top 3 we found one of the two: recall 0.5. In the top 5 we found both: recall 1.0. Precision at 3 is 0.33 because only one of the three is correct. And the first correct result is at rank two, so RR is 1/2.
3MRR and comparing two retrievers
MRR is the average RR across all questions. It rewards putting the right one on top, and matters when you send the model only one or two chunks.
Five questions and the results of two systems for each. We compute mean recall at 3 and MRR.
golden = {"q1": {"d1"}, "q2": {"d4", "d5"}, "q3": {"d9"}, "q4": {"d2"}, "q5": {"d7"}}
runs = {
"vector": {"q1": ["d1", "d3", "d2"], "q2": ["d5", "d8", "d6"], "q3": ["d2", "d6", "d9"],
"q4": ["d3", "d1", "d8"], "q5": ["d7", "d5", "d1"]},
"hybrid": {"q1": ["d1", "d2", "d3"], "q2": ["d4", "d5", "d8"], "q3": ["d9", "d2", "d6"],
"q4": ["d2", "d3", "d1"], "q5": ["d5", "d7", "d1"]},
}
def rr(ranked, rel):
return next((1 / i for i, d in enumerate(ranked, 1) if d in rel), 0.0)
for name, run in runs.items():
recall = sum(len(set(run[q][:3]) & rel) / len(rel) for q, rel in golden.items()) / len(golden)
mrr = sum(rr(run[q], rel) for q, rel in golden.items()) / len(golden)
print(f"{name:7} recall@3 = {recall:.2f} MRR = {mrr:.2f}")
Hybrid raised recall from 0.70 to 1.00 and MRR from 0.67 to 0.90. Look at the fifth question: hybrid put the right one second instead of first, losing a little MRR there and gaining elsewhere. That is why you look at the average and at the questions that got worse together, not at the average alone.
4Faithfulness
A correct answer with no source is still a problem: the model guessed it. Faithfulness is the share of answer sentences supported by the retrieved documents.
Look at the picture: an answer of three sentences, and how much of each sentence appears in the source.
For each answer sentence, how many of its words appear in the source? A rough check, but it catches outright invention without any model.
import re
source = ("يحق للموظف إجازة سنوية مدتها واحد وعشرون يوم عمل. "
"تقدم الطلبات قبل أسبوعين. يمكن ترحيل عشرة أيام إلى السنة التالية.")
answer = ("لك واحد وعشرون يوم عمل إجازة سنوية. "
"ويمكنك ترحيل عشرة أيام إلى السنة التالية. "
"ويمكنك صرف رصيد الإجازة نقدا عند الاستقالة.")
def words(t):
return set(re.findall(r"[ء-ي]+", t))
src_words = words(source)
for sentence in filter(None, (s.strip() for s in answer.split("."))):
w = words(sentence)
support = len(w & src_words) / len(w)
verdict = "supported" if support >= 0.6 else "NOT IN THE SOURCE"
print(f"{support:.0%} {verdict:17} {sentence}")
The first two sentences are supported, and the third, about cashing out the balance, has no source: the model invented it. It may even be true in reality, but the system cannot back it up, which is enough to count it as a problem. In production, a model judge asking per sentence "does the text support this?" replaces this check, calibrated as in the evaluation-first lesson.
How It Works, Step by Step: how to find where a RAG system fails
For each golden question, write down the documents or chunks that contain its answer.
Measure recall at k, where k is what you actually send; if it is below about 0.8, fix retrieval first.
Once recall is good, measure faithfulness: does every sentence in the answer have a source in the documents?
Finally measure answer correctness, and record the four numbers and the questions that got worse in the experiment table.
Worked example
For each golden question, write down which documents (or chunks) contain its answer. That is all you need to compute retrieval metrics, without any language model.
Try it without code
Myth vs Reality
| Myth | Reality |
|---|---|
| The final answer alone is enough to evaluate the system. | If it is wrong you will not know whether retrieval or the model failed, so you change the model while the right document never reached the context. |
| A correct answer is fine even if no document supports it. | The model answered from its own knowledge, and may answer just as confidently when that knowledge is outdated or wrong. It is an unfaithful answer. |
| High recall at 20 means a good system. | What matters is the k you actually send. If you send one chunk, what matters is MRR or recall at 1. |
Try it yourself
You take only one chunk into the context. Which metric matters most?
If the right one is not first, the model never sees it. Recall at 20 could be excellent while your system fails.
An answer that is true in reality, but nothing in the retrieved documents supports it. How do you grade it?
A RAG system promises to answer from your documents. An answer without a source means it may answer just as confidently when its knowledge is outdated or wrong.
Quick review cards
1 / rank of the first correct result across questions. Rewards putting the right one on top, and matters when you take few results.Check Your Understanding
Recall at 5 is 0.95, yet answers are wrong 30% of the time. Where do you look?
In generation, not retrieval: the right document is usually in the context, and the model does not use it well. Check faithfulness, the chunk ordering, the template instructions, and whether low precision drowns the right chunk among misleading ones. And make sure you really send 5 chunks, not 3.
In the simple faithfulness check, why can the sentence about cashing out leave be rejected even if it is true in reality?
Because faithfulness measures support from the retrieved documents, not truth in the world. It had no source (0%), so the system cannot back it up, which is enough to count it as a problem.
Resources for this lesson
Grouped by type. Start with the official docs for precision, or an article or video for the explanation.
Official docs2
Done with this lesson?
Sign in to save your progress.