Evaluating a RAG System: Retrieval and Answers

Measure retrieval and generation separately: recall@k, MRR, groundedness and answer correctness, with an evaluation set you build from real questions.

RAGAdvanced2 min readUpdated 11 Oct 2026

~/notes/rag/evaluating-rag

Two systems, two sets of metrics

If the right passage is never retrieved, no prompt can fix the answer. Evaluate retrieval first, then generation.

Build the evaluation set

For 50–200 real or realistic questions, record:

the question
the document chunk(s) that contain the answer
a reference answer, or the key facts it must contain
some questions whose answer is not in the corpus

Retrieval metrics

Metric	Meaning
Recall@k	Share of questions where a correct chunk is in the top k
MRR	Average of 1 / rank of the first correct chunk
Precision@k	Share of the top k that are relevant

def recall_at_k(results, gold, k=5):
    hits = sum(any(doc in gold[q] for doc in docs[:k]) for q, docs in results.items())
    return hits / len(results)

Answer metrics

Metric	Checks
Groundedness / faithfulness	Every claim is supported by the retrieved context
Correctness	Matches the reference answer's key facts
Refusal accuracy	Says "not found" when the answer is not in the corpus

Model-based grading can score these at scale, but spot-check its judgements by hand and keep the grading rubric specific.

Use it like a test suite

Run the evals on every change: chunk size, embedding model, retriever, reranker or prompt. Keep the change only if the metrics improve without breaking refusals.

Finished this note?

LinkedIn X WhatsApp