# Evaluating a RAG System: Retrieval and Answers

> Measure retrieval and generation separately: recall@k, MRR, groundedness and answer correctness, with an evaluation set you build from real questions.

- Source: https://akshaythakare.com/notes/rag/evaluating-rag
- Topic: RAG
- Level: advanced
- Reading time: 2 min
- Tags: rag, evaluation, llm, testing
- Updated: 2026-10-11

## Two systems, two sets of metrics

If the right passage is never retrieved, no prompt can fix the answer. Evaluate retrieval first, then generation.

## Build the evaluation set

For 50–200 real or realistic questions, record:
- the question
- the document chunk(s) that contain the answer
- a reference answer, or the key facts it must contain
- some questions whose answer is **not** in the corpus

## Retrieval metrics

| Metric | Meaning |
| --- | --- |
| **Recall@k** | Share of questions where a correct chunk is in the top k |
| **MRR** | Average of 1 / rank of the first correct chunk |
| Precision@k | Share of the top k that are relevant |

```python
def recall_at_k(results, gold, k=5):
    hits = sum(any(doc in gold[q] for doc in docs[:k]) for q, docs in results.items())
    return hits / len(results)
```

## Answer metrics

| Metric | Checks |
| --- | --- |
| **Groundedness / faithfulness** | Every claim is supported by the retrieved context |
| **Correctness** | Matches the reference answer's key facts |
| **Refusal accuracy** | Says "not found" when the answer is not in the corpus |

Model-based grading can score these at scale, but spot-check its judgements by hand and keep the grading rubric specific.

## Use it like a test suite

Run the evals on every change: chunk size, embedding model, retriever, reranker or prompt. Keep the change only if the metrics improve without breaking refusals.

> **Warning: Do not tune on your test set**
>
> Hold back part of the questions. Otherwise you improve the score, not the system.