Hybrid Search and Reranking for Better RAG

Vector search misses exact matches

Embeddings capture meaning, so "refund policy" finds "returns and money back". But they can miss exact tokens: product codes, error codes, names and acronyms such as ERR-4012 or SKU100045.

Hybrid search

Run both and merge:

A common merge is Reciprocal Rank Fusion: each document scores the sum of 1 / (k + rank) over the lists it appears in (k is often 60).

def rrf(*rankings, k=60):
    scores = {}
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking, start=1):
            scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)

Reranking

Retrieve broadly (say the top 50), then score each candidate against the question with a cross-encoder reranker, which reads question and passage together. Keep the top 5 for the prompt.

question → hybrid retrieve top 50 → rerank → top 5 → LLM

Rerankers are slower per document than embeddings, which is why they run only on a short candidate list.

Also use metadata

Filter by permissions, product, language or date before ranking. It is cheaper and prevents leaking documents a user should not see.