RAG Assistant That Answers From These Notes, With Citations

The problem

This site has dozens of technical notes. A reader with a specific question — "why does my join double revenue?" — has to search, skim and connect several notes. An assistant could answer directly, but only if every answer is traceable to the notes and it admits when the notes don't cover something.

What this build will deliver

• An ingestion job that turns notes into searchable chunks.
• Hybrid search (vectors plus keywords) in Qdrant.
• An answer API that cites the exact note sections it used.
• An evaluation set that runs on every change.

Success looks like

• The right section appears in the top 5 results for most evaluation questions.
• Answers contain no claims that aren't supported by the cited text.
• Questions outside the notes get "the notes don't cover this", not a guess.

Key design decisions

Chunk by headings, not fixed sizes. Notes are already structured; one section usually holds one idea. Long sections are split with overlap.

Store the citation with the chunk. Each chunk carries note URL#section-anchor, so answers can link to the exact paragraph.

Hybrid retrieval. Vectors find paraphrases; keywords find exact terms like QUALIFY or COPY INTO. Results are merged, then reranked.

Grounded prompt with an exit. The model is told to answer only from the provided sections and to say so when they don't contain the answer.

Incremental indexing. Only notes whose updatedAt changed are re-embedded; the embedding model version is stored so a model change triggers a full rebuild.

1. 1

   Evaluation set first

   Write 40 real questions with the note section that answers each, plus 10 questions the notes do not cover.
2. 2

   Ingestion

   Fetch published notes, split by headings, attach URL and anchor metadata.
3. 3

   Index

   Embed chunks with an open model, load into Qdrant with payload filters; build the keyword index.
4. 4

   Retrieval

   Hybrid search plus reranking; measure hit rate at 5 on the evaluation set before any LLM work.
5. 5

   Answering

   Grounded prompt, structured output of answer plus citations, refusal path for uncovered questions.
6. 6

   API and UI

   FastAPI endpoint; a small search box on the notes page that shows answer and sources.
7. 7

   Guard rails

   Rate limits, max context and output tokens, logging of question, retrieved chunks and answer.

Retrieval quality is measured and tuned before prompt work: if the right section isn't retrieved, no prompt can fix the answer.

Targets are filled in with real numbers once measured; until then this page states intentions, not results.

• RAG or fine-tuning for a knowledge base? RAG: content changes and answers must cite sources.
• How do you choose chunk size? Follow document structure, then measure retrieval on real questions.
• Why hybrid search? Vectors miss exact tokens; keywords miss paraphrases.
• How do you evaluate a RAG system? Separate retrieval hit rate from answer faithfulness, and keep a regression set.
• How do you stop hallucinations? Grounded prompts, citations, an explicit "not covered" path and human review of samples.

Related reading: RAG fundamentals, Chunking strategies, Embeddings and vector search, Evaluating LLM applications.