# RAG Assistant That Answers From These Notes, With Citations

> A planned retrieval-augmented assistant over this site's technical notes: structure-aware chunking, hybrid search, answers that cite the exact note section, and an evaluation set that gates every change.

- Source: https://akshaythakare.com/ideas/rag-assistant-for-notes
- Stack: Python, FastAPI, Hugging Face, Qdrant, LLMs, RAG, Docker
- Tags: rag, llm, vector-search, evaluation, blueprint
- Updated: 2026-10-11

## Overview & goal

## The problem

This site has dozens of technical notes. A reader with a specific question — "why does my join double revenue?" — has to search, skim and connect several notes. An assistant could answer directly, **but only if every answer is traceable to the notes** and it admits when the notes don't cover something.

## What this build will deliver

- An ingestion job that turns notes into searchable chunks.
- Hybrid search (vectors plus keywords) in **Qdrant**.
- An answer API that cites the exact note sections it used.
- An evaluation set that runs on every change.

> **Note: Blueprint status**
>
> This is a plan. Nothing here has been built or measured yet; components are marked **Planned** in the architecture explorer.

## Success looks like

- The right section appears in the top 5 results for most evaluation questions.
- Answers contain no claims that aren't supported by the cited text.
- Questions outside the notes get "the notes don't cover this", not a guess.

## Architecture

```mermaid
flowchart TB
  accTitle: RAG assistant architecture
  accDescr: Notes are chunked, embedded and indexed in Qdrant; the answer API retrieves with hybrid search, reranks, prompts the LLM with the chunks and returns an answer with citations; an evaluation set checks every change.
  N[Notes API] --> I[Ingestion: chunk by headings]
  I --> E[Embed]
  E --> Q[(Qdrant: vectors + keywords)]
  U[Question] --> A[Answer API]
  A -- hybrid search --> Q
  A --> R[Rerank top 20 to top 5]
  R --> L[LLM with grounded prompt]
  L --> C[Answer + cited sections]
  V[Evaluation set] -.-> A
```

## Key design decisions

**Chunk by headings, not fixed sizes.** Notes are already structured; one section usually holds one idea. Long sections are split with overlap.

**Store the citation with the chunk.** Each chunk carries `note URL#section-anchor`, so answers can link to the exact paragraph.

**Hybrid retrieval.** Vectors find paraphrases; keywords find exact terms like `QUALIFY` or `COPY INTO`. Results are merged, then reranked.

**Grounded prompt with an exit.** The model is told to answer only from the provided sections and to say so when they don't contain the answer.

**Incremental indexing.** Only notes whose `updatedAt` changed are re-embedded; the embedding model version is stored so a model change triggers a full rebuild.

## Build plan

### Evaluation set first
Write 40 real questions with the note section that answers each, plus 10 questions the notes do not cover.
  
### Ingestion
Fetch published notes, split by headings, attach URL and anchor metadata.
  
### Index
Embed chunks with an open model, load into Qdrant with payload filters; build the keyword index.
  
### Retrieval
Hybrid search plus reranking; measure hit rate at 5 on the evaluation set before any LLM work.
  
### Answering
Grounded prompt, structured output of answer plus citations, refusal path for uncovered questions.
  
### API and UI
FastAPI endpoint; a small search box on the notes page that shows answer and sources.
  
### Guard rails
Rate limits, max context and output tokens, logging of question, retrieved chunks and answer.

Retrieval quality is measured and tuned **before** prompt work: if the right section isn't retrieved, no prompt can fix the answer.

## Acceptance criteria

| Check | Measure | Target to reach before calling it done |
| --- | --- | --- |
| Retrieval | Expected section in top 5 | Recorded and improving on each iteration |
| Faithfulness | Claims supported by cited text (human-checked sample) | No unsupported claims in the sample |
| Coverage honesty | Out-of-scope questions answered with "not covered" | All 10 out-of-scope questions |
| Freshness | Edited note searchable after re-index | Within one ingestion run |
| Cost and latency | Tokens and seconds per answer | Recorded per request |

Targets are filled in with real numbers once measured; until then this page states intentions, not results.

## Interview angles

- **RAG or fine-tuning for a knowledge base?** RAG: content changes and answers must cite sources.
- **How do you choose chunk size?** Follow document structure, then measure retrieval on real questions.
- **Why hybrid search?** Vectors miss exact tokens; keywords miss paraphrases.
- **How do you evaluate a RAG system?** Separate retrieval hit rate from answer faithfulness, and keep a regression set.
- **How do you stop hallucinations?** Grounded prompts, citations, an explicit "not covered" path and human review of samples.

Related reading: [RAG fundamentals](/notes/rag/rag-pipeline-fundamentals), [Chunking strategies](/notes/rag/chunking-strategies), [Embeddings and vector search](/notes/rag/embeddings-and-vector-search), [Evaluating LLM applications](/notes/llms/evaluating-llm-applications).