Chunking Strategies for Retrieval
Fixed-size, recursive, structure-aware and parent-child chunking — and how to choose based on your documents and questions.
RAGIntermediate2 min readUpdated 11 Oct 2026
Why chunking matters
The retriever can only return what you chunked. Too small and a chunk lacks the context to be useful; too large and its embedding averages many topics, so it matches nothing precisely.
Strategies
| Strategy | How | Good for |
|---|---|---|
| Fixed-size | N tokens with M overlap | Uniform prose, quick baselines |
| Recursive | Split on paragraphs → sentences → words until under the limit | General text |
| Structure-aware | Split on headings, sections, code blocks, table rows | Docs, wikis, Markdown, technical notes |
| Semantic | Split where embedding similarity between sentences drops | Long unstructured text |
| Parent-child | Embed small chunks, return their larger parent section | Precise matching and enough context |
Structure-aware chunking for technical docs
def chunk_markdown(md: str, max_tokens: int = 400):
sections = split_on_headings(md) # keep the heading path, e.g. "Snowflake > COPY INTO"
for s in sections:
for piece in recursive_split(s.body, max_tokens, keep_code_blocks_whole=True):
yield {"text": f"{s.heading_path}\n\n{piece}", "heading": s.heading_path}Prefixing each chunk with its heading path is one of the cheapest, most effective improvements: the chunk now carries its own context.
Practical defaults
- Start around 300–500 tokens with 10–15% overlap, then measure
- Never split inside a code block or a table row
- Store metadata (source, section, date, version) for filtering and citations
- Re-chunk and re-embed when you change the strategy — mixed strategies in one index are hard to reason about
Finished this note?