How to Choose Chunk Sizes and Rank Retrieved Chunks

In the last post I said: retrieve first, then generate. A reader asked the obvious next question — how do you choose the chunk size, and how do you decide which retrieved chunks actually reach the model? This post is the answer I promised.

There is no single right chunk size. There is a default that works, a way to tell when it is wrong, and a ranking step that matters more than most people think.

Why chunk size matters

Chunking is the step where your documents get cut into pieces, each piece gets embedded, and those embeddings become what the retriever searches. Get the cut wrong and the best embedding model in the world cannot save you — the right passage is either split across two chunks, or buried inside a chunk about something else.

Two failure modes, and they pull in opposite directions:

  • Too small. A 64-token chunk is precise, but it has no context. “The company’s revenue grew by 3% over the previous quarter” is useless as a chunk — which company, which quarter? The retriever finds it, the model cannot use it.
  • Too large. A 2,000-token chunk has all the context, but its embedding is an average of several ideas. It matches many queries weakly and none strongly, and it eats context window that could have held two better chunks.

Chunk size is the most consequential setting in a RAG pipeline. It shapes retrieval precision, answer quality, embedding cost, and index size, all at once.

The default that works

Start here, and only move when you have measured a reason to:

  • 512 tokens, 10–20% overlap, recursive splitting.

Recursive splitting means: try to split on paragraph breaks first, then line breaks, then sentences, then spaces. It is the default in LangChain’s RecursiveCharacterTextSplitter and the approach most 2026 guides converge on. A 512-token chunk is roughly 350–400 words, or about 2,000 characters in English.

Overlap exists for one reason: an idea that straddles a split point should not fall through the cracks. At 20% overlap on 512-token chunks you generate roughly 25% more chunks than with zero overlap, so overlap is not free. Keep it under 25% unless you can show it helps.

Match the size to the question, not the document

The same corpus serves different questions. A useful rule, reflected across several 2026 guides:

Question type Chunk size
Factoid — “what is the ticket ID format?” 128–256 tokens
Technical docs, code + explanation 512–768 tokens
Analytical, multi-hop questions 512–1024 tokens
Long-form reports, legal, contracts 768–1024 tokens
Whole-section comprehension 1024–2048 tokens

The principle is simple: a chunk should be big enough to contain the answer plus its local context, and small enough that the top results are about the answer, not about everything around it.

One more constraint that surprises people: the best size depends on the embedding model. A 2025 multi-dataset study (arXiv:2505.21700) found that some embedding models do better with larger chunks and others with smaller ones. There is no universal optimum — which is why the default above is a starting point, not a law.

When to break the default

Three techniques that beat “just tweak the size” when the default is not enough:

Semantic chunking. Embed each sentence, split where the meaning changes. It groups sentences by topic instead of by length. It is slower, and it can produce tiny 40-token fragments that hurt retrieval — always enforce a minimum chunk size if you use it. One 2026 practitioner benchmark reports semantic chunking giving large accuracy lifts; another reports recursive 512-token chunking beating it (Vectara 2026 study, as reported second-hand — I have not verified the original study, so treat that result as contested, not settled).

Small-to-big (parent–child) retrieval. Retrieve small, precise child chunks, then hand the model the larger parent chunk they came from. You get tight retrieval and full context. This is the pattern to reach for with legal and contract documents: a 200-token child for retrieval, a 1,000-token parent for generation.

Contextual retrieval. Before embedding a chunk, prepend a short line saying where it fits — which document, which section, which product. Anthropic published this in September 2024 and reported that it cut failed retrievals by 49%, and by 67% when combined with a reranker. Their measured failure rate went from 5.7% to 2.9%, then to 1.9% (top-20 retrieval). The cost they reported was about $1.02 per million document tokens, paid once at index time. Those are Anthropic’s own figures on their own test sets — but the direction is consistent with everything else in this post: chunks without context are the problem.

Late chunking is worth knowing about, though it is newer: embed the whole document first, then pool the token embeddings into chunk embeddings. It keeps each chunk aware of the document around it. Published by Günther et al. in 2024, presented at ICLR 2025. Not yet a production default.

Ranking: retrieve wide, rerank narrow

Getting the right chunk into the candidate set is only half the job. The other half is ordering — because the model only ever sees the top few.

A practical pipeline, and the one most production RAG systems use in 2026:

  1. Retrieve wide. Run dense vector search and BM25 keyword search in parallel, each returning its top 50. Dense search misses exact terms like ticket IDs and error codes; BM25 misses paraphrases. You need both. This is the hybrid search I mentioned in the last post.
  2. Fuse. Merge the two ranked lists with Reciprocal Rank Fusion (RRF). The formula scores each document by 1 / (60 + rank) in each list and adds them up. It works on ranks, not scores, so you never have to normalise two different scoring systems. The constant 60 comes from the original 2009 paper (Cormack, Clarke, and Buettcher) and has held up.
  3. Rerank. Pass the fused top 50 to a cross-encoder reranker — a model that reads the query and each passage together and scores true relevance. Cohere Rerank, Voyage rerank, and the open BGE reranker are the common choices in 2026. Keep the top 3–5.
  4. Generate. Only those top chunks go to the model.

Practitioner benchmarks report hybrid retrieval reaching roughly 91% recall@10 against about 78% for dense-only (aiworkflowlab.dev — a practitioner blog, not a peer-reviewed result). Adding a cross-encoder reranker is commonly reported as a +5 to +15 NDCG@10 lift over first-stage retrieval alone. Treat both numbers as indicative, not gospel — but the shape of the finding is consistent across sources: hybrid search fixes recall, reranking fixes precision.

The cost is real but small: the two parallel searches add a few milliseconds; reranking 50 candidates adds roughly 50–200 ms depending on the model. That is cheap compared to a wrong answer.

How to know if any of this helped

Do not tune chunk size by feel. Build a small evaluation set first — 30 to 50 real questions from your users, with the correct answer or the correct source passage noted for each. Then measure two things before and after every change:

  • Did the right chunk get retrieved at all? (Recall — the hybrid search step owns this.)
  • Did the right chunk rank near the top? (Precision — the reranker owns this.)

Change one thing at a time: chunk size, or overlap, or the reranker — never all three at once. Keep the change only if the numbers move. This is the same discipline as any other performance work, and RAG is no exception.

The short version

  • Start at 512 tokens, 10–20% overlap, recursive splitting.
  • Small chunks for facts, larger chunks for reasoning. Match the chunk to the question.
  • Retrieve small, generate from the parent chunk, when documents are dense.
  • Add context to each chunk before embedding — it is the cheapest large win available.
  • Retrieve wide with hybrid search, fuse with RRF, rerank to the top 3–5.
  • Measure on your own questions. Keep only what the numbers justify.

Next up: GraphRAG — building the graph lazily, only over what gets retrieved.

References

Comments

Leave a Reply

Create your own website, free on WordPress.com.Create your own website.
Start free