RAG Deep: Chunking Strategies Tested at Scale

RAG Deep: Chunking Strategies Tested at Scale Chunking is the most argued over and least measured decision in RAG. Ask five engineers what size to use and you get five confident answers: 256, 512, 1,000, semantic, agentic. Most of those answers come from a blog post somebody half remembers, not from a measurement. We ran the measurement. Six chunking strategies, one real corpus, 25 questions, and the same retriever for all of them. The headline is not what we expected: chunk size barely moved the result, and the fanciest method cost 56 seconds to build an index that performed no better than the simplest one. This part shows the numbers, the two ideas that actually move retrieval quality, and the default we now use. This is Deep Part 6 of the AI Engineer Deep series. Deep Part 1 built attention by hand, Deep Part 2 built a tokenizer from scratch, Deep Part 3 trained tiny transformers and read the loss curves, Deep Part 4 measured the KV cache and sampling, and Deep Part 5 measured context windows and forgetting. The earlier RAG from Zero article introduced retrieval, and the first chunking test ran a toy version of this experiment. This part is the serious rerun: a bigger corpus, more strategies, and a retriever that is not a keyword overlap scorer. The corpus and the test The corpus is 94 passages, 22,516 words, taken from our own published articles on this site: transformers, tokenization, training, inference, context windows, RAG, MCP, and LLMOps. Each passage is one section of one article, 80 to 600 words. Nothing is synthetic. We wrote 25 questions, each one aimed at exactly one passage, phrased the way a reader would ask it, not copied from the passage text. The retriever is all-MiniLM-L6-v2, a 384 dimension embedding model, cosine similarity, top 10. For each strategy we report recall at 5, recall at 10, and nDCG at 10, which rewards putting the right chunk near the top instead of anywhere in the list. One honest caveat before the table. The passages average about 240 words, so a 512 token chunker rarely gets to use its full budget. Mean chunk size for fixed 512 came out at 299 tokens, not 512, because most passages fit whole. That means this test compares chunkers on section sized text, which is exactly what a documentation corpus looks like, but it is not a test on 50 page PDFs. The conclusion is scoped to that, and we say so. Fixed size chunking: the boring winner Four fixed sizes, GPT-2 tokens, no overlap except where noted. Here is the whole result. StrategyChunksMean tokensRecall at 5Recall at 10nDCG at 10Build time Fixed 256161195.10.920.920.8050.2 s Fixed 512105299.10.880.920.8050.1 s Fixed 512, overlap 64111291.10.880.960.7960.1 s Fixed 102494334.10.880.960.8020.1 s Recursive 1000 chars, overlap 200180196.60.880.960.7700.0 s Semantic, threshold 0.4592233.90.840.960.78756.1 s Read the nDCG column first. The best is 0.805, the worst is 0.770, and four of the six strategies sit between 0.787 and 0.805. That is a spread of 0.035 across every idea in the chunking debate. Recall at 10 is 0.92 or 0.96 for all six. On this corpus, with this retriever, the chunker is a second order decision. The embedding model and the questions matter more than the splitter, and anybody selling you a chunking strategy as the difference between a working RAG system and a broken one is selling you something. Fixed 256 has the best recall at 5, 0.92, tied for best nDCG. Smaller chunks put the answer in a tighter window, so when the retriever finds it, it ranks it high. But 256 also has the lowest recall at 10, 0.92: two questions never found their passage at all, because the split cut the key sentence away from its context. Fixed 1024 has the opposite shape: nothing is cut, recall at 10 is 0.96, but each retrieved chunk carries more text the generator does not need. Fixed 512 with 64 tokens of overlap is the balance point in our table: recall at 10 of 0.96, nDCG 0.796, and only 111 chunks to index. That is the default we use, and the next part of the series builds on exactly this chunking. Overlap: a small gain with a real cost Compare fixed 512 with and without overlap. Recall at 10 rises from 0.92 to 0.96, one extra question answered, and nDCG actually falls slightly, 0.805 to 0.796, because the overlapping chunks push near duplicates into the top 10. Six extra chunks for one extra answered question is a fair trade at this scale. At production scale the arithmetic changes. An independent study on papers, contracts, and meeting transcripts found that overlapping 512 token chunks by 64 to 256 tokens moved recall by under 3 points either way while adding up to 96 percent more indexed text. Overlap is cheap insurance on a small corpus and a real storage and embedding bill on a large one. We use 64 tokens, about 12 percent of the chunk, which matches the usual guidance of 10 to 20 percent overlap, and we would not go higher without measuring. Recursive splitting: the sensible default, same result LangChain's RecursiveCharacterTextSplitter is the default most teams reach for. It tries separators in order, paragraph break, line break, sentence, word, then character, splitting only as finely as it must to fit the size limit. The documented example values are chunk size 1,000 characters with 200 characters of overlap, and note that the size is in characters, not tokens. On our corpus it produced 180 chunks at a mean of 196.6 tokens, recall at 10 of 0.96, and the lowest nDCG in the table at 0.770. Respectable, and slightly worse than plain fixed 512 on ranking quality. The splitter respects sentence boundaries, which sounds like it should help, but on section sized text the fixed splitter already lands on clean boundaries most of the time, so the extra cleverness buys nothing. Use recursive splitting when your documents have real structure, markdown headers, code blocks, lists, and LangChain has dedicated splitters for those. For plain prose, it is a wash. Semantic chunking: the expensive lesson Semantic chunking is the idea that chunks should follow meaning, not character counts. Split the text into sentences, embed each one, and cut where the cosine similarity between neighbouring sentences drops below a threshold. It is the most recommended technique in RAG tutorials, and on our run it was the worst value in the table. Our threshold of 0.45 was too aggressive: it cut at nearly every sentence boundary and produced 922 chunks with a mean size of 33.9 tokens, mostly single sentences. Recall at 5 fell to 0.84, the lowest in the table, because a single sentence rarely carries enough context for the embedding to match the question. nDCG recovered to 0.787 only because recall at 10 still reached 0.96. And the build took 56.1 seconds against 0.1 seconds for fixed chunking, because every sentence had to be embedded before a single chunk existed. The lesson is not that semantic chunking is useless. It is that the threshold is the whole method, and a bad threshold is worse than no method at all. LlamaIndex's implementation uses the 95th percentile of cosine distances as its cut point, which is far less aggressive than a fixed 0.45. A separate study from Vectara, presented at NAACL 2025, tested semantic against fixed size chunking on document retrieval, evidence retrieval, and answer generation, and concluded the extra compute cost is not justified by consistent performance gains. On non synthetic data, fixed size often performed better, and embedding quality mattered more than the chunking choice. Our numbers point the same way. Chroma's own evaluation found their semantic chunkers reached the highest token level recall, 0.919 for their LLM based chunker and 0.913 for cluster based, against a recursive baseline, so semantic chunking can win when the threshold and the corpus suit it. But note what that evaluation also found: OpenAI Assistants' default of 800 tokens with 400 overlap scored below average recall and lowest on the other metrics. Defaults are not measurements. Measure yours. The two ideas that actually move the number If chunk size moves nDCG by 0.03, what moves it by more? Two ideas from the literature, both verified in published experiments. First, contextual retrieval, from Anthropic in September 2024. The problem chunking cannot fix is that a chunk, once cut, loses its context. A chunk that says "the company grew revenue by 3 percent" is unretrievable unless the query mentions the company. Anthropic's fix is to ask a small model to write 50 to 100 tokens of context situating each chunk in its whole document, then prepend that context before embedding and before building the BM25 index. On their tests, retrieval failure in the top 20 chunks fell from 5.7 percent at baseline to 3.7 percent with contextual embeddings alone, a 35 percent cut, to 2.9 percent with contextual BM25 added, a 49 percent cut, and to 1.9 percent with reranking on top, a 67 percent cut. The cost they quote is $1.02 per million document tokens, with prompt caching, for 800 token chunks in 8,000 token documents. That is the largest single gain in the chunking literature, and it does not come from the chunker at all. It comes from giving each chunk its context back. Second, late chunking, from Jina AI. Instead of splitting the text and then embedding each piece, run the transformer over the whole document first, up to the embedding model's context window, then average the token vectors inside each chunk boundary. The chunk embedding keeps information from the surrounding text. On the BEIR benchmark with 256 token chunks, late chunking beat naive chunking on SciFact, 66.10 against 64.20 nDCG at 10, on TRECCOVID, 64.70 against 63.36, on FiQA, 33.84 against 33.25, and most of all on NFCorpus, 29.98 against 23.46. It did nothing on Quora, 87.19 both ways, and the paper is honest about the limits: on some reading comprehension sets naive chunking won with large chunks, and it does not help on synthetic needle tasks. The catch is that it needs a long context embedding model, 8,192 tokens for the model in the paper, and it costs one full document pass per document. Both ideas point the same way. The failure is not the cut. The failure is the lost context around the cut. Fix that, and the exact chunk size matters even less. Our opinionated take We spent a week on chunking and the answer is boring, which is why it is worth writing down. Use fixed size chunks of 512 tokens with 64 tokens of overlap, or LangChain's recursive splitter at 1,000 characters with 200 overlap if your documents have structure. Do not use semantic chunking unless you have set the threshold from your own data and measured it against fixed size on your own questions, because a wrong threshold turns your corpus into sentence confetti, as ours did, and costs 500 times the build time for no gain. Put the effort where the numbers say it belongs: write 25 questions with known answers and measure recall before and after every change, add a filename and heading path to each chunk, which one study found lifted recall on contracts from 38.9 to 53.2, and read the contextual retrieval post before you reach for a fancier splitter. Chunking is plumbing. Measure it once, pick the boring default, and spend the saved week on retrieval, which is where RAG actually lives or dies. The next part measures exactly that. What to do this week Take one real corpus you care about, 20 to 100 documents. Split it into passages, write 20 questions with one known answer each, and run fixed 256, fixed 512 with overlap, and recursive splitting through the same embedding model. If the nDCG spread is under 0.05, stop tuning chunk size and move on. Then try one contextual retrieval experiment: prepend the document title and section heading to each chunk and remeasure. That one line of metadata is the cheapest version of Anthropic's idea, and on most corpora it buys more than any splitter change. What is Next in This Series Deep Part 7 measures the retrieval side: BM25 against dense search, reciprocal rank fusion, and cross encoder reranking, all on the same 111 chunks from this part. The question it answers is whether hybrid search earns its complexity, and what reranking can and cannot fix. After that, Deep Part 8 builds our own eval harness, because none of these measurements mean anything without a test set you trust. Originally published on LogicStack. Run the code examples right in your browser with our free interactive Python and SQL playgrounds — no setup, no sign-up.