A retrieval-augmented generation (RAG) system can only ground its answers in the passages it retrieves. RAG chunking strategies decide how those passages are cut, so they shape what your system can find.
What Is Chunking in RAG?
Chunking in RAG is the step that splits documents into smaller passages, called chunks, before they are embedded and indexed. At query time, a retriever searches those chunks and passes only the most relevant ones to the large language model (LLM) as context.
An embedding is a list of numbers that captures the meaning of a piece of text, so similar texts end up close together. A retriever is the search component that compares the question's embedding with each chunk's embedding and returns the closest matches.
The main chunking strategies for RAG are fixed-size, recursive, structure-aware, semantic, parent-child, late chunking, contextual retrieval and LLM-based (agentic) chunking. None of the benchmarks we reviewed show one of them winning on every dataset. A practical approach is to start with clean, structured input and a simple structure-aware recursive baseline, then measure before you pay for more expensive methods.
Why Chunking Strategy Affects Retrieval Quality
Chunking strategy affects retrieval quality because each chunk becomes one embedding, and that embedding is all the retriever sees. If a chunk mixes several topics or holds only half an idea, the retriever has a weaker signal to match against the question.
- Chunks that are too large: One embedding has to stand for several ideas, so its match to any single question weakens. Large chunks also use more of the LLM's context window.
- Chunks that are too small: A sentence cut away from its paragraph can lose the subject it refers to. The LLM then gets too little text to answer from.
- Different question types: Fact lookups and broad "how does this work" questions favor different chunk sizes, as the size section below shows.
The right setting also depends on the embedding model and the documents. Shaukat et al. tested 36 chunking methods across six subject domains and five embedding models. Their 36-strategy chunking benchmark, a March 2026 arXiv preprint, states that "No single chunking strategy is universally optimal."
The Main RAG Chunking Strategies
RAG chunking strategies differ in where they place chunk boundaries and how much compute they need. The methods below run from cheapest to most expensive, so you can stop at the first one that meets your retrieval target.
For web content, start with the first three. The later methods target specific failure modes and add indexing cost.
Fixed-Size Chunking
Fixed-size chunking splits text every N tokens or characters. An optional overlap repeats a few tokens between neighboring chunks so a sentence cut at the boundary appears in both.
Fixed-size chunking is fast and predictable, and every chunk fits the embedding model's input limit. The drawback is that cuts land mid-sentence or mid-table, which can split one idea across two chunks. That makes fixed-size chunking a useful baseline to measure other methods against.
Very small fixed chunks perform badly. In the same 36-strategy benchmark, paragraph group chunking reached a mean nDCG@5 of about 0.459, while 100-character fixed chunks scored 0.244.
nDCG@5 (normalized discounted cumulative gain) scores how well the top five results are ranked, giving more credit when relevant chunks appear near the top. Because the losing baseline was so small, the result says little about tuned splits of 400 to 512 tokens.
Recursive Chunking
Recursive chunking tries a list of separators in order until each chunk fits the size limit. The LangChain text splitter docs describe how RecursiveCharacterTextSplitter keeps larger units such as paragraphs intact and moves to smaller units, such as sentences and then words, only when a unit is too large.
Because it prefers natural breaks, recursive splitting keeps most paragraphs whole at close to fixed-size cost. That makes it a sensible default for plain text, and the semantic chunking evidence below supports it as a baseline.
Structure-Aware Chunking (Markdown and HTML Headers)
Structure-aware chunking splits a document along its own structure, such as Markdown headers, HTML sections, code blocks and tables. Long sections are then split again with a recursive splitter so each chunk still fits the size limit.
Each chunk can carry its heading path as metadata, for example "Billing > Invoices > Refunds." The heading path tells the retriever and the LLM which part of the document the chunk came from. You can also prepend it to the chunk text before embedding.
This method depends on input that keeps its structure. Raw scraped HTML mixes headings with menus, scripts and cookie banners. Web pages are much easier to split after HTML-to-Markdown conversion, which strips that noise and keeps the heading hierarchy.
A minimal Python version uses two LangChain splitters:
from langchain_text_splitters import (
MarkdownHeaderTextSplitter,
RecursiveCharacterTextSplitter,
)
headers = [("#", "h1"), ("##", "h2"), ("###", "h3")]
header_splitter = MarkdownHeaderTextSplitter(headers_to_split_on=headers)
sections = header_splitter.split_text(markdown_text) # one Document per section
size_splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(
chunk_size=400, chunk_overlap=0
)
chunks = size_splitter.split_documents(sections)
for chunk in chunks:
chunk.metadata["source_url"] = source_url # keep for citationsEach chunk keeps its h1, h2 and h3 values in metadata, plus the page URL. We found no published benchmark that isolates Markdown-header chunking, so treat this method as design reasoning.
The Vectara study linked in the next section offers indirect support. Its appendix notes that splitting documents into structured sections before fixed-size chunking solves the boundary problems it observed.
Semantic Chunking
Semantic chunking embeds each sentence and starts a new chunk where the similarity between neighboring sentences drops. The goal is to cut where the topic changes instead of at a set length.
Semantic chunking costs more because every sentence needs its own embedding before the final chunk embeddings are made. Vectara's semantic chunking study concluded that "The results show that the computational costs associated with semantic chunking are not justified by consistent performance gains."
In that October 2024 study, semantic chunking won clearly only on artificially stitched documents that mixed many topics. On the stitched Miracl* set, breakpoint semantic chunking scored 81.89 F1@5 against 69.45 for fixed-size chunks of a few sentences, using the stella_en_1.5B_v5 embedder. F1@5 balances precision and recall in the top five results.
On real, non-stitched datasets, the study found that fixed-size chunking often performed as well or better. The stitched result suggests semantic chunking may help when one file covers many unrelated topics, such as a long changelog. Our semantic chunking deep dive covers the method in more detail.
Parent-Child (Hierarchical) Chunking
Parent-child chunking indexes small child chunks for precise matching but returns the larger parent section to the LLM. It addresses the trade-off between precise search and enough context to answer.
A typical setup treats each page section as a parent and cuts it into child chunks of a few sentences each. When a child matches the question, the system looks up its parent ID and sends the full parent instead. The LangChain ParentDocumentRetriever reference describes this pattern as "Retrieve small chunks then retrieve their parent documents."
The 36-strategy benchmark above included a parent-child setup with 500-token parents and 100-token children, and it ranked in the lower half of the methods tested, so treat the pattern as something to test on your data. It pairs well with structure-aware chunking, since a Markdown section is a natural parent.
Late Chunking and Contextual Retrieval
Late chunking and contextual retrieval both address one problem: a chunk read on its own can lose the document context it depends on. A chunk that says "this plan includes single sign-on" is ambiguous if the plan name appears only in an earlier section.
Late chunking runs the whole document through a long-context embedding model first, then averages the token embeddings inside each chunk's span. Each chunk vector is built from tokens that already "saw" the surrounding text. In Jina AI's late chunking paper, fixed-size 256-token chunks improved from 52.2 to 54.0 average nDCG@10 with late chunking.
That average covers three embedding models and four BEIR datasets, and the authors created the method. Late chunking did not help on their synthetic tasks where surrounding context was irrelevant.
Anthropic's contextual retrieval post, published in September 2024, describes using an LLM to write a short context, usually 50-100 tokens, for each chunk. That line is added to the chunk before embedding and before BM25 indexing. BM25 is a classic keyword-ranking method that scores chunks by how often the query's words appear in them, adjusted for length.
Merola and Singh describe the same indexing step: an LLM is prompted with the document to generate context for each chunk, which is prepended to the chunk. Merola and Singh compared both methods in a 2025 preprint with 50 queries and about 300 documents. Their contextual pipeline added BM25 and a reranker, a second model that rescores the top retrieved chunks.
The independent late chunking comparison found that "contextual retrieval preserves semantic coherence more effectively but requires greater computational resources. In contrast, late chunking offers higher efficiency but tends to sacrifice relevance and completeness." In the same study, late chunking helped with one embedding model and hurt sharply with another, BGE-M3.
LLM-Based (Agentic) Chunking
LLM-based chunking, also called agentic chunking, asks an LLM to read the document and decide where chunks start and end. Some versions also group related statements, called propositions, into chunks.
LLM-based chunking adds compute at indexing time. In the 36-strategy benchmark above, an LLM-based chunker used about 5.7 GB of RAM versus about 0.87 GB for paragraph grouping, and the authors flag that cost as a bottleneck for frequently updated data. Merola and Singh also note that generative chunkers do not always produce the exact same wording, so check output against the source.
Accuracy evidence is limited. An LLM boundary chunker scored a mean nDCG@5 of about 0.415 in the 36-strategy benchmark, close to the top method, but that is one benchmark with one setup. Reserve LLM-based chunking for small, high-value collections and measure it against your recursive baseline first.
RAG Chunking Strategies Compared
Fixed-size, recursive and structure-aware chunking are the low-cost options, while semantic, contextual and LLM-based methods add compute at indexing time. The cost column shows relative indexing cost based on how each method works, not measured figures.
| Strategy | How it splits | Cost | Best for | Watch out for |
|---|---|---|---|---|
| Fixed-size | Every N tokens or characters | Low | Baselines, uniform plain text | Cuts mid-sentence; tiny chunks retrieve poorly |
| Recursive | Paragraph, line, sentence, then word breaks until chunks fit | Low | Default for most prose | Ignores headings unless you split on them first |
| Structure-aware | Markdown or HTML headers, then recursive inside long sections | Low | Docs sites, help centers, clean Markdown | Needs input that keeps its structure |
| Semantic | Where similarity between neighboring sentences drops | Medium to high | Files that mix many unrelated topics | Extra embedding cost without consistent gains on real documents |
| Parent-child | Small child chunks indexed, larger parent returned | Low to medium | Precise matching plus fuller answer context | More storage; ranked lower half in one benchmark |
| Late chunking | Embeds the whole document, then pools tokens per chunk | Medium | Chunks that depend on earlier context | Needs a long-context embedder; results vary by model |
| Contextual retrieval | LLM adds a context line to each chunk before indexing | High | Ambiguous chunks in long documents | Extra LLM generation step per chunk at indexing |
| LLM-based (agentic) | LLM decides boundaries or groups propositions | Highest | Small, high-value collections | High indexing compute; output wording can vary |
How to Choose Chunk Size and Overlap
Start with chunks of about 256 to 512 tokens, split with a recursive or structure-aware splitter, then tune size on your own questions. Treat this range as a starting point, since published results show the best size moves with the dataset and embedding model.
Bhat et al.'s multi-dataset chunk size study found that "smaller chunks (64-128 tokens) are optimal for datasets with concise, fact-based answers, whereas larger chunks (512-1024 tokens) improve retrieval in datasets requiring broader contextual understanding." That May 2025 preprint measured recall@k, the share of questions where a correct passage appears in the top k results.
With the Stella 1.5B v5 embedder and no overlap, Recall@1 on SQuAD* fell from 0.6419 at 64 tokens to 0.3855 at 1,024 tokens. On TechQA*, the same setup rose from 0.0486 to 0.6192. The asterisk marks datasets the authors built by stitching documents together.
- Fact lookups: Short answers, such as a price or a setting name, tend to favor smaller chunks.
- Broad questions: Questions that need an explanation tend to favor larger chunks or parent-child retrieval.
- Embedding limits: Keep every chunk below the embedding model's maximum input length. Some APIs cut off extra text by default; the Cohere embed truncate parameter defaults to
END, so the tail of an oversized chunk is not represented in its embedding.
PDFs are a special case. NVIDIA's chunking benchmark reported that "page-level chunking achieved the highest average accuracy (0.648)" across five PDF datasets, while token-based approaches scored "between 0.603 and 0.645."
Those scores measure end-to-end answer accuracy on paginated PDFs with one fixed NVIDIA pipeline. Web pages and Markdown have no pages, so the page-level result does not carry over to web content.
Overlap evidence is mixed. Jina AI's paper tested a 16-token overlap on 256-token chunks and reported that "The results in Table 6 do not show a clear advantage of using overlaps." NVIDIA's own overlap test, which it calls "not a full-grid search," found 15% worked best on one dataset. A chemistry-domain chunking study recommends "large chunks and low or zero overlap" as a starting point for its corpora. Start with no overlap or a small one and keep it only if recall improves on your test set.
Why Clean Input Matters Before You Chunk
Clean input matters because every piece of extracted text ends up in some chunk, including menus, cookie banners, footers and sidebars. Those boilerplate chunks can match queries and push the article text out of the top results.
Check what your chunks contain before you tune chunk size. A retrieval problem that looks like a size issue can start at extraction.
Olostep's web scraping for RAG entry lists four input needs: clean text, logical boundaries, source URLs for citation and consistent Markdown. Olostep's guide to preparing LLM-ready data explains why raw HTML falls short as model input.
A web-to-chunks workflow for a docs site or help center looks like this:
- Fetch individual pages as clean Markdown or JSON with a single-URL scraping API, keeping the source URL with each result.
- Ingest a full docs site by crawling whole sites for RAG, using include and exclude URL patterns to skip pages you don't need.
- Store the raw Markdown, so you can re-chunk later without fetching every page again.
- Split each page on its Markdown headers, then run a recursive splitter on long sections.
- Save the source URL and heading path with every chunk, so answers can cite the exact page section.
- Refresh changed pages in bulk with batch URL processing, then re-run chunking on the new Markdown.
Olostep covers the fetching steps (1, 2 and 6) and returns clean Markdown or JSON with source URLs. It does not chunk or embed content; your splitter, embedding model and vector database handle those steps.
How to Evaluate a Chunking Strategy on Your Data
Evaluate a chunking strategy by measuring retrieval on real questions before you judge the final answers. NVIDIA's write-up gives the same advice: "We recommend evaluating these strategies on your own data to confirm they work well for your specific use case."
- Write 30 to 100 real user questions and mark the passage or page that answers each one.
- Measure retrieval first with recall@k, hit rate and nDCG, before you score generated answers.
- Compare two or three strategies, changing one variable at a time, such as splitter, chunk size or overlap.
- Read failed retrievals by hand to see whether the cause is a bad split, noisy text or different wording.
- Track chunking time, chunk count and index size alongside retrieval scores.
Hit rate is the share of questions with at least one relevant chunk in the top k results. Recall@k and nDCG add detail on how many relevant chunks appear and how high they rank.
A strategy that wins on your test set is the right choice for that corpus, even if a benchmark on other data disagrees. Re-run the test when you change embedding models.
Frequently Asked Questions
These short answers summarize the guidance above. The earlier sections cover the evidence behind each one.
Which Chunking Strategy Is Best for RAG?
No strategy is best on every dataset, so start with structure-aware recursive splitting and measure retrieval on your own questions before trying costlier methods.
What Chunk Size Should I Use for RAG?
Start around 256 to 512 tokens, then test smaller chunks for fact lookups and larger chunks for broad questions that need more context.
How Much Overlap Should Chunks Have?
Evidence on overlap is mixed, so start with no overlap or a small one and keep it only if recall improves on your own test set.
Is Semantic Chunking Better Than Recursive Chunking?
Not consistently on real documents, according to Vectara's study, and semantic chunking costs more because it embeds every sentence before chunking.
Do Long-Context LLMs Make Chunking Unnecessary?
Not in most cases: the EMNLP long-context versus RAG study found in 2024 tests that long context scored higher on average when well resourced, but RAG over chunks used far fewer tokens.
How Should I Chunk Tables and Code?
Keep each table or code block intact as one chunk with its heading, and avoid splitting a table mid-row or a code block mid-function.