jatin.blog ~ $
$ cat ai-engineering/chunking-strategies.md

Chunking Strategies for Retrieval

How fixed-size, recursive, semantic, structural, and parent-document chunking affect retrieval.

Jatin Bansal@blog:~/ai-engineering$ open chunking-strategies

Chunking determines what the embedding model represents and what the retriever can return. A poor boundary can separate a fact from its context or combine unrelated sections into one vector. These errors occur before retrieval and cannot always be repaired by reranking.

Choose the retrieval unit

A chunk is the smallest unit committed to a vector index. It is both the retrieval unit returned for a query and the text summarized by one embedding vector. Retrieval benefits from precise chunks, while embedding quality benefits from enough context to form a coherent meaning.

  • The retrieval atom wants to be small. Small chunks are precise: the returned text contains the answer with little surrounding noise, which keeps the generator’s context window lean.
  • The embedding payload wants to be coherent. An embedding is a single point summarizing the whole chunk; if the chunk spans five unrelated topics, the average lands nowhere in particular, and the chunk fails to be retrieved for any of those topics.

Chunking is the policy that produces these units from source documents. Reranking and query rewriting can improve ordering, but neither can reconstruct context discarded at index time.

Chunking methods

The main methods differ in the source structure they preserve and their indexing cost.

1. Fixed-size, character or token

Slice the document every N characters or every N tokens. Optionally overlap each chunk with the previous by some fraction (10–20% is conventional). This is the baseline; fast to compute, deterministic, oblivious to structure. It will cut sentences in half, split a code function across two chunks, and orphan headers from their bodies. Use it only when you genuinely don’t know anything about the input format.

2. Recursive character splitting

The default workhorse of production RAG. Given an ordered list of separators (["\n\n", "\n", ". ", " ", ""]), the splitter tries to keep chunks under a target size by splitting on the strongest separator first, then falling back to weaker ones only when a section is still too large. The effect is that paragraph boundaries are preserved when possible, sentence boundaries when not, and word boundaries as a last resort. LangChain’s RecursiveCharacterTextSplitter is the reference implementation; LlamaIndex’s SentenceSplitter does something similar with a sentence-tokenizer twist.

3. Structural chunking

Use the document’s own structure. For markdown: split on headers and treat each section as a chunk, carrying the header path forward in the metadata so retrieval can return “Section 4.2.1 > Tuning ef_search.” For HTML: split on block-level elements. For source code: use an AST parser (Python’s ast, tree-sitter for any language) to split by function or class. For JSON/XML: split by object boundary. This is the highest-fidelity option when the input has structure that maps to retrieval intent.

4. Semantic chunking

Embed every sentence, walk through the document, and start a new chunk whenever cosine similarity between consecutive sentences drops below a threshold. The intuition is that semantic boundaries are real, even when no markup signals them. The cost is that you pay one embedding call per sentence just to chunk the corpus, before you embed anything for retrieval. Worth it for unstructured prose where structural signals are absent; usually overkill when good markup exists. LlamaIndex’s SemanticSplitterNodeParser is the reference implementation.

5. Parent-document and sentence-window retrieval

Decouple the embedding payload from the returned payload. Parent-document retrieval embeds small child chunks (say, 256 tokens each), indexes them, but stores a parent_id pointing back to a larger parent chunk (1–2k tokens). At query time the top-k child chunks are retrieved, deduplicated to their parents, and the parents are what get injected into the prompt. Sentence-window retrieval is a variant: embed each sentence, and at retrieval time return a window of ±N sentences around the match.

Both patterns are the right answer more often than people realize, because they collapse the retrieval-atom-versus-embedding-payload tension into something the user explicitly controls.

Contextual retrieval

A 2024 Anthropic post on contextual retrieval demonstrated that prefixing each chunk with a short LLM-generated description of where the chunk sits in the source document (50–100 tokens) reduces retrieval failures by 35–49% before any reranking. The method adds one model call per chunk at index time and none at query time. It suits small, static corpora where the retrieval gain justifies that cost.

Code: from raw to recursive to semantic

A fixed-size chunker makes the basic contract explicit:

python 
1
2
3
4
5
6
7
8
9

# stdlib only — no install required
def fixed_size_chunks(
    text: str,
    chunk_size: int = 1000,
    overlap: int = 150,
) -> list[str]:
    assert 0 <= overlap < chunk_size
    step = chunk_size - overlap
    return [text[i : i + chunk_size] for i in range(0, len(text), step)]
typescript 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

function fixedSizeChunks(
  text: string,
  chunkSize = 1000,
  overlap = 150
): string[] {
  if (overlap < 0 || overlap >= chunkSize) throw new Error("invalid overlap");
  const step = chunkSize - overlap;
  const chunks: string[] = [];
  for (let i = 0; i < text.length; i += step) {
    chunks.push(text.slice(i, i + chunkSize));
  }
  return chunks;
}

Now the recursive splitter, which is what 80% of production systems should actually use. The LangChain Python package and the LangChain JS package keep the API in step.

python 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

# pip install langchain-text-splitters tiktoken
from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(
    encoding_name="cl100k_base",
    chunk_size=512,          # target tokens per chunk
    chunk_overlap=64,        # ~12% overlap
    separators=["\n\n", "\n", ". ", " ", ""],
)

with open("article.md") as f:
    docs = splitter.create_documents([f.read()])

for d in docs[:3]:
    print(len(d.page_content), repr(d.page_content[:80]))
typescript 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

// npm install @langchain/textsplitters
import { RecursiveCharacterTextSplitter } from "@langchain/textsplitters";
import { readFileSync } from "node:fs";

const splitter = new RecursiveCharacterTextSplitter({
  chunkSize: 512,       // characters here, not tokens — pick consciously
  chunkOverlap: 64,
  separators: ["\n\n", "\n", ". ", " ", ""],
});

const text = readFileSync("article.md", "utf-8");
const docs = await splitter.createDocuments([text]);

for (const d of docs.slice(0, 3)) {
  console.log(d.pageContent.length, JSON.stringify(d.pageContent.slice(0, 80)));
}

The Python version measures chunk size in tokens through from_tiktoken_encoder; the JavaScript version defaults to characters and needs a token counter when precise budgets matter. The separators list defines the cut hierarchy; for markdown corpora, prepend ["\n# ", "\n## ", "\n### "] to keep section headers intact.

For semantic chunking, the basic algorithm is small enough to write directly. Embed each sentence, take a rolling similarity, cut when the similarity to the previous sentence drops below a percentile threshold.

python 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

# pip install openai numpy
from openai import OpenAI
import numpy as np
import re

client = OpenAI()

def split_sentences(text: str) -> list[str]:
    # crude — swap for spaCy or pysbd in production
    return [s.strip() for s in re.split(r"(?<=[.!?])\s+", text) if s.strip()]

def embed_batch(sentences: list[str]) -> np.ndarray:
    resp = client.embeddings.create(
        model="text-embedding-3-small",
        input=sentences,
    )
    return np.array([d.embedding for d in resp.data])

def semantic_chunks(text: str, breakpoint_percentile: int = 90) -> list[str]:
    sents = split_sentences(text)
    if len(sents) < 2:
        return sents
    vecs = embed_batch(sents)
    # similarity between consecutive sentence pairs
    sims = [
        float(np.dot(vecs[i], vecs[i + 1]) /
              (np.linalg.norm(vecs[i]) * np.linalg.norm(vecs[i + 1])))
        for i in range(len(sents) - 1)
    ]
    # cut wherever similarity dips below the chosen percentile (i.e. a big drop)
    threshold = np.percentile(sims, 100 - breakpoint_percentile)
    chunks, current = [], [sents[0]]
    for i, sim in enumerate(sims):
        if sim < threshold:
            chunks.append(" ".join(current))
            current = [sents[i + 1]]
        else:
            current.append(sents[i + 1])
    chunks.append(" ".join(current))
    return chunks

The TypeScript shape is identical with the OpenAI Node SDK; the only meaningful difference is that you’ll want a real sentence tokenizer (compromise or wink-nlp) instead of a regex. For production semantic chunking on real corpora, lean on LlamaIndex’s SemanticSplitterNodeParser rather than rolling your own; there are edge cases (heading-only sentences, very short paragraphs, language detection) that the libraries already handle.

Failure modes to test

Chunk size and the generator are coupled. A 2k-token chunk feels generous until you remember that top-k retrieval means k of them go into the prompt. With k=5 and a 2k chunk size you’ve spent 10k tokens before the system prompt or query. Pick chunk size knowing how many chunks the generator will see and how much headroom you need for instructions and the answer.

Embedding providers may silently truncate input. Every embedding model has a context limit (text-embedding-3-small is 8191 tokens). Chunks that exceed it are silently truncated by the provider; you get an embedding for a different document than the one you stored. Always count tokens before sending, and assert the count against the model’s limit.

Facts can straddle a chunk boundary. “The deployment failed at 03:14 UTC because the readiness probe returned 503.” If the timestamp and the cause end up in different chunks, both chunks individually answer “what happened” badly. Overlap reduces this, but only papers over the symptom. The direct fixes are recursive or structural splitting (keep semantic units intact) and parent-document retrieval (embed small, return the whole surrounding section).

Overlap inflates everything. 20% overlap means 20% more chunks, 20% more embedding cost, 20% more index storage, and a near-guarantee of duplicate hits in top-k results that you’ll need to dedupe in the generator step. It’s a real trade, not a free win.

Small chunks make multi-hop reasoning worse, not better. Multi-hop questions require the model to combine information that lives in two or more places. Smaller chunks scatter that information across more retrieval candidates, increasing the chance that at least one needed hop is missed in top-k. Larger chunks (or parent-document retrieval) keep co-occurring facts co-located.

Structural chunking is fragile to upstream changes. A markdown splitter that relies on ## headers breaks the day someone writes a doc with Bold Title instead. AST-based code splitters break on parse errors. Both modes degrade gracefully if you chain back to a recursive splitter as the fallback path; both fail silently if you don’t.

Semantic chunking is O(N) embeddings before you’ve embedded anything for retrieval. On a corpus of millions of sentences this is the dominant cost. Reserve semantic chunking for high-value, low-volume corpora, or use it only on the long-tail documents that recursive splitting handles badly.

Chunking decisions are not portable across embedding models. A chunk size tuned for an 8191-token asymmetric retrieval model may not suit a 512-token symmetric sentence model. Re-tune model changes against a labeled retrieval eval set.

Further reading