Chunking Documents for Embedding Quality

On any corpus of long documents, how text is split into embeddable units affects retrieval more than which embedding model produced the vectors — and it is the parameter teams tune last, after spending weeks comparing models. This page sets chunk boundaries, size and overlap from the structure of the content, and measures the result. It sits under vector search integration strategies in search engine selection and architecture.

Prerequisites

  1. Documents with some structure to split on — headings, paragraphs, list items. Splitting unstructured walls of text is a harder problem with worse outcomes.
  2. An embedding pipeline that can re-run over the whole corpus, because changing chunking means re-embedding everything.
  3. A small evaluation set: queries with the passages that should answer them.
  4. A decision about what a search result represents — the passage or the document it came from.

Diagnosis: a vector is a summary, and summaries of mixed content are useless

An embedding represents one coherent idea reasonably well and several unrelated ideas badly, because the vector lands somewhere between them and is close to none. That is why chunk size matters: it decides how many ideas each vector has to summarise.

The failure at both extremes is easy to demonstrate. Embed a whole 40-page manual and query for a specific installation step: the document vector is an average over installation, safety, troubleshooting and warranty, and it matches the query weakly. Embed each sentence and query the same way: the matching sentence is retrieved, and it reads “Then tighten the bracket” with no indication of what is being installed.

query: "how do I mount the bracket on a brick wall"

whole document (12,000 tokens)     similarity 0.61   returned 8th
single sentence (14 tokens)        similarity 0.79   returned 1st, but unusable alone
passage with heading (340 tokens)  similarity 0.84   returned 1st, and answers the question

The middle row is what you are aiming for: enough context to be interpretable, narrow enough to be about one thing.

Retrieval quality against chunk size Whole documents average unrelated topics together, single sentences lack context, and passage-sized chunks retrieve best. whole document one vector, many topics weak match to anything passage — 200–500 tokens one idea, with context interpretable on its own single sentence precise but contextless "then tighten the bracket" the useful band is wide, so precision here matters less than avoiding either extreme and both extremes are what you get from a default configuration
The middle band is forgiving; the extremes are not. Most chunking work is about staying out of the two red boxes rather than finding an exact optimum.

Solution Steps

1. Split on structure, never on character count

A fixed character split cuts mid-sentence, producing chunks that begin with a dangling clause and embed poorly. Paragraph and heading boundaries are already semantic boundaries in the source, and respecting them costs a few lines.

# chunk.py — accumulate whole paragraphs up to a token target
def chunk_document(doc, target=350, overlap=60):
    chunks, buf, count = [], [], 0
    for para in doc["paragraphs"]:
        n = len(para.split())
        if count + n > target and buf:
            chunks.append(buf[:])
            keep = []
            total = 0
            for p in reversed(buf):                 # carry the tail forward as overlap
                total += len(p.split())
                keep.insert(0, p)
                if total >= overlap:
                    break
            buf, count = keep, total
        buf.append(para); count += n
    if buf:
        chunks.append(buf)
    return chunks

2. Prepend the heading path to every chunk

A chunk about “installation” is far more retrievable when the vector also knows what is being installed. The heading path costs a few tokens and adds the context the passage assumes but does not state.

def contextualise(doc, chunk_paras: list[str]) -> str:
    heading_path = " > ".join(doc["heading_stack"])     # e.g. "Mounting > Brick walls"
    return f"{doc['title']} — {heading_path}\n\n" + "\n\n".join(chunk_paras)

This single change is frequently worth more than any model comparison, because it fixes the failure where every chunk in a manual looks alike to the embedding.

3. Keep a pointer to the parent document

Users want the page, not the fragment. Retrieval matches a chunk; the result should present the document, optionally scrolled to the matching passage.

{
  "chunk_id": "doc-8812#c4",
  "parent_id": "doc-8812",
  "heading_path": "Mounting > Brick walls",
  "text": "…",
  "embedding": [...],
}

4. Deduplicate at the parent level when returning results

Without this, a query that matches four chunks of the same document returns that document four times and crowds out everything else.

def collapse_to_parents(hits, size=10):
    seen, out = set(), []
    for h in hits:
        pid = h["parent_id"]
        if pid in seen:
            continue
        seen.add(pid)
        out.append(h)                    # keep the best-scoring chunk per document
        if len(out) == size:
            break
    return out

5. Size overlap from the boundary risk, not from a rule of thumb

Overlap exists so that an answer spanning a boundary is not lost. Ten to twenty per cent of the chunk size is the usual band; more inflates the corpus without improving recall, and none risks losing exactly the passages that straddle two sections.

chunk 350 tokens, overlap 60   -> 17% overlap, corpus grows ~20%
chunk 350 tokens, overlap 175  -> 50% overlap, corpus grows ~100% for marginal recall

Verification

Confirm chunks respect boundaries and carry context:

python3 chunk.py --doc doc-8812 --show 2
# --- chunk 3 ---
# Installation Guide — Mounting > Brick walls
#
# When mounting on brick, use the supplied wall plugs...
# --- chunk 4 ---
# Installation Guide — Mounting > Brick walls
#
# ...use the supplied wall plugs. Drill to a depth of...   <- overlap present

Confirm retrieval improves against the evaluation set:

python3 evaluate_chunking.py --variants whole,fixed512,structural350
# whole            recall@10 0.52
# fixed512         recall@10 0.71
# structural350    recall@10 0.86

Confirm parent collapsing is working:

python3 search.py --query "mount bracket brick wall" --show-parents
# 1  doc-8812  (chunk 4)
# 2  doc-9104  (chunk 1)
# 3  doc-7731  (chunk 12)
# => no parent appears twice

Measuring chunking without a labelled set

The evaluation above assumes queries paired with the passages that answer them, and many teams do not have that. A cheaper proxy gets most of the way there and can be built in an afternoon.

Take fifty real queries from the logs. For each, run retrieval under two chunking variants and record whether the top three chunks come from the document a human would consider correct — which is usually obvious at a glance even without formal grading. That produces a comparable number for each variant without constructing a judgment set, and it is sufficient to choose between chunking strategies, which is a coarse decision.

python3 compare_chunking.py --queries real-50.txt --variants structural350,fixed512
# structural350  correct parent in top 3: 44/50
# fixed512       correct parent in top 3: 33/50

The formal judgment set is worth building eventually, for the finer decisions described in relevance evaluation. For chunking specifically, the differences are large enough that a rough measure decides it correctly.

Effect sizes for chunking compared with later tuning Chunking differences are large enough for a rough measure, while index parameter differences need a formal judgment set. chunking large — a rough count decides model choice moderate index params small — needs labels
The measurement effort should match the effect size. Chunking is coarse enough to settle without formal labels; the later steps are not.

Common Pitfalls

Changing chunking without re-embedding

Chunk boundaries and embeddings are inseparable: a new chunking strategy produces different text, which produces different vectors. Applying new chunking to newly ingested documents while leaving old ones as they were creates a corpus where retrieval quality depends on when a document happened to be indexed — a difference nobody will think to look for.

Chunking structured records as if they were prose

A product record with a title, twelve attributes and a short description is not a long document and should not be chunked at all. Chunking applies to long-form content; applying it to short structured records splits a coherent entity into fragments that are individually meaningless. The test is whether the whole record fits comfortably inside one chunk — if it does, embed it whole.

Overlap that duplicates the answer across chunks

Large overlaps mean the same passage appears in several chunks, so a query matching it retrieves three near-identical results. Parent collapsing hides this in the result list and it still consumes candidate slots. Keep overlap modest and rely on collapsing rather than using overlap to guarantee coverage.

Operational notes

Keep the chunker deterministic and versioned. Given the same document and the same parameters it must produce byte-identical chunks, or re-embedding produces gratuitous churn and comparing two runs becomes impossible. A unit test asserting stable output for a fixture document catches any accidental non-determinism introduced by a refactor.

Store the chunking parameters alongside the vectors, exactly as with the model version. A corpus embedded under one chunking strategy and queried alongside one embedded under another produces inconsistent retrieval, and without the parameters recorded there is no way to detect that state.

Measure chunking before comparing models. The variance between chunking strategies on a typical documentation corpus is larger than the variance between reasonable embedding models, so a model comparison run under bad chunking measures mostly noise. Fixing chunking first also makes the model comparison cheaper, because it narrows the differences you are trying to detect.

Order of work for a vector retrieval project Fix chunking first, then compare models, then tune index parameters, because each step narrows the variance the next one measures. 1. chunking largest variance 2. model choice moderate variance 3. index parameters smallest variance Running these in reverse order — the common instinct — measures noise at every step.
Chunking dominates, model choice matters, index parameters refine. Working in that order makes each measurement cleaner than the last.

A last note on content that resists chunking altogether: tables, code blocks and reference lists carry meaning in their structure, and splitting them mid-row or mid-function produces fragments that embed as noise. Treat those as atomic — one chunk per table or per code block, regardless of the token target — and accept the occasional oversized chunk rather than the guaranteed nonsense of a split one.