Chunking strategies for RAG: nine tested on 92k Dutch docs

It is a Tuesday morning at a municipal archive in the Randstad. An archivist types a query into a beta search tool we built: welke moties zijn er aangenomen over windenergie tussen 2018 en 2022? The system has to read back across 92,000 documents. Council resolutions, moties, college letters, zoning plans, most of them PDFs, some scanned with bad OCR, a handful still in DOCX from 2009. The wrong chunk boundary means the right motie sits two paragraphs away from whatever vector the search returns first. Recall is the whole game.

We spent the spring of 2026 benchmarking nine chunking strategies on that corpus. Same embedding model, same reranker, same query set. Only the chunker changed. This is the field guide we wish we had had on day one.

The corpus and the queries

The archive holds 92,184 documents from a single Dutch gemeente going back to 2002. Median length is around 1,400 words; the tail runs past 80,000 words for omgevingsvergunningen with appendices. Roughly 71% are born-digital PDFs, 22% are scanned PDFs (we re-OCRed everything with Surya before indexing), and the rest is a mix of DOCX, ODT, and HTML pulled from the old Joomla site.

Our evaluation set is 612 queries written by three working archivists, each tied to between one and twelve ground-truth documents they had already located by hand. The metric is recall@10 at the document level: did at least one chunk from a target document make the top ten?

The embedding model is held constant at jina-embeddings-v3, a multilingual model with an 8,192-token context window. The retriever is a flat HNSW index. The reranker is BGE-reranker-v2-m3 over the top 50. Nothing in that stack changes between runs. Only the chunker moves.

The baseline you have to beat

Most production RAG systems we audit at municipalities and law firms run a hybrid retriever: BM25 over chunks plus dense embedding retrieval, fused with reciprocal rank fusion. That is the bar. On this corpus, hybrid BM25 plus dense (chunks at fixed 512 tokens) lands at recall@10 = 0.78. Anything below that is a regression; anything above it has to justify the operational cost.

Nine chunkers, head to head

Each chunker was tuned for at least one full day, then locked. Numbers below are document-level recall@10 against the 612 labeled queries. Same dense-only retrieval, no hybrid, no reranker, so you can see what the chunker itself is doing.

Hybrid BM25 plus dense over strategy #1 hits 0.78. Late chunking on its own, with no BM25, beats it. With BM25 added on top, late chunking gets to 0.86. That is the headline.

Why fixed chunking lost (and why everyone still ships it)

Fixed-size chunking is in every quickstart for a reason: deterministic, fast, trivially parallel. It also throws away every signal the document gave you. A motie in a Dutch council resolution has a predictable shape: Beslispunt, Motivering, Stemverhouding. Chop that into 512-token bricks and the beslispunt ends up severed from its motivering half the time. The embedding is then a soup of two unrelated arguments.

The cheapest fix is also the most underrated: walk the document outline first. Strategy #8 (heading-aware) was four days of work, mostly regex and a markdown converter, and it added 16 points of recall over the fixed baseline. If you cannot ship anything else this quarter, ship that.

Late chunking, in one paragraph

Late chunking flips the usual order. Instead of slicing the document into chunks and then embedding each one in isolation, you embed the whole document once under full attention, then pool token-level embeddings into chunks afterwards. Every chunk's vector now carries context from the full document. The Stemverhouding chunk at the bottom of a motie knows it belongs to that motie. The technique was formalised by Jina AI in Günther et al., 2024, and it only works with long-context embedding models. Below 8k context, it is not interesting.

from transformers import AutoModel
import torch

model = AutoModel.from_pretrained(
    "jinaai/jina-embeddings-v3", trust_remote_code=True
)
tok = model.tokenizer

# 1. Tokenise the whole document once.
doc = open("besluit-2022-431.txt").read()
inputs = tok(doc, return_tensors="pt", truncation=True, max_length=8192)

# 2. Token embeddings under full-document attention.
with torch.no_grad():
    hidden = model(**inputs).last_hidden_state[0]

# 3. Pick chunk boundaries however you like (headings work well).
boundaries = pick_boundaries_from_outline(doc, tok)  # [(start, end), ...]

# 4. Mean-pool token vectors inside each chunk.
chunk_vectors = [hidden[s:e].mean(dim=0) for s, e in boundaries]

That is the whole trick. The boundaries can come from any of the other eight strategies. We used heading-aware boundaries inside the late-chunking pipeline, which is why strategy #9 reads as a superset of strategy #8.

Where the other contenders broke

A few results surprised us enough to be worth flagging.

Sentence-aware did worse than paragraph-aware

We expected the Dutch spaCy sentence splitter to outperform paragraph splitting. It did not. Council resolutions average 38 words per sentence and rarely fewer than three sentences per paragraph; packing six sentences into a chunk gave us awkward boundaries where two thirds of a motie sat in one chunk and the dispositive line (draagt het college op) in the next.

Semantic chunking was inconsistent

Embedding-drop chunking (#7) looks great on a clean policy memo and falls apart on a scanned 2009 PDF with OCR noise. The cosine "boundaries" landed inside sentences, around OCR garbage tokens. We retried with a cleaning pass and recovered most of the gap, but never beat structure-aware.

Sliding windows traded recall for latency

The sliding window at 1024 / 256 helped recall (0.69) but tripled the index size, and reranking the top-50 became the new bottleneck. On a four-year-old archive this matters less; on a live corpus that grows by 1,200 documents a month, it is the difference between a €180 / mo and a €640 / mo Postgres bill.

What to try first, in order

If you are currently shipping fixed 512-token chunks with no overlap, here is the order we would hand a small team.

1. Add overlap (64 tokens). One-line change. Buys you about 5 points.
2. Walk the document outline. Headings, numbered sections, frontmatter. About 10 to 12 more points.
3. Add hybrid BM25 if you do not already have it. About 4 points and it covers the cases dense retrieval misses (proper names, dossier numbers).
4. Only then consider late chunking. It needs a long-context embedding model, a GPU budget, and a re-indexing window. The recall gain is real, but it is the last 5 points, not the first.

The cost question nobody asked us

One thing the benchmark did not capture: tokenisation cost shifts at scale. Re-embedding the 92,000-document archive under late chunking with jina-embeddings-v3 ran us roughly €240 of GPU time on a rented A100, against €18 for the fixed-chunk baseline. For the gemeente, that is pocket change against an archivist's hour. For a SaaS pricing model where you ingest user docs on the fly, it is a different conversation. Smaller, quantised models like Google's Gemma 3 QAT release are starting to make on-laptop inference plausible for the embedding side too. We are watching that closely for the next benchmark.

What we shipped

When we built this for the gemeente, the thing we ran into was not the chunker. It was the OCR. About 22% of the archive came in as scans, and the recall difference between Tesseract and Surya was bigger than the difference between any two chunkers on this list. We ended up running Surya twice on anything that looked uncertain and reconciling the outputs. That is the unglamorous half of every RAG system we put into production.

If you only do one thing today, open three documents from your corpus and look at where the headings are. If the structure is there, your chunker should be using it before you read another paper.