Doubling the chunk fixes 52% of broken tables and costs 19% of the signal

Ask a retrieval system for the late-payment rate in a contract, then open the contract and find a different figure. The model answered correctly from the chunk it was given: a block of numbers with no row naming the columns.

The model was never wrong. At LlamaIndex’s shipped defaults — 1,024-token chunks with 200 tokens of overlap — a 400-token table is cut off from its header row 24.3 per cent of the time, because chunk starts advance 824 tokens and only 624 of those positions leave room for the whole table.

Why a retrieval system gets a number wrong when the document has it right

The splitter did it, long before any model was involved. It cut the document into fixed-length windows, one of them ended between the caption and the figures, and the halves went into the index as records that never meet again. Blame the model and you’ll spend a fortnight rewriting prompts that can’t reach the problem.

Anthropic puts the same failure in prose in its Contextual Retrieval post: “But these traditional RAG systems have a significant limitation: they often destroy context.” Its example is a chunk saying revenue “grew by 3% over the previous quarter” and never saying whose. What the post doesn’t print is how often that happens, which two integers in your splitter’s parameter table will tell you.

Chunking is the first thing I look at on an applied AI build, ahead of the prompt and the model. Retrieval depth stays at twenty throughout, because the one tolerance I borrow below was measured there.

The default chunk overlap in LlamaIndex is 200 tokens, not 20

LlamaIndex’s SentenceSplitter ships with chunk_size at 1,024 tokens and chunk_overlap at 200, while the framework’s own guidance page says 20. You might expect the two to agree. Only one describes the splitter you get.

The parameter table for SentenceSplitter sits behind fifteen other classes on LlamaIndex’s node-parser API reference, four-fifths of the way down, and its DEFAULT column carries both: 1024 for chunk_size, 200 for chunk_overlap. The Basic Strategies page, under its Chunk Sizes heading, says otherwise in prose: “The default chunk size is 1024, while the default chunk overlap is 20.”

People have argued about that pair on the forums for years, so the observation isn’t mine. The proof is. Line 11 of constants.py reads DEFAULT_CHUNK_OVERLAP = 20 # tokens; line 22 of sentence.py reads SENTENCE_CHUNK_OVERLAP = 200, both at commit 619d746. Settings builds the splitter for anyone who never sets a node parser, and nothing on that path reads the constant the prose describes. An interpreter settles it:

from llama_index.core.node_parser import SentenceSplitter
print(SentenceSplitter().chunk_size, SentenceSplitter().chunk_overlap)  # 1024 200

LangChain counts something else. Its TextSplitter reference describes chunk_overlap as “Overlap in characters between chunks”, and length_function defaults to plain len. RecursiveCharacterTextSplitter splits on ["\n\n", "\n", " ", ""], where nothing tells a table row from prose: a row ends in a newline exactly as a paragraph does.

(A third parameter in that table, secondary_chunking_regex, cuts at commas, full stops and semicolons. It gets collected later.)

The general case for chunking on a document’s own units is in Most fine-tuning requests are really retrieval problems. This piece does the table.

Check the probability that a table is split across two chunks

Subtract the overlap from the table length, divide by the stride — chunk size minus overlap — and you have the share of start positions at which the table cannot fit one window. The shipped stride is 1,024 − 200 = 824 tokens.

Chunk i covers tokens 824i to 824i + 1,024, so the last 200 tokens of every window are the first 200 of the next. A table of length T starting at offset p inside a stride survives whole when p + T is at most 1,024, which holds at 1,024 − T of the 824 offsets. At T = 400, 624 are safe and 200 are not; call 200 ÷ 824 = 0.24272 what it is: 24.3 per cent.

Treat that fixed stride as an idealisation, because SentenceSplitter doesn’t advance by one. It fills a chunk until the next sentence would overflow, then seeds the next by walking back through whole sentences while the carried total stays at or under chunk_overlap. So real windows run at or under 1,024 tokens and the real overlap at or under 200, sometimes zero. Shorten either and the probability rises: every figure below is a floor. TokenTextSplitter is the fixed-stride one, and it reads DEFAULT_CHUNK_OVERLAP, so it strides 1,004.

Two clamps bound the line. Below T = 200 the table is shorter than the overlap and always fits one window, so the probability is zero; at or above T = 1,024 it is one. Between them it is a straight line in T.

Probability a table is cut across a boundary, by table length in tokens, at chunk_size 1,024 and chunk_overlap 200:

The estimate most people reach for is wrong by a factor of two. “My overlap is 200 and my tables run 400, so half survive” gives 50 per cent against a real 24.3, or 50 ÷ 24.272 = 2.06 times too high. The overlap covers the gap between chunk starts, not the table, and that gap is 824 tokens.

Six lines:

def p_split(table_tokens, chunk_size=1024, chunk_overlap=200):
    stride = chunk_size - chunk_overlap
    if table_tokens <= chunk_overlap:
        return 0.0
    if table_tokens >= chunk_size:
        return 1.0
    return (table_tokens - chunk_overlap) / stride

How many tables in a 400-document corpus lose their header?

At LlamaIndex’s shipped defaults, 272 of Lamplighter’s 1,200 tables reach the model in a chunk without their header row. Lamplighter is a fiction this site keeps reusing, a retrieval system over a contracts archive that doesn’t exist. I picked the counts so every division lands where a reader can check it. What I didn’t pick are the splitter defaults and three figures from Anthropic’s Contextual Retrieval post, each quoted below where it is used.

Four hundred contracts at 8,000 tokens each, a length lifted from that post’s cost model and its “8k token documents”, make 400 × 8,000 = 3,200,000 tokens. A document takes (8,000 − 200) ÷ 824 = 9.47 windows, rounded up to 10, so 400 documents give 4,000 chunks. Tables run in four bands: 930 at 300 tokens, 200 at 600, 50 at 900 and 20 at 1,800, so 279,000 + 120,000 + 45,000 + 36,000 = 480,000 tokens over 1,200 tables, a mean of 400.

Route A sums the bands: 930 × 100 ÷ 824 = 112.86, 200 × 400 ÷ 824 = 97.09, 50 × 700 ÷ 824 = 42.48, and the twenty 1,800-token tables past the clamp at one apiece, for 272.43. Route B cuts at the clamp instead. The 1,180 tables under 1,024 tokens hold 444,000 tokens, a mean of 444,000 ÷ 1,180 = 376.27, and the line carries them all: (376.27 − 200) ÷ 824 = 0.21392, then 0.21392 × 1,180 = 252.43. Add the clamped twenty and both give 272, or 22.7 per cent.

Run the same corpus through LangChain’s character defaults and the figure moves to 425. TextSplitter declares chunk_size 4,000 and chunk_overlap 200 in characters, so at four characters to a token, a convention rather than a measurement, the window is 1,000 tokens on a 50-token overlap. The bands give 930 × 250 ÷ 950 = 244.74, 200 × 550 ÷ 950 = 115.79, 50 × 850 ÷ 950 = 44.74 and twenty past the clamp, for 425.27. Thirty test cases cannot tell ninety per cent from eighty prints 442 for this corpus, from the mean rather than the bands; the 17-table gap is the clamp a mean cannot see. The 153 tables between 425 and 272 come from the unit on one integer: 200 tokens against 200 characters.

What a bigger chunk costs: embedding dilution under mean pooling

Doubling chunk_size to 2,048 tokens removes 52.3 per cent of the broken tables and costs 19.4 per cent of the cosine. The benefit first. Stride goes to 2,048 − 200 = 1,848 and the 1,800-token band drops under the clamp, so every band is on the line: 480,000 − 240,000 = 240,000 excess tokens, then 240,000 ÷ 1,848 = 129.87, call it 130 tables, down from 272. The reduction is 1 − 129.87 ÷ 272.43 = 0.52329, below the bare ratio of strides, 1 − 824 ÷ 1,848 = 0.5541, because the longest band was already pinned at one.

Now the price. Most sentence encoders average their token vectors, and Section 3 of Sentence-BERT says which: “We experiment with three pooling strategies: Using the output of the CLS-token, computing the mean of all output vectors (MEAN-strategy), and computing a max-over-time of the output vectors (MAX-strategy). The default configuration is MEAN.” Seven years on, sentence-transformers’ Pooling class still defaults pooling_mode to 'mean'. The paper’s own ablation in Section 6 cuts both ways: pooling has “a rather minor impact” under a classification objective but “a large impact” under a regression one, which is the setting here. So I lean on MEAN being the default, not on the choice being harmless.

Call the rest a simplification, and the assumptions go first. Take a chunk of n tokens. Thirty are the answer sentence and point the query’s way, a count picked like the corpus counts; the other n − 30 are unit vectors orthogonal to the query and to one another. Mean pooling sums and divides. The query-facing part grows with the answer sentence while the rest adds in quadrature, so the cosine is 30 ÷ sqrt(900 + n − 30).

Route A: 30 ÷ sqrt(1,894) = 0.68934 at n = 1,024 and 30 ÷ sqrt(2,918) = 0.55537 at n = 2,048, a fall of 1 − 0.55537 ÷ 0.68934 = 0.19435. Route B rebuilds it from parts: signal 30 ÷ 1,024 = 0.029297, filler norm sqrt(994) ÷ 1,024 = 0.030789, vector sqrt(0.029297² + 0.030789²) = 0.042500, and 0.029297 ÷ 0.042500 = 0.6893. The rungs at n = 2,048 are 0.014648, 0.021935 and 0.026376, for 0.5554. Both match Route A to four places.

Three estimates, and the first two are wrong. Halve the signal share and you predict a 50 per cent fall. Notice the quadrature and you predict 1 − 1 ÷ sqrt(2) = 29.3 per cent instead. The real figure is 19.4, because 900 is a big share of the 1,894 under the root. Signal and filler just don’t trade evenly.

The answer sentence defends part of its own denominator.

But the orthogonality assumption is the weak point and it leans one way. Contract prose shares vocabulary with a contract query — party names, dates, notice, interest — so the filler isn’t orthogonal and the true fall might be well under 19.4 per cent; nobody here has run that measurement, and I’d want it on your corpus before signing off a chunk size. Cosine is not recall, and I won’t convert one into the other.

Header-aware splitting against a bigger chunk: where the crossover is

Prepending the caption and header row to every chunk that continues a table clears all 272 detachments for 2.5 per cent of the cosine. A uniform chunk clears none, since the probability only reaches zero below T = 200, so the two cannot be compared at zero. The honest case for the bigger chunk goes first. It is one integer on one line: no parser, no heading convention to agree with whoever makes the documents, no second code path. And it wins outright where documents carry no headings a parser can find.

Costed on Lamplighter, it loses. At 2,048 tokens 130 tables are still broken, so we’re reduced to a rate both can reach, and I’ll use 5.7 per cent. Anthropic’s Contextual Retrieval post reports a baseline top-20 chunk retrieval failure rate of 5.7 per cent against a metric it defines as “1 minus recall@20”. That is a vendor’s number on a vendor’s corpora, not a law of retrieval. But it beats a round 5 per cent. Across 1,200 tables it allows 0.057 × 1,200 = 68.4 broken ones. Every band is on the line once the window passes 1,800 tokens, so the 240,000 excess tokens need a stride of 240,000 ÷ 68.4 = 3,508.77. That puts chunk_size at 3,709, where the cosine is 30 ÷ sqrt(4,579) = 0.44334, a fall of 1 − 0.44334 ÷ 0.68934 = 0.35686. OpenAI’s embeddings guide prints 8192 as the max input for text-embedding-3-small, so 3,709 fits easily. The arithmetic stops you, not the API.

Now the header, in the same units and costed at the worst case. Anthropic’s cost model assumes “100 tokens of context per chunk”, close enough to a caption plus a header row and published rather than invented. Only the 272 chunks continuing a table need it, so charge all 4,000 anyway: 100 × 4,000 = 400,000 extra tokens, against an index holding 3,200,000 plus the 400 × 9 × 200 = 720,000 the overlap duplicates, so 400,000 ÷ 3,920,000 = 10.2 per cent more text, once. The chunk goes to 1,124 tokens. The cosine goes to 30 ÷ sqrt(1,994) = 0.67183, a fall of 1 − 0.67183 ÷ 0.68934 = 0.02540. Two and a half per cent.

19.435 ÷ 2.540 = 7.65. Against the doubled chunk, the move most teams actually reach for, the uniform window costs seven and a half times as much signal and leaves 130 tables broken where the header leaves none. Raising chunk_size is the cheap-looking fix and the expensive one. I’d argue against it on a call.

secondary_chunking_regex gets collected here, and it bites exactly twenty times. _split returns a run whole the moment its token count is at or under chunk_size, so the backup regex and separator are never consulted on anything shorter — in this corpus, the 1,800-token tables and nothing else. There the regex has only the figures’ own thousands separators and decimal points to cut on, and separator at its default single space finishes the job, so the cut lands mid-row between two words.

When to leave the chunk size alone

A table shorter than about 247 tokens carries its own detachment under the 5.7 per cent tolerance, and where most of a corpus sits below that length I wouldn’t touch the splitter. The tolerance is the borrowed one from the last section. The distribution is Lamplighter’s four bands. The value falls out of the two: (T − 200) ÷ 824 = 0.057 gives T = 200 + 0.057 × 824 = 246.97. Call it the detachment floor, 247 tokens. Lamplighter’s shortest band sits at 300 and its corpus rate is 22.702 ÷ 5.7 = 3.98 times the baseline, which is why it acts.

Now derate the number a reader is least sure of. Halve every table length and the mean lands on 200, exactly this splitter’s overlap, where a single-point estimate says zero. The bands disagree. The 930 tables at 150 tokens do fall to zero, but 200 × 100 ÷ 824 = 24.27, 50 × 250 ÷ 824 = 15.17 and 20 × 700 ÷ 824 = 16.99 still add to 56.43, or 4.7 per cent of 1,200 — and only there does it clear the tolerance. The decision flips, and this article’s recommendation goes with it. The evaluation piece ran the same halving against LangChain’s defaults and it did not flip, because 200 characters of overlap converts to 50 tokens rather than 200: (200 − 50) ÷ 950 = 0.158 leaves 190 tables broken.

Double the corpus and nothing moves: 2,400 tables, 272.43 × 2 = 544.86, so 545 detached rather than 272, at the same 22.7 per cent. A rate is scale-free and a count is not. Measure your mean table length first; if it’s short we’d tell you to spend the money elsewhere rather than take the work.

Without headings a parser can find, the floor stops working. The fix sits upstream, in whatever turns a PDF into text. LangChain’s MarkdownHeaderTextSplitter says so in its own signature: headers_to_split_on carries no default and is required, so the convention is yours to supply. And one more default could cost you the exercise. strip_headers is True, documented as “Strip split headers from the content of the chunk”, so the header-aware splitter, as shipped, lifts the header out of page_content and keeps it in metadata. Metadata is not what gets embedded.