Chunk size decides what the model gets to read. Make chunks too large and the sentence that answers the question is diluted by everything around it, so it ranks lower and may be cut off in the prompt. Make them too small and the answer gets separated from its conditions: "refunds are available within the refund window" lands in one chunk, "except on annual plans" in the next. LayBuild indexes every document at two sizes. This post covers the splitter, the numbers, what they cost you, and how to tell whether your chunks are the problem.
How the splitter works
Every source is converted to markdown first. Uploaded files (PDF, DOCX, PPTX, XLSX, CSV and the rest) go through a document converter, and web pages go through an HTML-to-markdown step. Then:
- The document is split into sections at every heading from
#to####. Deeper headings (#####,######) are not split points. - Inside a section, paragraphs (text separated by a blank line) are packed into a parent chunk until the next paragraph would push it past 350 tokens.
- A paragraph that is too big on its own is split into sentences at
.,!and?, and the sentences are packed instead. - Each new chunk starts with the last part of the previous one as overlap. For parents, the overlap target is 50 tokens, which the code converts to 37 words.
- Every parent is then split again, by paragraph, into child chunks of up to 150 tokens with 25 tokens of overlap (18 words). Children do not split on headings, but they inherit the parent's heading path.
KnowledgeDocument (markdown)
|
split at #, ##, ### and #### headings
|
section, e.g. "Billing > Refunds"
|
parent chunk: up to 350 est. tokens, 50 overlap
| -> stored in Postgres (full-text leg) and pgvector (vector leg)
|
+-- child chunk: up to 150 est. tokens, 25 overlap
| -> stored in pgvector only
+-- child chunk ...Before embedding, each chunk is prefixed with the document title and its heading path, like Pricing FAQ > Billing > Refunds. A chunk from deep in a long page still carries the words of the headings above it. That is one reason headings matter, which we cover in writing help docs an AI agent can retrieve.
When you edit a document, LayBuild hashes each new parent chunk and compares it with the stored ones. Only changed parents and their children are re-embedded. Unchanged chunks keep their vectors, and chunks that no longer exist are deleted from both Postgres and pgvector.
Why we estimate tokens as characters divided by 4
Token counts in the splitter are characters / 4, rounded up. So 350 tokens is about 1,400 characters and 150 tokens about 600. We use the estimate because it is cheap, deterministic and doesn't depend on which model is in use. The embedding model's tokenizer differs from the chat model's anyway, so an exact count for one would be wrong for the other.
The cost is accuracy. English prose comes out close to the estimate. Code, URLs, long numbers and non-English text usually produce more real tokens per character, so a chunk of those can exceed its nominal size. The default embedding model, bge-small-en-v1.5, reads at most 512 tokens. A 350-token parent leaves room for the title prefix and some estimation error. That headroom is also why we don't recommend raising the parent size much.
What each size is for
Child chunks match narrow questions. "Where do I change the invoice email?" is answered by one or two sentences, and a 600-character child that contains them scores higher than a 1,400-character parent where they are a small part. Parent chunks match broader questions and give the model more surrounding context when they win.
Both sizes live in the same pgvector table, and the vector search returns whichever scores best. The table also holds one vector per whole document, embedded from its title and content. Embedding models read a limited number of tokens, so for a long document that vector can only reflect its opening, and when it wins, the model sees the document's first 1,000 characters. That is one more reason to open every document with a summary.
LayBuild does not currently swap a matched child for its parent before building the prompt. Some systems do ("small-to-big" retrieval) so the model gets precision in matching and context in reading. The trade is not free here: retrieved passages share a 4,000-character budget, so four parents in place of four children would mean fewer distinct passages fit. The full-text leg searches parent chunks only.
The trade-offs in the numbers
The prompt limits matter as much as the chunk sizes. LayBuild sends at most 4 retrieved passages, cuts each one to 1,000 characters, and fits them into 4,000 characters shared with any pinned knowledge. A full parent chunk is about 1,400 characters, so when a full parent wins, its last few sentences don't reach the model. Put the answer near the start of a section and this rarely bites. Bury it at the end of a long section and it does.
Smaller chunks rank more precisely but lose context, and a question that needs two neighbouring paragraphs may retrieve only one. Larger chunks keep context but rank less precisely and get truncated.
Overlap reduces the chance of cutting a sentence's meaning in half at a boundary. It also costs storage and embedding calls, and two overlapping passages can both be retrieved. The prompt builder removes duplicates only when two passages share a title and the same first 96 characters, so near-duplicates with different starting points both use budget.
Where the splitter goes wrong
Many chunking problems come from the document's shape, not the numbers:
- Tables. A markdown table with no blank lines inside is one paragraph. If it is larger than a parent chunk, it gets split at periods, which cuts rows apart and separates values from their column headers.
- Periods that are not sentence ends. "e.g.", "v2.1" and URLs split sentences in the wrong place. This only happens inside paragraphs too long to fit a chunk, which is one more reason to keep paragraphs short.
- One very long sentence. A single sentence longer than the limit becomes its own oversized chunk.
- Missing headings. A PDF exported without heading styles comes through as one long section, so chunks cross topic boundaries and carry no heading path.
- Deep nesting. Content under
#####headings stays in the parent####section.
How to tell if your chunks are wrong
Symptoms first. The answer is correct but missing a condition or exception. The bot replies "I do not have specific information about that" to questions a doc clearly answers. The same passage appears twice. Answers quote the middle of a long section without its heading's context.
Then check. Take 20 questions that went badly and, for each one, find the passage you expected the agent to use. For each question, ask whether that passage, read alone, answers it. Sort the failures into three groups:
- The right passage was never retrieved. That is a retrieval or content problem. Look at wording and headings first.
- The right passage was retrieved but the answer was cut off or split across chunks. That is a chunking or prompt-budget problem.
- The right passage was retrieved whole and the answer was still wrong. That is a generation problem, and chunk sizes won't fix it.
Usually the fix is in the document: shorter sections, the answer first, headings phrased the way customers ask.
Frequently asked questions
What is parent-child chunking in RAG?
Parent-child chunking indexes each document twice: as small child chunks (up to 150 tokens in LayBuild) for precise matching, and as the larger parent chunks they come from (up to 350 tokens in LayBuild). Some systems inject the whole parent into the LLM prompt when a child matches, for more context. LayBuild does not make that swap; it stores both sizes in the vector table and sends whichever matched, and its full-text leg searches parent chunks only.
What is the optimal chunk size for RAG retrieval?
There is no size that is right for every knowledge base, so test on your own questions. LayBuild uses parents of up to 350 estimated tokens with 50 tokens of overlap and children of up to 150 with 25, which keeps chunks inside the default embedding model's 512-token input.
Why does fixed-length character chunking fail in production?
Fixed-length chunking cuts wherever the character count runs out, often mid-sentence or mid-table, so a condition like "except on annual subscriptions" can land in a different chunk from the rule it qualifies. Splitting at headings and paragraphs keeps each section together and lets every chunk carry its heading path.
Where to go next
For how chunks are searched and ranked once they exist, read how LayBuild runs hybrid search.