Chunk size decides what the model gets to read. Make chunks too large and the sentence that answers the question is diluted by everything around it, so it ranks lower and may be cut off in the prompt. Make them too small and the answer gets separated from its conditions: "refunds are available within the refund window" lands in one chunk, "except on annual plans" in the next. LayBuild indexes every document at two sizes. This post covers the splitter, the numbers, what they cost you, and how to tell whether your chunks are the problem.

How the splitter works

Every source is converted to markdown first. Uploaded files (PDF, DOCX, PPTX, XLSX, CSV and the rest) go through a document converter, and web pages go through an HTML-to-markdown step. Then:

  • The document is split into sections at every heading from # to ####. Deeper headings (#####, ######) are not split points.
  • Inside a section, paragraphs (text separated by a blank line) are packed into a parent chunk until the next paragraph would push it past 350 tokens.
  • A paragraph that is too big on its own is split into sentences at ., ! and ?, and the sentences are packed instead.
  • Each new chunk starts with the last part of the previous one as overlap. For parents, the overlap target is 50 tokens, which the code converts to 37 words.
  • Every parent is then split again, by paragraph, into child chunks of up to 150 tokens with 25 tokens of overlap (18 words). Children do not split on headings, but they inherit the parent's heading path.
text
KnowledgeDocument (markdown)
  |
  split at #, ##, ### and #### headings
  |
  section, e.g. "Billing > Refunds"
     |
     parent chunk: up to 350 est. tokens, 50 overlap
     |    -> stored in Postgres (full-text leg) and Qdrant (vector leg)
     |
     +-- child chunk: up to 150 est. tokens, 25 overlap
     |      -> stored in Qdrant only
     +-- child chunk ...

Before embedding, each chunk is prefixed with the document title and its heading path, like Pricing FAQ > Billing > Refunds. A chunk from deep in a long page still carries the words of the headings above it. That is one reason good headings matter so much, which we cover in writing help docs an AI agent can retrieve.

When you edit a document, LayBuild hashes each new parent chunk and compares it with the stored ones. Only changed parents and their children are re-embedded. Unchanged chunks keep their vectors, and chunks that no longer exist are deleted from both Postgres and Qdrant.

Why we estimate tokens as characters divided by 4

Token counts in the splitter are characters / 4, rounded up. So 350 tokens is about 1,400 characters and 150 tokens about 600. We use the estimate because it is cheap, deterministic and doesn't depend on which model you configured. The embedding model's tokenizer differs from the chat model's anyway, so an exact count for one would be wrong for the other.

The cost is accuracy. English prose comes out close to the estimate. Code, URLs, long numbers and non-English text usually produce more real tokens per character, so a chunk of those can exceed its nominal size. The default embedding model, bge-small-en-v1.5, reads at most 512 tokens. A 350-token parent leaves room for the title prefix and some estimation error. That headroom is also why we don't recommend raising the parent size much.

What each size is for

Child chunks match narrow questions. "Where do I change the invoice email?" is answered by one or two sentences, and a 600-character child that contains them scores higher than a 1,400-character parent where they are a small part. Parent chunks match broader questions and give the model more surrounding context when they win.

Both sizes live in the same Qdrant collection, and the vector search returns whichever scores best. The collection also holds one vector per whole document, embedded from its title and content. Embedding models read a limited number of tokens, so for a long document that vector can only reflect its opening, and when it wins, the model sees the document's first 1,000 characters. That is one more reason to open every document with a summary.

LayBuild does not currently swap a matched child for its parent before building the prompt. Some systems do ("small-to-big" retrieval) so the model gets precision in matching and context in reading. The trade is not free here: retrieved passages share a 4,000-character budget, so four parents in place of four children would mean fewer distinct passages fit. The full-text leg searches parent chunks only.

The trade-offs in the numbers

The prompt limits matter as much as the chunk sizes. LayBuild sends at most 4 retrieved passages, cuts each one to 1,000 characters, and fits them into 4,000 characters shared with any pinned knowledge. A full parent chunk is about 1,400 characters, so when a full parent wins, its last few sentences don't reach the model. Put the answer near the start of a section and this rarely bites. Bury it at the end of a long section and it does.

Smaller chunks rank more precisely but lose context, and a question that needs two neighbouring paragraphs may retrieve only one. Larger chunks keep context but rank less precisely and get truncated.

Overlap reduces the chance of cutting a sentence's meaning in half at a boundary. It also costs storage and embedding calls, and two overlapping passages can both be retrieved. The prompt builder removes duplicates only when two passages share a title and the same first 96 characters, so near-duplicates with different starting points both use budget.

Where the splitter goes wrong

Many chunking problems come from the document's shape, not the numbers:

  • Tables. A markdown table with no blank lines inside is one paragraph. If it is larger than a parent chunk, it gets split at periods, which cuts rows apart and separates values from their column headers.
  • Periods that are not sentence ends. "e.g.", "v2.1" and URLs split sentences in the wrong place. This only happens inside paragraphs too long to fit a chunk, which is one more reason to keep paragraphs short.
  • One very long sentence. A single sentence longer than the limit becomes its own oversized chunk.
  • Missing headings. A PDF exported without heading styles comes through as one long section, so chunks cross topic boundaries and carry no heading path.
  • Deep nesting. Content under ##### headings stays in the parent #### section.

How to tell if your chunks are wrong

Symptoms first. The answer is correct but missing a condition or exception. The bot replies "I do not have specific information about that" to questions a doc clearly answers. The same passage appears twice. Answers quote the middle of a long section without its heading's context.

Then check. Take 20 questions that went badly and, for each one, read the passages that were actually retrieved. If you self-host with Langfuse tracing enabled, the rag-retrieval span lists the IDs, titles and scores of every passage that passed the 0.4 floor. For each question, ask whether the retrieved passage, read alone, answers it. Sort the failures into three groups:

  • The right passage was never retrieved. That is a retrieval or content problem. Look at wording and headings first.
  • The right passage was retrieved but the answer was cut off or split across chunks. That is a chunking or prompt-budget problem.
  • The right passage was retrieved whole and the answer was still wrong. That is a generation problem, and chunk sizes won't fix it.

Usually the fix is in the document: shorter sections, the answer first, headings phrased the way customers ask. Change the numbers only after that.

Changing the numbers on a self-hosted install

The four values are environment variables: PARENT_CHUNK_SIZE (default 350), PARENT_CHUNK_OVERLAP (50), CHILD_CHUNK_SIZE (150) and CHILD_CHUNK_OVERLAP (25). New settings apply to documents as they are added or edited. To re-chunk everything, run bun run rag:backfill from apps/api. Every parent's content changes, so every chunk is re-embedded, which costs embedding calls in proportion to your corpus. Run your question set before and after. Our guide to benchmarking retrieval covers how to build one.

Where to go next

For how chunks are searched and ranked once they exist, read how LayBuild runs hybrid search. The splitter lives in apps/api/src/services/docParser.ts in the LayBuild repository.