The share of questions your AI agent answers correctly is mostly decided by your content, not by the model. When an agent that answers only from your documentation misses, the cause is nearly always one of four things: the answer is not written down, it is written down in a form retrieval cannot find, the answer was blocked by a grounding check, or the question needs data the agent cannot reach. This playbook is a loop for finding which one, fixing it, and checking that the fix worked. We do not promise a target percentage; the ceiling depends on how many of your questions have written answers at all.

Decide what "answered correctly" means before you measure

"Resolution rate" and "deflection rate" usually count conversations that did not reach a person. That includes customers who got a wrong answer and left. For this playbook, count something stricter: a customer question the AI answered, and a human reviewer agrees the answer was correct and complete.

That needs a labeled sample. Take a few hundred recent customer questions from your transcripts (not conversations, questions, since one conversation can contain several) and label each one:

  • Answered correctly.
  • Answered, but wrong or incomplete.
  • Fixed no-answer reply ("I do not have specific information about that in the knowledge base...").
  • Handed off to a person.
  • Out of scope: something support should not answer, or that needs account data.

The first label divided by everything except "out of scope" is your number. It is slower than reading a dashboard, and it is the only version of the number that means what it says. Keep this question set; you will reuse it.

Find the misses worth fixing first

You have two sources for where the AI falls short.

The fixed no-answer reply always starts with the same sentence, so you can count and collect the questions behind it. If you self-host, query the Message table for assistant messages that start with "I do not have specific information about that". On the hosted service, the MESSAGE_RECEIVED webhook fires for AI replies too and carries the role and content, so you can log them on your side.

The Knowledge Q&A page in the dashboard ranks the questions customers ask most often through the web widget, with a count of how many times each was asked. Questions are grouped by their normalized text (lowercase, punctuation removed), so "How do I reset my password?" and "how do i reset my password" count together, but a reworded question counts separately. Treat the ranking as a guide to frequency, not a precise tally.

Cross the two: a question that is both frequent and often met with the no-answer reply is your first fix.

Diagnose why each miss happened

For each question on your list, find which of the four causes applies. The fix is different for each.

The answer is not written down

Check this first, because it is the easiest to fix. Nobody wrote the refund policy for digital goods, or the setup steps for the new integration. Write it.

The answer exists but retrieval does not find it

LayBuild splits documents on markdown headers first, then paragraphs, then sentences, into parent chunks of about 350 tokens and child chunks of about 150. Search runs in parallel across a vector index, Postgres full-text search, full-text over your Q&A pairs and a knowledge-graph leg, and the results are merged with reciprocal rank fusion. Results below a minimum relevance score are dropped.

That design fails in predictable ways:

  • The answer is buried in a long section with a vague header ("Other information"), so the chunk it lands in is about several things at once.
  • The customer's words and your document's words do not overlap. Customers say "cancel my subscription"; your docs say "terminate your plan".
  • The answer lives in a table or a scanned PDF that did not convert cleanly to text.
  • The question is in a language other than English. The default embedding model and full-text configuration are English, so non-English questions against English documents retrieve less reliably.

To check, ask the question yourself as a test customer and compare the reply with the document you expected it to use. If you self-host, the retrieved sources are stored in the metadata of each AI message in Postgres, which makes this quicker. If the right document never shows up, this is your cause.

The answer was retrieved but blocked

After the model writes an answer, LayBuild checks how many of the answer's stemmed words appear in the retrieved sources. Below 25%, the answer is replaced with the fixed no-answer reply. This is a lexical overlap check. It can block a correct answer that paraphrases heavily, and it can let through a wrong answer that reuses the source's vocabulary. If the right source was retrieved and the customer still got the fixed reply, rewriting the source in plainer words that customers and the model are likely to reuse often helps.

The question needs data the agent cannot reach

"Where is my order?" for a specific order, "why was I charged twice?", "change the email on my account". LayBuild's API tools call an endpoint with the defaults you configure after the answer is generated; they do not pick an order number out of the message. These questions belong with a person or a self-serve page. Mark them out of scope in your sample and stop counting them against the AI.

Apply the right fix

Each fix has a cost, so match it to the cause.

Rewrite or add documentation when content is missing or poorly structured. Give each answer its own section with a header that names the question the way customers ask it ("Can I get a refund on a digital product?"). Because chunking starts at headers, a well-named section tends to become a clean, focused chunk.

Add a Q&A pair when customers ask one question in a fairly fixed way and the answer is short. Q&A pairs have their own retrieval leg, so they are a direct way to cover wording mismatches. The cost is maintenance: a Q&A pair does not update when the underlying policy changes, so keep a list and review it when policies change.

Pin a document when every conversation needs a piece of information, such as current shipping cut-off dates or an incident notice. Pinned documents skip retrieval and go into every prompt, capped at 4,000 characters per document and 12,000 characters in total. That space is shared with everything else in the prompt, so pin sparingly and unpin when the information stops being urgent.

Re-add URLs after the page changes. LayBuild fetches the URLs you list when you add them; it does not crawl, and it does not re-sync on a schedule. An edited help page is stale in the knowledge base until you add it again.

If you self-host, there are retrieval options that are off by default: BM25 keyword search and an LLM listwise reranker over the top results. Turn one on at a time and measure it against your question set before keeping it. Both can add latency, and the reranker adds an extra model call per question.

Measure the change yourself

After a round of fixes, replay the same question set through the widget or the REST API and label the answers again. Compare the two runs question by question, not just the totals, because a fix for one question can change retrieval for its neighbors.

An illustration with made-up numbers: suppose your first labeled sample of 200 questions has 20 out of scope, 110 answered correctly, 15 answered wrongly and 55 fixed no-answer replies. Your starting number is 110 out of 180, about 61%. If a round of fixes turns 30 of those no-answer replies into correct answers and introduces no new wrong ones, the next run is 140 out of 180, about 78%. Whatever your real numbers are, track the wrong answers as closely as the total. A change that turns no-answer replies into confident wrong answers makes the headline number go up and your support worse.

Keep an eye on what the agent learns by itself

LayBuild has a learning loop that runs after each AI reply. When an exchange looks like a real answer (not a refusal, not the fixed no-answer reply, not a greeting, and within length limits), it is published into the knowledge base as a document and a Q&A pair, without human approval. If the customer later rates the conversation 2 or lower, that learned knowledge is retracted.

This helps coverage, and it also means a wrong answer the customer did not rate can become a source for future answers. When you run this playbook, include a review of recently learned entries, and delete any that are wrong. We think the trade-off is worth being explicit about, and we would rather you check it than trust it.

When to stop

The loop has diminishing returns. Once the remaining misses are mostly account-specific questions, one-off edge cases, or topics you have decided a person should handle, more content work will not move the number much. That is the point to switch your effort to the handoff experience instead.

Next steps