Improving support agent answers with evidence, not guesses

Most answer quality problems in a retrieval-based support agent are content problems, not prompt problems. The customer asked something your documents do not cover, or cover in a form retrieval cannot find. Rewriting the prompt rarely fixes either. So the loop that works is simple and a little boring: collect the conversations where the agent failed, sort each failure by cause, fix the cause, and check the fix. This post describes that loop with LayBuild, including the part of it LayBuild runs automatically and the trade-off that comes with it.

Three kinds of evidence

You have three signals worth reading every week. Each one catches different failures.

Fixed no-answer replies

When retrieval finds nothing relevant, or the answer fails the grounding check, LayBuild replaces the answer with a fixed message that starts "I do not have specific information about that in the knowledge base." Every one of those is a question a customer asked that your content did not answer, at least as far as the agent could tell. This is the richest signal you have, and it does not depend on the customer's mood.

Conversation search in the dashboard covers the subject and the customer's name, not message text, so you cannot filter for these replies there today. If you self-host, query Postgres directly. This pulls each fixed reply with the question that preceded it:

-- Fixed no-answer replies from the last 30 days, with the customer message before each
SELECT c.id AS conversation_id, q.content AS question, m."createdAt" AS replied_at
FROM "Message" m
JOIN "Conversation" c ON c.id = m."conversationId"
JOIN LATERAL (
  SELECT content
  FROM "Message"
  WHERE "conversationId" = m."conversationId"
    AND role = 'USER'
    AND "createdAt" < m."createdAt"
  ORDER BY "createdAt" DESC
  LIMIT 1
) q ON true
WHERE m.role = 'ASSISTANT'
  AND m.content LIKE 'I do not have specific information about that in the knowledge base.%'
  AND m."createdAt" > now() - interval '30 days'
  AND c."orgId" = $1
ORDER BY m."createdAt" DESC;

On the hosted service, read through recent conversations in the dashboard and note the questions that got this reply; a spreadsheet with the question, the date and your diagnosis is enough.

Low ratings

Customers can rate a conversation from 1 to 5, with an optional comment. The analytics page shows the average, the share of ratings that are 4 or higher, and how many of each score you received, and each rating is also written to the audit log with its comment. Ratings of 1 and 2 are your second review queue. Unlike fixed replies, they catch answers that were given confidently and were wrong, which is the failure that hurts most.

Treat ratings as a pointer, not a verdict. A rating covers a whole conversation, including things the agent cannot control: a refund policy the customer dislikes, a slow human handoff, a product bug. Read the transcript before deciding the agent was at fault. Also remember that most customers never rate, so the absence of low ratings does not mean the absence of wrong answers.

If you self-host:

SELECT id, subject, rating, "ratingComment", "ratingSubmittedAt"
FROM "Conversation"
WHERE "orgId" = $1 AND rating <= 2
ORDER BY "ratingSubmittedAt" DESC
LIMIT 50;

Handoffs

Conversations that moved to a person, whether the customer asked or the agent could not help, show up under the handoff filters in the conversations list. Read what the human answered. If a teammate typed the same explanation three times this week, that explanation belongs in your knowledge base.

Sort each failure by cause before fixing it

For each conversation you pull, decide which of these happened. The fix differs for each.

• The content does not exist. Nobody wrote down the answer. Add it: a Q&A pair for a short factual answer, or a document for anything longer.
• The content exists but was not retrieved. The answer is in a document, but under different words than the customer used, or buried in a long section. Add a Q&A pair phrased the way customers ask, or restructure the document so each topic sits under its own markdown heading, since LayBuild splits documents on headings first when chunking.
• The content was retrieved but the answer was rejected. The grounding check needs at least 25% of the answer's content words to appear in the sources. A heavy paraphrase can fail it even when correct. Content that states the answer plainly, in the words a customer would use, gives the model less reason to paraphrase.
• The answer was wrong and passed. The grounding check is lexical, so a wrong answer built from the source's own words can get through. Usually the source is ambiguous or out of date. Fix or remove the source; do not try to prompt around it.
• The refusal was correct. The question was out of scope. Nothing to fix, although a recurring out-of-scope question may deserve a line in your widget greeting.

Change prompts last. Prompt edits affect every answer, so a change that fixes one failure can shift a hundred good answers in ways you will not see until customers do. When a prompt change is justified, for example the tone is wrong everywhere or the agent keeps asking for information it should not, keep a copy of the old prompt yourself, because the prompt editor does not keep version history. LayBuild appends its grounding, handoff and anti-injection directives to any custom prompt, so editing the persona cannot remove those rules.

Check that the fix worked

Before closing an item, ask the original question again in the widget, and one or two rephrasings of it. If you keep a golden question set (see regression-testing a support agent), add the question to it so a later content change cannot silently undo the fix.

The automatic part: auto-learning, and its trade-off

LayBuild also learns from conversations on its own. After each AI reply, it checks whether the exchange is worth keeping. An exchange qualifies when the answer is not a refusal, not the fixed no-answer reply and not a greeting, the question is at least 8 characters, the answer is between 40 and 8,000 characters, and the question is not a near duplicate (80% word containment) of one already learned. A qualifying exchange is published straight into the knowledge base as a document and a Q&A pair tagged as learned. There is no approval step.

If the customer later rates that conversation 2 stars or lower, LayBuild retracts what it learned from it: the learned document is deleted and matching learned Q&A pairs are removed.

We made this choice deliberately, and you should understand what it costs.

• The benefit: answers the agent already gave, which passed the grounding check, become directly retrievable for the next customer who asks the same thing in similar words. Common questions get faster, more consistent answers without anyone curating them.
• A wrong answer that passed the grounding check and was not rated low stays in the knowledge base. Since most customers do not rate, this is the common case, not the edge case.
• Once learned, a wrong answer is a source. Future answers that repeat it will pass the grounding check, because they overlap with the learned text. A mistake can reinforce itself.
• Retraction works per conversation, not per answer. A 1-star rating removes everything learned in that conversation, including answers that were fine, and a 5-star rating from a customer who was pleased about something else protects a wrong answer.

There is one optional extra check. If you enable the LLM reranker (RAG_ENABLE_RERANKER), the learning step also asks the model whether each answer looks supported and skips ones it judges weak. It adds a model call per learned exchange and it is still the model grading itself, so treat it as a filter, not a guarantee.

What this means in practice: make reviewing learned content part of the weekly loop. Learned Q&A pairs are in the Learned category with the auto-learned tag; open them, delete or correct anything wrong, and when you find a wrong one, fix the source document that produced it too. If your domain cannot tolerate an unreviewed answer being reused at all, for example regulated advice or pricing that changes often, tell us at contact before you go live so we can talk through the options.

A weekly routine that fits in an hour

• Pull last week's fixed no-answer replies and group them by topic. Fix the three biggest groups.
• Read every 1 or 2 star conversation. Classify each by cause and fix the source, not the symptom.
• Skim handoffs for answers your team typed more than once.
• Review last week's learned Q&A pairs and remove anything wrong.
• Re-ask the fixed questions and add them to your golden set.

Watch the trend of fixed replies and low ratings over weeks, not days. Content changes take effect immediately, but a week of conversations is a small sample.

Related reading

For how the grounding check and fixed reply work, read preventing hallucinations in customer support. For what to measure and how to read CSAT from AI conversations, see measuring CSAT in AI-driven channels, and for structuring content so retrieval finds it, knowledge base optimization for AI.