A CSAT score from an AI support channel tells you how the customers who chose to rate felt about whole conversations. It does not tell you how good the AI's answers were, and on its own it can move in the wrong direction when the AI gets worse. Read it next to two other numbers, how often the AI said it had no answer and how often chats went to a person, and it becomes useful. This post explains how CSAT works in LayBuild, where post-chat ratings mislead, and what to look at instead of a single average.
How CSAT works in LayBuild
Each conversation can carry one rating from 1 to 5, with an optional comment. The web widget shows the rating card when a conversation is closed or when an agent asks for a rating from the dashboard. A conversation gets closed in one of three ways: an agent closes it, it is closed through the API, or it has had no activity for 24 hours and the auto-close job closes it.
The analytics dashboard shows four things for the period you pick: the number of ratings, the average rating, the share of ratings that are 4 or 5, and the distribution across the five values. Admins and owners also get an email for each rating, which is useful early on and noisy later.
Ratings do one more job. LayBuild's learning loop publishes some AI answers back into the knowledge base automatically, without human review. If a customer rates a conversation 2 or lower, the knowledge learned from that conversation is retracted. So a low score is not only a number on a chart; it also removes content that might otherwise repeat the mistake.
Why post-chat CSAT is biased
None of these problems are specific to AI, but AI channels make several of them worse.
Only some customers rate
Response bias is the big one. Customers who rate are not a random sample of customers who chatted. People who are delighted and people who are annoyed are more likely to click a star than people who got an ordinary answer and left. Suppose one chat in twenty gets a rating (an illustration, not a benchmark): your CSAT then describes that one in twenty, and the score alone cannot tell you how the other nineteen went.
The rating card appears at the end, and many AI chats have no clear end
A customer asks a question, gets an answer and closes the tab. That conversation stays open until the 24-hour auto-close, and by then the customer is gone. Conversations that a human agent handles are more likely to be closed while the customer is still there, and agents can ask for a rating directly. The likely result is that human-handled conversations are over-represented in your ratings. Check this in your own data before you trust a blended average.
One score covers two very different experiences
A conversation that started with the AI and finished with a person gets one rating. If the customer gives it a 2, was that the AI failing to understand them, the wait for a human, or the human's answer? The score cannot say. The reverse also happens: a patient human rescues a bad AI conversation and the customer rates the rescue, not the AI.
The metric rewards the wrong behavior
An AI that confidently answers everything, including questions it should not, can produce happier ratings in the short term than an AI that says "I do not have specific information about that" and offers a person. The honest refusal feels worse in the moment. If you optimize purely for CSAT, you push the system towards guessing, which is the opposite of what a strict knowledge-base setup is for.
The two numbers to read next to CSAT
No-answer rate
When LayBuild's retrieval finds nothing relevant, or the generated answer does not overlap enough with the retrieved sources, the customer gets a fixed reply: "I do not have specific information about that in the knowledge base. Please contact {company} for further assistance, or ask about another topic." How often that reply appears is the most direct measure of gaps in your content. A rising no-answer rate with a flat CSAT means customers are hitting walls but not rating them.
The dashboard does not chart this today. You can count it yourself, because the reply always starts with the same sentence.
Handoff rate
The dashboard's handoff rate is the share of conversations in the period whose current status is HANDOFF. Two details matter. First, "current": once a handed-off conversation is closed, it counts as closed, so the handoff rate is a snapshot of what is waiting or in progress, not a count of every conversation that ever went to a person. Second, the dashboard's resolution rate is the share of conversations that are closed, which includes auto-closed chats where the customer simply left. Neither number, read alone, tells you whether a customer's problem was solved.
For a count of every handoff, use the audit log (each handoff writes a conversation.handoff_requested entry with a reason) or subscribe a webhook to CONVERSATION_HANDOFF and count the events on your side.
Reading the three together
The combinations are more informative than any single number.
| CSAT | No-answer rate | Handoff rate | Likely reading |
|---|---|---|---|
| Flat | Rising | Flat | Customers hit gaps and leave without rating or asking for a person. Fix content. |
| Falling | Flat | Rising | Customers are reaching people but the handoff experience is poor: waits, repetition, or nobody on shift. |
| Rising | Falling | Falling | Content fixes are working. Check a sample of transcripts to confirm answers are correct, not just confident. |
| Rising | Flat | Rising | More chats reach people and people are rating well. Likely human-handled chats dominating the ratings. |
These are starting hypotheses, not conclusions. Each one should end with you reading transcripts.
Splitting CSAT by who handled the conversation
The dashboard shows one blended average. If you self-host LayBuild, you can split it by whether a human agent replied in the conversation, straight from Postgres:
SELECT
CASE WHEN EXISTS (
SELECT 1 FROM "Message" m
WHERE m."conversationId" = c.id AND m.role = 'AGENT'
) THEN 'human replied' ELSE 'AI only' END AS handled_by,
COUNT(*) AS ratings,
ROUND(AVG(c.rating)::numeric, 2) AS avg_rating,
ROUND(100.0 * AVG(CASE WHEN c.rating >= 4 THEN 1 ELSE 0 END), 1) AS pct_4_or_5
FROM "Conversation" c
WHERE c."orgId" = $1
AND c.rating IS NOT NULL
AND c."createdAt" >= now() - interval '30 days'
GROUP BY 1;And the no-answer rate for the same period:
SELECT
COUNT(DISTINCT c.id) AS conversations,
COUNT(DISTINCT m."conversationId") AS with_no_answer_reply
FROM "Conversation" c
LEFT JOIN "Message" m
ON m."conversationId" = c.id
AND m.role = 'ASSISTANT'
AND m.content LIKE 'I do not have specific information about that%'
WHERE c."orgId" = $1
AND c."createdAt" >= now() - interval '30 days';On the hosted service, the MESSAGE_RECEIVED webhook fires for AI replies as well as customer messages and includes the message role and content, so you can build the same counts in your own store.
Run the first query again without the c.rating IS NOT NULL line to get the number of conversations in each group, then compare it with the number of ratings. If the AI-only group has a much lower rating rate, the blended CSAT is mostly a score for your human team.
A weekly routine that holds up
- First, note the rating count as well as the average. A CSAT built on a handful of ratings will swing on one angry customer.
- Next, read every 1 and 2 rating with its comment and transcript. Decide whether the cause was a content gap, a wrong answer, a slow handoff or something outside support (a shipping delay, a pricing change).
- Then check the no-answer rate and pull the questions behind it. The ranked questions list on the Knowledge Q&A page shows which customer questions come up most often; cross it with the no-answer replies to find the gaps worth fixing first.
- Finally, sample a few 4 and 5 ratings from AI-only conversations and check the answers were actually right. A confident wrong answer about a return window can earn a 5 today and a chargeback next month.
We do not publish a target CSAT, and we would be suspicious of anyone who quotes one without knowing your rating rate and how your conversations end.
Related reading
- Deflection vs resolution explains why "did not reach a human" is not the same as "problem solved".
- How LayBuild hands a chat to a human covers what triggers a handoff and what the agent sees.
- Preventing hallucinations in customer support describes the grounding check behind the fixed no-answer reply.