An AI reply that ends a conversation is not the same thing as a solved problem. Most published "AI resolution rates" count the first and report it as the second, which is why they look so good. If you want a number you can defend to your head of support, define resolution by what the customer did after the answer, not by whether a human got involved.

This post gives a definition that holds up, lists the ways the number gets inflated, and walks through what LayBuild actually does with a customer message, so you know which signals exist to measure against.

Four endings that all get counted as resolved

When a conversation handled only by AI ends, it ends in one of roughly four ways:

  • The customer got an answer and confirmed it, by saying so, closing the chat, or rating it well.
  • The customer got an answer and went quiet. Maybe it worked. Maybe they gave up and emailed instead.
  • The customer got the fixed "I do not have specific information about that" reply and left.
  • Nobody said anything for a day and the conversation closed on a timer.

Only the first is clearly a resolution. The second is ambiguous. The third and fourth are failures that look like resolutions if your metric is "conversations closed without a human".

A definition that survives scrutiny

Count a conversation as AI-resolved only when all of these are true:

  • No human agent replied in it.
  • It was never handed off.
  • The last AI reply was a real answer, not a refusal and not the fixed no-answer reply.
  • The same customer did not come back about the same issue within a window you choose and keep fixed (seven days is a common starting point; pick one and don't move it to improve the number).

Then report a second, stricter number next to it: AI-resolved conversations with a CSAT rating of 4 or 5. Report the CSAT response rate too, because a high average from very few ratings says little.

The denominator is every conversation that reached the AI, including ones that were handed off or refused. Dropping the hard ones from the denominator is the most common way these numbers get inflated.

How the number gets inflated

A few patterns show up again and again in resolution reporting, including in vendor marketing.

Counting abandonment as success is the big one. A customer who leaves after an unhelpful answer produces the same "no human needed" record as a satisfied one.

Counting refusals is the next. A reply that says "I don't know, please contact support" ends the AI's part of the conversation. It did not resolve anything.

Making handoff hard to reach is subtler. If customers cannot get to a person, "not handed off" stops meaning anything. In LayBuild, customer-requested handoff is controlled by the ALLOW_HUMAN_HANDOFF organization setting and is off by default. If you leave it off, your handoff rate will be low for reasons that have nothing to do with answer quality. Turn it on before you measure.

Measuring per message instead of per conversation also flatters the result, since a conversation with five good replies and one failure becomes an 83% success.

What happens to one message inside LayBuild

Knowing the pipeline tells you which outcomes are possible and which are logged. Before the pipeline runs, LayBuild counts the AI replies already in the conversation. At 25 it stops, sends a message saying it is connecting the customer with a person, and hands off. Otherwise the message goes through a LangGraph state machine. These are the real node names from the code:

text
                        customer message
                               |
        +-----------+----------+----------+--------------+
        |           |                     |              |
    preflight    memory             cagTemplate        fastQa
  (guardrails, (customer's        (prompt + pinned  (Q&A pair or
   intent,      past facts)        documents)        cached answer)
   query rewrite)
        |           |                     |              |
        +-----------+----------+----------+--------------+
                               |
                         preflightGate
                 /             |               \
          blocked:        fastQa hit:        otherwise
       fixed refusal    skip to guardrail        |
                                             retrieve
                                                 |
                                             generate
                                  (no docs: fixed reply, no LLM call;
                                   after LLM: grounding check)
                                                 |
                                            reflection
                                        (optional, off by default)
                                                 |
                                             toolCall
                                                 |
                                             guardrail
                                                 |
                                              handoff

preflight checks the admin's blocked-terms list, about 25 prompt-injection patterns and off-topic patterns (maths, trivia, poems, translation). A match ends the run with a fixed refusal. It also detects a request for a human and, for follow-ups like "what about the second one", asks the model to rewrite the question into a standalone search query. memory loads stored facts about this customer, cagTemplate loads the agent's prompt and pinned documents, and fastQa looks for a stored Q&A pair or a recently cached answer. These four run in parallel.

preflightGate routes: a blocked message ends the run; a Q&A match skips generation entirely and goes to the output guardrail; everything else goes to retrieve, which runs hybrid search and drops results below the relevance threshold.

generate is where strict knowledge-base mode lives. If retrieval found nothing and there is no memory or pinned content, the model is never called and the customer gets the fixed reply: "I do not have specific information about that in the knowledge base. Please contact {company} for further assistance, or ask about another topic." If the model does answer, the answer is checked for word overlap with the sources; below 25% overlap, it is replaced with the same fixed reply.

reflection is an optional second model call that labels the answer supported or unsupported. It is off unless the operator enables it. toolCall runs any API tools attached to the agent with their configured defaults and attaches the first successful result. guardrail redacts card numbers, US Social Security numbers, emails and phone numbers from the answer and enforces the length cap. handoff decides whether a requested handoff is allowed by the organization setting.

The signals you can measure against

Every outcome in the definition above maps to something LayBuild records.

  • Conversation status is one of OPEN, HANDOFF or CLOSED.
  • The CONVERSATION_HANDOFF webhook carries a reason: ai_unable when a customer's request was accepted, ai_turn_limit_reached at 25 AI replies, and agent_claim when a person picks the conversation up.
  • The CONVERSATION_CLOSED webhook says who closed it (customer, agent or system), and conversations closed by the 24-hour inactivity timer carry the reason inactivity_24h.
  • Every AI reply is sent as a MESSAGE_RECEIVED webhook with role ASSISTANT, so you can detect the fixed no-answer reply by its opening words.
  • CSAT is a 1 to 5 rating per conversation, and there is an audit log.

The event-driven workflows post shows how to collect these into your own store.

A worked example with your own numbers

This is an illustration, not data from any customer. Suppose you had 1,000 conversations reach the AI in a month and, after tagging them, found this: 150 were handed off, 100 ended on the fixed no-answer reply, 300 went quiet and closed on the inactivity timer, and 450 ended with the customer closing the chat or rating it. Of those 450, 50 customers came back about the same issue within seven days.

The generous metric, "not handed off", gives 850 out of 1,000, or 85%. The definition above gives 450 minus 50, so 400 out of 1,000, or 40%. The 300 quiet conversations are genuinely unknown; you can report them as a separate bucket, but you should not add them to the resolved column. The gap between 85% and 40% is the gap between what gets put on a slide and what your support team experiences.

Limits of the checks

The grounding check is lexical. An answer that reuses the source's vocabulary but gets a number wrong will pass, and a correct answer that paraphrases heavily can be replaced with the fixed reply. So the "last reply was a real answer" test tells you the model answered, not that the answer was right. Read a sample of resolved transcripts every week. A few dozen conversations read carefully will tell you more than the rate.

Two more caveats. LayBuild's learning loop publishes some answered exchanges back into the knowledge base without human review, and removes them if the conversation is rated 2 stars or lower. An unrated wrong answer can therefore become a source for later answers, which is another reason to read transcripts. And retrieval is English-first, so expect lower resolution on non-English conversations.

Next steps