An earlier version of this post described a Redis sliding window for chat history. LayBuild doesn't work that way. Redis holds caches, rate limits and locks, and chat history lives in Postgres. The real design has three parts: a short slice of the current conversation, a small set of long-term facts about each customer, and excerpts from their recent past conversations. Each part has a fixed budget, and each budget means something gets forgotten. This post covers what is kept, what is dropped, and why.

Three kinds of memory, three budgets

text
current conversation (Postgres messages)
  last 12 loaded -> last 6 kept, at most 3,000 characters and 800 tokens

customer facts (per customer and agent)
  20 most recent loaded -> ranked by similarity to the question -> top 8

past conversations (same customer, same agent)
  last 4 messages from each of the 3 most recent, 800 characters each

facts + past excerpts -> one "past interactions" block, at most 2,000 characters

All of this goes into the prompt next to the retrieved knowledge passages. Retrieved passages and pinned documents share a separate 4,000-character budget.

Short-term memory: the last six messages

For each new customer message, LayBuild loads the last 12 messages of the conversation from Postgres and keeps the most recent 6 for the prompt. It then trims to fit 3,000 characters and 800 tokens, counted with the cl100k_base tokenizer (or estimated at 4 characters per token if the tokenizer fails to load). Consecutive assistant messages are collapsed so only the latest one is kept.

Six messages is about three exchanges. We keep it small on purpose. The prompt has to hold the system instructions, up to four retrieved passages, pinned knowledge and customer memory, and we would rather spend the budget on your content than on old chat turns. A long history also gives the model more earlier text to repeat or contradict, and the default model is a small one (Llama 3.1 8B Instruct).

The cost is real. In a long conversation, anything said more than six messages ago is gone from the prompt unless it was saved as a fact. If a customer gave their order number at the start of a twenty-message conversation, the model can no longer see it by the end. Human agents reading the dashboard see the full history; the model does not.

Follow-up questions are rewritten before search

Short history causes a second problem: "what about the second one?" is useless as a search query. When a message contains pronouns or references ("it", "that", "the second one", "what about") or is under three words long, and the conversation has at least two earlier messages, LayBuild asks the model to rewrite it as a standalone search query. The rewrite uses the last 4 messages (each cut to 250 characters), runs at temperature 0, and is limited to 64 tokens. The instructions tell it to keep product names, plan names and error codes exactly as written and not to add facts. The rewritten query is used for retrieval; the model still answers the customer's original message.

This costs one extra model call on those turns. A bad rewrite can send retrieval in the wrong direction, and the customer never sees it. If you self-host with tracing enabled, the preflight span records whether a query was rewritten.

Long-term facts about a customer

After each AI reply, LayBuild extracts durable facts from the customer's side of the exchange: name, company, role, plan, product used, order or ticket reference, preferences, recurring problems. Extraction uses a model call with instructions to return JSON and to include only things the customer stated about themselves, plus a set of pattern matches. The instructions tell the model never to extract passwords, card numbers, CVVs, one-time codes or government ID numbers. That is an instruction to a model, not a filter, so treat it as a best effort.

Facts are stored per customer and per agent. Exact duplicates are skipped. When a new question arrives, LayBuild loads the customer's 20 most recent facts, ranks them by embedding similarity to the question, removes duplicates and keeps the top 8.

The trade-offs:

  • There is no contradiction handling. If a customer said they were on Pro last month and on Starter today, both facts are stored, and ranking by similarity rather than recency means the older one can win.
  • Facts have no expiry. They drop out of consideration only when 20 newer facts push them out, even if they are still the most relevant ones, and they stay stored after that.

Excerpts from past conversations

LayBuild also takes the last 4 messages from each of the customer's 3 most recent other conversations with the same agent, capped at 800 characters per conversation. The prompt labels them as past conversations and tells the model not to repeat questions that were already answered. Facts and excerpts together are cut to 2,000 characters.

This only works when LayBuild can tie the new conversation to the same customer record. A visitor who can't be linked to an earlier conversation starts with no long-term memory.

Memory also counts as grounding

In strict knowledge-base mode, memory is part of what the model may answer from. That lets the bot answer "which plan am I on?" from a stored fact. It also has two side effects worth knowing about.

First, the no-retrieval short circuit (skip the model when there is nothing to ground on) doesn't fire for a customer with memory. The model runs and the overlap check decides.

Second, the 25% stemmed-overlap check counts memory as a source. A customer's own earlier claim, such as "your colleague promised me a refund", sits in the grounding text, and an answer that repeats it can pass. How LayBuild limits made-up answers covers the check in detail.

What we chose not to build

We don't summarize long conversations into a running summary. Summaries cost a model call per turn, and errors in a summary carry into every later turn with no way for the customer to see or correct them. We don't do vector search over every past message either. The fact store covers the durable things customers repeat, like plan and company, at much lower cost. And the conversation itself hands off: after 25 AI turns, LayBuild can pass the conversation to a human, which is usually the right move for conversations long enough to outgrow six messages of memory.

Practical advice

Design for short conversations. If your flows need customers to give an order number and then answer several follow-ups, ask for the identifier close to where it is used, or make sure it is stated in a way the fact extractor picks up ("my order number is ..."). Watch conversations that run past a dozen messages, since the model is working with a partial view by then.

Where to go next

For when a conversation should move to a person, read safe human handoff. For how the knowledge side of the prompt is built, read how LayBuild runs hybrid search.