Sparse and dense retrieval answer different types of questions, and a single index rarely handles both well. Sparse retrieval (BM25, tsvector) matches exact tokens (error codes, product names, SKUs), while dense retrieval uses neural embeddings to handle paraphrase and semantic queries. Public embedding leaderboards rank models on datasets that look nothing like your help center. The only benchmark that tells you whether dense, sparse, or hybrid retrieval works best for your support content is one you run on your own questions. This post gives the method and none of our results, because numbers measured on someone else's corpus do not transfer to yours.

What you are comparing

Dense retrieval embeds the question and each passage with a neural model and ranks passages by vector similarity. It handles paraphrase ("money back" versus "refund") and fails on rare exact tokens like error codes, SKUs and version numbers.

Sparse lexical retrieval ranks passages by matching terms. BM25 and Postgres full-text search both belong here. It finds "E1042" every time and misses paraphrases entirely.

Learned sparse models (SPLADE and similar) produce term weights with a neural model, so they sit between the two. LayBuild does not include a learned-sparse leg, but the method below works for any of them.

Hybrid retrieval runs more than one of these and merges the results. That is what LayBuild does by default, as described in how LayBuild runs hybrid search.

Build a labelled question set first

The question set matters more than the metric. A few rules:

  • Take questions from real customer conversations, not from the person who wrote the docs. Doc authors phrase questions in the docs' own vocabulary, which flatters lexical search.
  • Keep the customer's wording, typos included.
  • Label each question with where the answer lives. Label at the document level and add a short quoted span of the answer. Chunk IDs change every time you re-chunk; a span lets you count a hit whenever a retrieved passage contains it.
  • Include questions your docs cannot answer, labelled as unanswerable. A good retriever returns nothing above threshold for them.
  • Cover the shapes you actually see: exact tokens (error codes, plan names), paraphrases, multi-part questions, rewritten follow-ups and, if you serve them, non-English questions.
  • Freeze the set and version it. If you tune against the same questions you report on, set aside a held-out portion you only look at after tuning.

A hundred well-labelled questions will show large differences between approaches. Small differences need more questions before you trust them, so treat a one-point gap on a small set as noise.

Metrics to measure

Recall@k is the share of answerable questions where at least one relevant passage appears in the top k. Set k to the number of passages your system actually puts in the prompt. In LayBuild that is 4. Recall@20 is irrelevant if the model only ever sees four.

Mean Reciprocal Rank (MRR) averages 1 / rank of the first relevant passage, counting 0 when there is none in the list. It rewards putting the right passage first, which matters because LayBuild packs passages into a fixed character budget in rank order.

For the unanswerable questions, measure the false-positive rate: how often something scores above your relevance threshold. A retriever that always returns something looks fine on recall and causes wrong answers downstream.

Measure latency on your own infrastructure, at the 50th and 95th percentile, with caches cold. A hosted embedding API adds a network round trip that a local lexical index does not.

Here is a small script that computes recall@k and MRR from a label file and one run file per approach:

ts
import { readFileSync } from 'node:fs';

interface Label {
  questionId: string;
  relevantIds: string[];
}

interface Run {
  questionId: string;
  rankedIds: string[];
}

function loadJson<T>(path: string): T {
  return JSON.parse(readFileSync(path, 'utf8')) as T;
}

function evaluate(labels: Label[], runs: Run[], k: number) {
  const runsById = new Map(runs.map((r) => [r.questionId, r.rankedIds]));
  // Unanswerable questions have no relevant ids; score them separately, not here.
  const answerable = labels.filter((l) => l.relevantIds.length > 0);
  let hits = 0;
  let reciprocalRankSum = 0;
  for (const label of answerable) {
    const ranked = runsById.get(label.questionId) ?? [];
    const relevant = new Set(label.relevantIds);
    const firstRelevant = ranked.findIndex((id) => relevant.has(id));
    if (firstRelevant !== -1 && firstRelevant < k) hits += 1;
    if (firstRelevant !== -1) reciprocalRankSum += 1 / (firstRelevant + 1);
  }
  return {
    questions: answerable.length,
    recallAtK: hits / answerable.length,
    mrr: reciprocalRankSum / answerable.length,
  };
}

const [labelsPath, runPath, kArg] = process.argv.slice(2);
if (!labelsPath || !runPath) {
  throw new Error('usage: bun evaluate.ts labels.json run.json [k]');
}
const result = evaluate(loadJson<Label[]>(labelsPath), loadJson<Run[]>(runPath), Number(kArg ?? 4));
console.log(JSON.stringify(result, null, 2));

If you label with document IDs plus an answer span, map each retrieved passage back to its document (or check the span) before writing rankedIds.

Run the comparison fairly

Most retrieval comparisons are unfair by accident. Check these before you believe a result:

  • Use the same chunks for every approach. Changing the chunker and the retriever at once tells you nothing about either.
  • Re-embed the whole corpus for each dense model. Vectors from different models are not comparable, and they often have different dimensions (bge-small-en-v1.5 produces 384).
  • Apply the same query preprocessing to every arm: lowercasing, follow-up rewriting, stripping greetings.
  • Disable result caches. LayBuild caches retrieval results for 300 seconds by default, which makes a second run look faster and hides changes.
  • Run each leg alone, then fused. Fusion can hide a leg that contributes nothing.
  • Read per-question differences, not only averages. Sort the questions where the approaches disagree and look at why. The patterns there tell you more than the headline number.

Running it against LayBuild

On LayBuild you cannot run the retriever on its own, but the question set is still worth building. Run it through your agent's chat and grade the answers. That tests retrieval and generation together, which is what your customers experience.

What LayBuild defaults to, and why

The defaults are a small English dense model (BAAI/bge-small-en-v1.5), Postgres full-text search with the english configuration, full-text search over Q&A pairs, a knowledge graph leg, and BM25 off. We picked a small embedding model because it is cheap to run for every question and every chunk. Support queries also mix exact tokens and paraphrases, so the lexical leg covers what a small model misses.

The cost of that choice is language. Both the embedding model and the full-text configuration are English. If most of your customers write in other languages, a multilingual embedding model is the first thing to test, and it means re-embedding everything. The pgvector table is created with a fixed vector dimension per partial index, so a model with a different dimension needs a fresh index.

Reading your results

A few patterns come up repeatedly. If lexical search wins on exact-token questions and dense wins on paraphrases, hybrid is the right call and the remaining work is in fusion and thresholds. If dense retrieval misses your product names, check that those names appear in headings, since LayBuild prefixes every embedded chunk with the document title and heading path. If every approach misses the same questions, the answer probably is not in your docs. That is a content gap, and no retriever fixes it.

Frequently asked questions

What is the difference between sparse and dense retrieval?

Sparse retrieval (such as BM25 or full-text inverted indexes) matches exact query tokens and keywords, so it is good at error codes, SKUs, and specific product names. Dense retrieval turns text into vector embeddings and ranks by similarity, so it can match a question to a passage that uses different words for the same thing.

When should you use hybrid retrieval instead of vector-only search?

Use hybrid retrieval when your customers ask both paraphrased questions ("how do I get my money back?") and questions with exact tokens ("Error 403 on API key"), which is common in support. Combining dense embeddings with sparse lexical ranking often improves Recall@k, but measure it on your own questions.

How do you benchmark retrieval models on custom documentation?

Evaluate retrievers using a ground-truth dataset drawn from real user chats rather than synthetic queries. Measure Recall@k (the share of answerable questions with at least one relevant passage in the top k, with k set to the number of passages your system puts in the prompt) and Mean Reciprocal Rank (MRR), testing dense, sparse, and hybrid configurations against identical test sets.

What is dense sparse retrieval and how does it differ from hybrid search?

A dense-sparse system runs a dense embedding leg (cosine similarity via pgvector) and a sparse lexical leg (BM25 or tsvector) and merges the results, which is what hybrid search means. Dense and sparse retrieval on their own are different methods with different failures; hybrid is the combination. Once you run both, the remaining work is in how the ranked lists are fused and where the thresholds sit, so benchmark each leg alone and then fused.

Where to go next

Before changing retrievers, check your chunking with how LayBuild splits your docs, and your content with writing help docs an AI agent can retrieve.