How to benchmark dense vs sparse retrieval on your own docs

Public embedding leaderboards rank models on datasets that look nothing like your help center. The only benchmark that tells you whether dense, sparse or hybrid retrieval works best for your support content is one you run on your own questions. This post is the method. It contains no results from us, because numbers measured on someone else's corpus do not transfer to yours.

What you are comparing

Dense retrieval embeds the question and each passage with a neural model and ranks passages by vector similarity. It handles paraphrase ("money back" versus "refund") and fails on rare exact tokens like error codes, SKUs and version numbers.

Sparse lexical retrieval ranks passages by matching terms. BM25 and Postgres full-text search both belong here. It finds "E1042" every time and misses paraphrases entirely.

Learned sparse models (SPLADE and similar) produce term weights with a neural model, so they sit between the two. LayBuild does not include a learned-sparse leg, but the method below works for any of them.

Hybrid retrieval runs more than one of these and merges the results. That is what LayBuild does by default, as described in how LayBuild runs hybrid search.

Build a labelled question set first

The question set matters more than the metric. A few rules:

• Take questions from real customer conversations, not from the person who wrote the docs. Doc authors phrase questions in the docs' own vocabulary, which flatters lexical search.
• Keep the customer's wording, typos included.
• Label each question with where the answer lives. Label at the document level and add a short quoted span of the answer. Chunk IDs change every time you re-chunk; a span lets you count a hit whenever a retrieved passage contains it.
• Include questions your docs cannot answer, labelled as unanswerable. A good retriever returns nothing above threshold for them.
• Cover the shapes you actually see: exact tokens (error codes, plan names), paraphrases, multi-part questions, rewritten follow-ups and, if you serve them, non-English questions.
• Freeze the set and version it. If you tune against the same questions you report on, set aside a held-out portion you only look at after tuning.

A hundred well-labelled questions will show large differences between approaches. Small differences need more questions before you trust them, so treat a one-point gap on a small set as noise.

Metrics that answer the real question

Recall@k is the share of answerable questions where at least one relevant passage appears in the top k. Set k to the number of passages your system actually puts in the prompt. In LayBuild that is 4. Recall@20 is irrelevant if the model only ever sees four.

Mean Reciprocal Rank (MRR) averages 1 / rank of the first relevant passage, counting 0 when there is none in the list. It rewards putting the right passage first, which matters because LayBuild packs passages into a fixed character budget in rank order.

For the unanswerable questions, measure the false-positive rate: how often something scores above your relevance threshold. A retriever that always returns something looks fine on recall and causes wrong answers downstream.

Measure latency on your own infrastructure, at the 50th and 95th percentile, with caches cold. A hosted embedding API adds a network round trip that a local lexical index does not.

Here is a small script that computes recall@k and MRR from a label file and one run file per approach:

import { readFileSync } from 'node:fs';

interface Label {
  questionId: string;
  relevantIds: string[];
}

interface Run {
  questionId: string;
  rankedIds: string[];
}

function loadJson<T>(path: string): T {
  return JSON.parse(readFileSync(path, 'utf8')) as T;
}

function evaluate(labels: Label[], runs: Run[], k: number) {
  const runsById = new Map(runs.map((r) => [r.questionId, r.rankedIds]));
  // Unanswerable questions have no relevant ids; score them separately, not here.
  const answerable = labels.filter((l) => l.relevantIds.length > 0);
  let hits = 0;
  let reciprocalRankSum = 0;
  for (const label of answerable) {
    const ranked = runsById.get(label.questionId) ?? [];
    const relevant = new Set(label.relevantIds);
    const firstRelevant = ranked.findIndex((id) => relevant.has(id));
    if (firstRelevant !== -1 && firstRelevant < k) hits += 1;
    if (firstRelevant !== -1) reciprocalRankSum += 1 / (firstRelevant + 1);
  }
  return {
    questions: answerable.length,
    recallAtK: hits / answerable.length,
    mrr: reciprocalRankSum / answerable.length,
  };
}

const [labelsPath, runPath, kArg] = process.argv.slice(2);
if (!labelsPath || !runPath) {
  throw new Error('usage: bun evaluate.ts labels.json run.json [k]');
}
const result = evaluate(loadJson<Label[]>(labelsPath), loadJson<Run[]>(runPath), Number(kArg ?? 4));
console.log(JSON.stringify(result, null, 2));

If you label with document IDs plus an answer span, map each retrieved passage back to its document (or check the span) before writing rankedIds.

Run the comparison fairly

Most retrieval comparisons are unfair by accident. Check these before you believe a result:

• Use the same chunks for every approach. Changing the chunker and the retriever at once tells you nothing about either.
• Re-embed the whole corpus for each dense model. Vectors from different models are not comparable, and they often have different dimensions (bge-small-en-v1.5 produces 384).
• Apply the same query preprocessing to every arm: lowercasing, follow-up rewriting, stripping greetings.
• Disable result caches. LayBuild caches retrieval results for 300 seconds by default, which makes a second run look faster and hides changes.
• Run each leg alone, then fused. Fusion can hide a leg that contributes nothing.
• Read per-question differences, not only averages. Sort the questions where the approaches disagree and look at why. The patterns there tell you more than the headline number.

Running it against LayBuild

If you self-host, the API package includes an evaluation script. From apps/api, run bun run rag:eval --dataset=your-questions.json. The dataset is a JSON array of objects with id, question, and optionally groundTruthAnswer and expectedDocId. The script retrieves the top 4 passages per question (change it with --topK=) and writes rag-eval-report.md with context precision, context recall and lexical groundedness per question. Set RAG_ENABLE_EVAL=true to add LLM-judged faithfulness and answer relevance, at the cost of extra model calls.

Two caveats about that script. It compares expectedDocId to retrieved passage IDs exactly, so label with the passage IDs shown in the report or adapt the matching. And it runs the retriever without an organization or agent scope, so it evaluates knowledge stored without an organization. To evaluate one tenant's knowledge, adapt the script to pass that tenant's IDs. You can compare arms by toggling NLP_ENABLE_BM25, running with and without Qdrant configured (no Qdrant means no vector leg), or changing the embedding model setting (HF_EMBED_MODEL for the Hugging Face provider) and re-indexing with bun run rag:backfill.

On a hosted plan you cannot swap the embedding model, but the question set is still worth building. Run it through your agent's chat and grade the answers. That tests retrieval and generation together, which is what your customers experience.

What LayBuild defaults to, and why

The defaults are a small English dense model (BAAI/bge-small-en-v1.5), Postgres full-text search with the english configuration, full-text search over Q&A pairs, a knowledge graph leg, and BM25 off. We picked a small embedding model because it is cheap to run for every question and every chunk. Support queries also mix exact tokens and paraphrases, so the lexical leg covers what a small model misses.

The cost of that choice is language. Both the embedding model and the full-text configuration are English. If most of your customers write in other languages, a multilingual embedding model is the first thing to test, and it means re-embedding everything. The Qdrant collection is created with a fixed vector dimension, so a model with a different dimension needs a fresh collection.

Reading your results

A few patterns come up repeatedly. If lexical search wins on exact-token questions and dense wins on paraphrases, hybrid is the right call and the remaining work is in fusion and thresholds. If dense retrieval misses your product names, check that those names appear in headings, since LayBuild prefixes every embedded chunk with the document title and heading path. If every approach misses the same questions, the answer probably is not in your docs. That is a content gap, and no retriever fixes it.

Where to go next

Before changing retrievers, check your chunking with how LayBuild splits your docs, and your content with writing help docs an AI agent can retrieve. If you want to run the eval script, the source is in the LayBuild repository.