Hybrid Search + Reranking: the Production Default for RAG

In the first post, I introduced retrieval-augmented generation: retrieve first, then generate. That works well until retrieval itself becomes the bottleneck. In production, the most common failure is not the language model. It is the retriever returning the wrong passages, or missing the right one entirely. This post is about the retrieval pipeline most production RAG systems actually run: keyword search and vector search side by side, fused with Reciprocal Rank Fusion, then reranked with a cross-encoder.

The idea is straightforward. No single retrieval method is good at everything. Keyword search is precise but brittle. Vector search is broad but misses exact identifiers. Running both, and letting a reranker order the survivors, covers each method’s weakness. The rest of this post walks through each stage in order, what it costs, and when it does not help.

The problem: two kinds of queries

Every RAG corpus gets two kinds of questions, and each kind breaks a different retriever.

The first kind is conceptual. “How do I reduce customer churn?” The answer lives in documents about retention strategies and preventing subscriber loss. Those documents share almost no words with the question. Vector search handles this well, because it matches meaning rather than words.

The second kind is exact. “What does error code E-4217 mean?” The answer is the one document that contains that exact string. Vector search handles this badly. An embedding places “E-4217” in a general “error handling” region of the vector space, and returns documents about error handling in general. Rare tokens — ticket IDs, product SKUs, version strings, RFC numbers, clause references like “Section 4(b)(iii)” — collapse into near-identical embeddings. The answer is sitting in the corpus, and pure vector search walks past it.

Keyword search has the opposite profile. It finds the exact token you typed, every time. But it cannot connect “contract termination procedures” to “end of agreement protocols”. Zero shared terms means zero relevance. It cannot handle typos, abbreviations, or paraphrases.

So the two methods fail in opposite directions. That is the whole argument for running both: their failure modes are complementary, and each one catches what the other misses.

Anthropic measured this directly in their contextual retrieval work. Embeddings, they wrote, excel at semantic relationships but can miss crucial exact matches, and BM25 is the older technique that catches them. Their example was “Error code TS-999” in a support database: the embedding finds error-code content in general, while BM25 finds the exact string. On their own test sets, adding contextual BM25 cut the top-20 retrieval failure rate from 5.7% to 2.9% — a 49% reduction. Adding a reranker on top took it to 1.9%, a 67% reduction. Those are Anthropic’s figures on their own data, so treat the exact percentages as theirs, not as universal. The pattern, though, is consistent across every source I checked: keyword search first, reranker second, each stage fixing a different failure.

BM25: the boring half

BM25 is a 30-year-old ranking function. It scores documents by term frequency, inverse document frequency, and document length normalisation. There is no model to host, no GPU to provision, and no drift to monitor. On millions of documents it answers in single-digit milliseconds on a single core.

That simplicity is the point. BM25 does one job — exact matching — and it does it cheaply and predictably. It is not a replacement for vector search. It is the half of the pipeline that catches the queries vector search drops.

Dense search: the broad half

Dense search does the job BM25 cannot. It encodes the query and every document into vectors, and ranks by similarity. It bridges vocabulary gaps, handles paraphrases, and finds conceptually related material that shares no words with the question.

Its weakness is the one described above: exactness. It is also the more expensive half to run and to keep current, since every document change means re-embedding. But for conceptual queries it is the only method that works at all, which is why it became the default — and why, on its own, it is not enough.

Putting them together: RRF fusion

Running two retrievers gives you two ranked lists. The problem is that their scores live on different scales. BM25 scores are unbounded and depend on corpus statistics. Cosine similarity lives between -1 and 1, with most relevant documents clustering above 0.7. A weighted sum of the two scores looks natural, but it is fragile in practice: min-max normalisation is sensitive to outliers, and z-score normalisation changes shape with every query.

Reciprocal Rank Fusion avoids the problem by ignoring scores entirely and working on ranks only. The formula, from Cormack, Clarke, and Buettcher’s 2009 SIGIR paper, is:

RRF_score(d) = sum over each list of 1 / (k + rank(d))

A document ranked 1st in one list and 3rd in another scores 1/61 + 1/63. A document ranked 40th in both scores 1/100 + 1/100. Documents near the top of multiple lists win. The constant k = 60 comes from the original paper, and it has held up well enough that most implementations never touch it. Lower k makes top ranks dominate more; higher k flattens the distribution.

The one thing worth tuning is not k, but the weight given to each retriever. Most vector databases let you weight each branch. A reasonable starting point:

  • Identifier-heavy queries (catalogs, error codes, ticket IDs): BM25 0.7, dense 0.3.
  • Conceptual or paraphrase-heavy queries: BM25 0.3, dense 0.7.
  • Mixed corpora: 0.5 / 0.5, and let the reranker sort it out.

Do not tune k until you have a labelled eval set of at least 200 queries. Anything less and you are chasing noise. And do not tune anything at all until you have measured the 50/50 default on your own questions first.

How much does fusion actually gain? Adaptive Recall, in a July 2026 guide to production RAG, puts the hybrid gain at 5% to 15% higher recall@10 and NDCG@10 than the stronger of the two methods alone, with the biggest wins on queries where one method alone would fail. A practitioner guide (AI Workflow Lab) reports a starker split on enterprise corpora — recall@10 around 91% for hybrid, 78% for dense-only, 65% for BM25-only — but that is a practitioner blog, not a peer-reviewed result, so treat it as indicative rather than exact. The direction is the same in both: hybrid beats either method alone, and the gap is largest exactly where one method fails outright.

Reranking: precision is a separate job

Fusion fixes recall. It gets the right document into the candidate set. Reranking fixes precision. It puts the right document at the top of the list. These are complementary stages, not alternatives, and confusing them is the most common mistake I see.

A cross-encoder reranker reads the query and each candidate passage together, through every layer of the model, and scores true relevance. A bi-encoder — your embedding model — encodes the query and the document separately and compares the two vectors. The cross-encoder is far more accurate and far slower per pair. That is why you never run it over the whole corpus. You run it over the top 20 to 50 fused candidates and keep the top 3 to 5.

The pipeline, in order:

  1. Retrieve wide. Dense search and BM25 in parallel, top 50 from each. Retrieve at least 20 from each — because the two lists overlap, the merged set is smaller than the sum.
  2. Fuse. RRF with k = 60.
  3. Rerank. Pass the fused top 20–50 to a cross-encoder. Keep the top 3–5.
  4. Generate. Only those chunks go to the model.

The reported lift from reranking is roughly +5 to +15 NDCG@10 over first-stage retrieval alone. That figure comes from a practitioner synthesis of BEIR and MTEB-style benchmarks (The Agent Ecosystem), not a single peer-reviewed study, so read it as a range rather than a promise. Anthropic’s numbers are the cleanest measured case: as noted above, adding a reranker to their hybrid setup cut the failure rate from 2.9% to 1.9%.

One more point from the same sources, and it matters: a reranker cannot recover a document that was never retrieved. It can only re-order what retrieval surfaced. If your labelled set shows the correct chunk missing from the top 50, the problem is recall, not ranking. Fix chunking, metadata, or the retrieval stage first. And the reverse is worth saying too: if latency rules out a reranker, hybrid search without reranking is still better than dense-only with reranking in most evaluations. Recall is the more fundamental fix.

What it costs

The cost is real but small, and it is mostly latency, not money.

Running two retrievers in parallel adds roughly 6 ms to p50 latency over dense-only search (AI Workflow Lab, practitioner measurement). BM25 itself is single-digit milliseconds. Reranking 50 candidates adds roughly 50–200 ms, depending on the model — self-hosted on a GPU is often 50–100 ms, a hosted API 100–400 ms for the same batch. Total retrieval lands around 60–220 ms. The LLM generation step takes 500–3,000 ms, so the whole retrieval cascade is essentially free relative to generation — unless your latency budget is a typeahead search box or a voice pipeline, where 150–300 ms is unacceptable.

Money is a rounding error next to the LLM bill, and reranking can even cut total cost: sending three clean chunks instead of ten noisy ones lowers the completion tokens you pay for.

Which reranker? The practical choice in 2026:

  • Cohere Rerank 3.5 / Rerank 4 — managed API, the fastest path, no GPU to operate. Best under roughly 50,000 queries a day.
  • Voyage Rerank 2.5 — managed API, cheaper, the natural pick if you already use Voyage embeddings.
  • BGE reranker v2-m3 — open weights, Apache 2.0, multilingual, the default for self-hosted enterprise deployments. A single GPU handles it; at high volume the unit economics beat the API.
  • FlashRank (MiniLM-L-12) — quantized cross-encoder, sub-100 ms on CPU, the cheap default when you cannot send data to a third-party API.
  • LLM-as-reranker — prompt a large model to score candidates. Highest accuracy on subtle relevance judgements, but 1–3 seconds per batch. Use it only for async pipelines or high-value queries.

Pricing quoted in 2026 guides — Cohere around $2 per 1,000 searches, Voyage around $0.05 per million tokens — should be checked against the vendor pricing pages before you rely on them. Cohere moved self-serve pricing during 2026.

The spread between the top rerankers is small. One practitioner benchmark table (AI Workflow Lab) puts them close together on quality, with nDCG@10 from 0.66 (FlashRank, CPU) to 0.74 (Cohere). The bigger truth from the same source: the retriever sets the ceiling. No reranker pushes the hit rate above roughly 88% if the retrieval stage missed the document. Fix recall before you shop for rerankers.

When hybrid search does not help

Hybrid is the default, not a law. Three cases where it adds little or nothing:

Tiny corpora. Under roughly 2,000 chunks, vector search alone usually retrieves the right material, and a reranker adds latency without moving accuracy. If the whole relevant document set fits in the prompt, retrieval quality barely matters — send more chunks instead.

Narrow, consistent vocabulary. In highly technical corpora where everyone uses the same terms, or conversational data where BM25 adds noise, dense-only can perform equally well. Adaptive Recall says this plainly: hybrid wins on average across benchmarks, but there are domains where it does not. Measure on your data before committing to the added complexity.

Broken stage one. If the correct chunk is missing from the top 50, no reranker will save you. Fix the retrieval stage first, as described above.

The recipe

If you are starting from a dense-only pipeline today, this is the change, in order:

  1. Add BM25 alongside your vector index. Most vector databases now support it natively — Elasticsearch and OpenSearch have it built in, Qdrant since version 1.7, Weaviate with a configurable alpha, Pinecone with sparse-dense vectors, PostgreSQL with pgvector plus full-text search. Pick whichever one your stack already has; do not introduce a new database for this.
  2. Retrieve 50 from each, fuse with RRF at k = 60, weights 50/50. Keep the top 20.
  3. Add a cross-encoder reranker. Start with a managed API (Cohere or Voyage) or FlashRank if you need CPU-only. Rerank the top 20–50, keep the top 3–5.
  4. Measure on your own questions. Build a small eval set — 30 to 50 real questions, with the correct source passage noted for each — and check two numbers before and after: did the right chunk get retrieved at all (recall), and did it rank near the top (precision). Change one thing at a time.

Three mistakes to avoid, all from production write-ups: use the same tokenization for ingestion and query-time BM25, or the term statistics will not match; strip metadata from chunks before BM25 indexing, or BM25 will match on source markers and section headers instead of content; and do not set the per-method top-k too low — retrieve at least 20 from each side, or the merged set is thinner than you think.

The short version

  • Vector search misses exact terms. BM25 misses meaning. Run both.
  • Fuse with Reciprocal Rank Fusion, k = 60. It works on ranks, so you never normalise two incompatible scoring systems.
  • Rerank the fused top 20–50 with a cross-encoder. Keep the top 3–5 for the model.
  • Hybrid fixes recall, reranking fixes precision. A reranker cannot save a document retrieval never found.
  • The whole cascade costs roughly 60–220 ms against a 500–3,000 ms generation step. Cheap — unless you are building a typeahead.
  • Measure on your own questions. Keep only what the numbers justify.

Next up: GraphRAG and LazyGraphRAG — what happens when the answer is not in one chunk, but in the relationship between chunks.

References

Comments

Leave a Reply

Create your own website, free on WordPress.com.Create your own website.
Start free