A search for an error code and a search for an explanation are different problems. A user asking about ERR_CONNECTION_RESET provides a precise identifier. Another asking why a connection keeps dropping provides an intent. A useful retrieval system should handle both without requiring users to know how documents were indexed.
The evidence for keeping a lexical baseline
The BEIR paper, first submitted on April 17, 2021, evaluated 10 retrieval systems across 18 datasets. Its authors found BM25 to be a robust baseline; reranking and late-interaction approaches performed well on average but required more computation. These are findings from that benchmark, not a guarantee about every corpus or today's embedding models. Read the BEIR paper.
The practical implication is to measure a lexical baseline before replacing it. Embeddings can help with paraphrases, but exact identifiers, version strings, and rare product names deserve their own evaluation slice.
Combine ranks before tuning score weights
Raw BM25 and vector similarity scores are not naturally interchangeable. Reciprocal rank fusion (RRF) avoids directly adding those scores. Each result receives a contribution based on its position in each ranked list. Elasticsearch documents the contribution as 1 / (rank_constant + rank), with ranks starting at one. RRF reference.
Here is a small, standalone implementation. Inputs are document IDs ordered from best to worst; duplicate IDs within one list contribute once.
from collections import defaultdict def reciprocal_rank_fusion(rankings, rank_constant=60): if rank_constant < 1: raise ValueError("rank_constant must be positive") scores = defaultdict(float) for ranking in rankings: unique_ids = dict.fromkeys(ranking) for rank, document_id in enumerate(unique_ids, start=1): scores[document_id] += 1 / (rank_constant + rank) return sorted(scores, key=lambda doc: (-scores[doc], doc))
The value 60 is an illustrative starting point, not a tuned recommendation for your data. Candidate counts and the fusion constant should be part of the evaluation configuration.
Design a comparison that can fail
Build three runs over the same query set: lexical only, dense only, and fused. Keep document versions, access filters, and relevance labels fixed. Include exact-name queries, paraphrases, short ambiguous queries, and questions with no answer in the corpus.
Inspect disagreements. If dense search misses a version-specific page while BM25 finds it, that is useful evidence. If fusion promotes an outdated duplicate, improve document identity and freshness handling before adding another model.
Only then test a reranker on the fused candidates. Record its effect on ranking quality, latency, and operating cost. A stronger ranking is valuable only if the application can afford the additional work.
The decision to make
Hybrid search is a candidate architecture, not an automatic upgrade. Keep it when measured gains justify its extra index, tuning, and maintenance work. The best result of an evaluation may be a simpler system that already meets the product's needs.