Hybrid Search for Better RAG Retrieval

The Problem#

Pure vector search is good at semantic similarity, but it can miss exact terms: error codes, function names, ticket ids, product SKUs, and policy section numbers. Pure keyword search catches exact terms but misses paraphrases. RAG systems need both.

Why It Matters#

The model can answer only from the context it receives. Better retrieval usually improves RAG quality more than prompt changes because it gives the model the right evidence before generation starts.

Core Concepts#

Hybrid search combines lexical retrieval, vector retrieval, metadata filtering, and reranking. Lexical search finds exact matches. Vector search finds semantic matches. Metadata filters enforce tenant, product, language, version, or time range. A reranker sorts the merged candidate list using a more expensive relevance model.

Implementation#

Use a staged retrieval pipeline:

query
  -> rewrite optional acronyms
  -> run BM25 search
  -> run vector search
  -> merge and deduplicate candidates
  -> apply authorization and freshness filters
  -> rerank top 50
  -> send top 5 to the model

Keep each stage observable. Log which retriever found each chunk and why it was kept or dropped.

Example#

Imagine an internal support assistant. A user asks:

Why did checkout fail with PAY-4027 after the August billing rollout?

Keyword search should match PAY-4027, "August billing rollout", and exact runbook titles. Vector search should also retrieve documents that say "payment authorization rejected after pricing service release", even if they never mention "checkout failed." After merging results, the system might keep these candidates:

1. incident-2026-08-billing-rollout.md      found by BM25 and vector search
2. payment-error-codes.md                   found by BM25
3. pricing-service-deployment-runbook.md    found by vector search

The reranker should move the incident document above the generic error-code page because it matches the user's time window and symptom. The final answer can then cite the incident and explain the fix instead of giving a generic payment-error definition.

Common Mistakes#

• Using only vector similarity for identifier-heavy domains.
• Filtering after generation instead of before retrieval.
• Sending too many chunks and hoping the model sorts them out.
• Adding a reranker without measuring latency and recall impact.

Production Considerations#

Build a regression set with real questions and expected documents. Track recall at k, citation quality, latency, and answer acceptance rate after every retrieval change.

Security#

Authorization filters belong inside retrieval. Never retrieve unauthorized chunks and rely on the model to ignore them.

Performance#

Run BM25 and vector search in parallel when possible. Rerank only a bounded candidate set so one broad query cannot exhaust the service.

Summary#

Hybrid search makes RAG more reliable by combining exact matching, semantic search, metadata constraints, and reranking. It is often the fastest path from impressive demo to dependable product behavior.