Hybrid search + reranking

You built your first RAG in the A3 journey. Chunk, embed, search, ground. On your evaluation set it scored 78% helpful. You shipped a beta. Users found it useful. You felt good.

Then a real user typed: "What is the CVE-2024-31839 fix?"

Your RAG returned three chunks about security best practices, none of which mentioned CVE-2024-31839. Wrong answer. User frustrated. You debugged and found: your embedding model turned "CVE-2024-31839" into a generic "vulnerability" vector. The exact identifier your user cared about was lost in semantics.

The next user typed: "How much does the Pro plan cost per month for teams of 50?"

Your RAG returned chunks about "Enterprise pricing" and "Team plans" and "Volume discounts" — but not the specific pricing table for the Pro plan with 50 seats. The answer was IN the docs. Your retriever just didn't find it.

Both bugs come from the same limitation: dense vector search is optimized for meaning, not exact match. Real user queries need BOTH.

The moment you accept this, you climb the retrieval quality ladder. Every production RAG system in 2025 — Perplexity, ChatGPT's browsing mode, Anthropic's contextual retrieval, Cursor's codebase search, Notion Q&A — is running some version of the same 3-stage pipeline:

The 3-stage hybrid retrieval pipeline — the only diagram you need

Rule: Stage 1 casts a wide net (recall), Stage 2 fuses candidates (deduplication + normalization), Stage 3 picks the best (precision). Every stage has a knob. Once you understand each stage's job, tuning becomes surgical.

Let me walk through each stage — the mechanism, the failure modes it prevents, and the cheap way to add it to your existing RAG.

Stage 1a: Dense retrieval — semantic search (what you already have)

What it is: embed the query into a vector, find k-nearest documents in embedding space using cosine similarity. This is what OpenAI's text-embedding-3-small, Cohere's embed-v3, Voyage's voyage-3, and open-source models like BGE-M3 or E5 do.

Anatomy:

text
    ══════════ DENSE (SEMANTIC) RETRIEVAL ══════════

    Query: "How do I reset my password?"
           ↓
    Query vector (768-dim): [0.023, -0.14, 0.87, ...]
           ↓
    Cosine similarity against corpus vectors
           ↓
    Top-100 by similarity:
      1. "Forgot password? Click here..." (0.91)
      2. "Account recovery steps..."       (0.89)
      3. "Change your credentials..."      (0.85)
      ...

What it's great at:

• Paraphrase matching — "reset password" ≈ "recover account" ≈ "change credentials"
• Cross-language (with multilingual embeddings) — "restablecer contraseña" finds English docs about password reset
• Conceptual similarity — "How do transformers work?" retrieves docs about attention mechanisms even without the word "transformer"

What it fails at:

• Exact identifiers — "CVE-2024-31839", "issue #4721", "model gpt-4o-2024-08-06"
• Numbers, prices, quantities — "50 seats", "$99/month", "128GB"
• Rare technical terms — jargon that appears in the corpus but is far from the query in embedding space
• Negation — "How do I NOT get billed?" often retrieves billing docs (embedding ignores "NOT")

Stage 1b: Sparse retrieval — keyword search (BM25, the classic that never dies)

What it is: for each term in the query, score documents by how often that term appears, weighted by how rare the term is globally. This is BM25 (Best Match 25) — the algorithm behind Elasticsearch, Solr, Lucene, and basically the entire pre-2015 search world.

BM25 formula (simplified):

text
    Score(doc, query) = Σ IDF(term) × TF(term, doc) × length_normalization

    IDF(term) = log(N / df(term))
        High for RARE terms (e.g. "CVE-2024-31839")
        Low for COMMON terms (e.g. "the", "and")

    TF(term, doc) = normalized frequency of term in doc

What it's great at:

• Exact identifiers — a CVE code, an SKU, an order number, a model name will absolutely match
• Rare terms — if a word appears in 3 docs out of 10 million, BM25 will find those 3 first
• Zero-cost interpretability — you can see WHY a doc matched (which terms hit)

What it fails at:

• Vocabulary mismatch — user says "car", doc says "automobile", BM25 finds nothing
• Semantic paraphrase — see above
• Zero recall for out-of-vocabulary queries

Modern variants:

• SPLADE — a learned sparse encoder that expands each term with related terms. Better than pure BM25, still sparse (so still keyword-searchable).
• ColBERT — token-level late interaction. Not truly sparse, but similar strengths.

The insight: dense catches semantics, sparse catches exact terms. They fail on complementary axes. Combining them gives you both.

Stage 2: Fusion — merging two ranked lists into one

The problem: dense returns top-100 by cosine similarity (0-1 range). Sparse returns top-100 by BM25 (unbounded, could be 3.2 or 47.9). You can't just add the scores — the scales are incomparable.

Reciprocal Rank Fusion (RRF) solves this. It ignores raw scores and uses only ranks.

text
    ══════════ RECIPROCAL RANK FUSION ══════════

    For each candidate document, compute:

    RRF_score(doc) = Σ  1 / (k + rank_i(doc))
                    i

    where i iterates over each ranked list (dense, sparse)
    and rank_i is the doc's position in list i (1 = best).
    k is a smoothing constant, typically 60.

    Example:
      Doc A: rank 1 in dense, rank 5 in sparse
        → RRF = 1/(60+1) + 1/(60+5) = 0.0164 + 0.0154 = 0.0318

      Doc B: rank 3 in dense, rank 1 in sparse
        → RRF = 1/(60+3) + 1/(60+1) = 0.0159 + 0.0164 = 0.0323  ← wins

    Docs that appear high in EITHER list bubble up.
    Docs that appear high in BOTH lists dominate.

Why RRF over weighted score sum: RRF ignores absolute scores, only cares about ranks. Immune to score-scale drift when models update. Elasticsearch, Weaviate, and Qdrant all ship RRF as a first-class primitive as of 2024.

Alternatives (with trade-offs):

• Weighted score fusion — normalize both scores to [0, 1], add with weights (α × dense + (1-α) × sparse). Simpler, requires tuning α per domain, breaks when either model updates.
• Learned fusion — train a classifier to combine scores. Highest quality, requires labeled data.

RRF is the default. Use it.

Stage 3: Reranking — the precision layer that changes everything

The problem: your top-50 candidates after fusion have HIGH RECALL (the right answer is probably in there) but IMPERFECT PRECISION (the top-1 might not be the best match). Passing top-50 straight to the LLM wastes context window and dilutes signal.

Cross-encoder reranking re-scores each (query, doc) pair with a much more expensive but much more accurate model. It processes query and document TOGETHER (concatenated as input), letting attention flow across both — unlike Stage 1 embeddings which process query and doc SEPARATELY.

text
    ══════════ CROSS-ENCODER RERANKING ══════════

    BEFORE RERANK (top-5 by fusion):
      1. "Change your credentials..."     (RRF 0.032)  ← generic
      2. "Password reset flow..."          (RRF 0.031)
      3. "Account recovery steps..."       (RRF 0.030)
      4. "Login troubleshooting..."        (RRF 0.028)
      5. "MFA setup guide..."              (RRF 0.025)

    RERANK (query + each doc → cross-encoder → relevance score):
      "How do I reset my password?" + Doc1 → 0.72
      "How do I reset my password?" + Doc2 → 0.94  ← promoted
      "How do I reset my password?" + Doc3 → 0.81
      "How do I reset my password?" + Doc4 → 0.55
      "How do I reset my password?" + Doc5 → 0.31

    AFTER RERANK:
      1. "Password reset flow..."          (0.94)  ← now #1
      2. "Account recovery steps..."       (0.81)
      3. "Change your credentials..."      (0.72)
      4. "Login troubleshooting..."        (0.55)
      5. "MFA setup guide..."              (0.31)  ← drops out if we take top-3

Options:

• Cohere Rerank v3.5 — SaaS, ~100ms for 50 docs, best-in-class quality, ~$0.001 per query
• BGE-reranker-v2-m3 — open-source, ~200ms on GPU for 50 docs, close to Cohere quality
• Voyage rerank-2 — SaaS, cheaper than Cohere, similar quality
• In-house cross-encoder — fine-tune on your domain, best possible quality, MLops burden

Latency budget: rerank adds ~100-300ms. If your P99 latency budget is tight (chat streaming), rerank the top-20 (not top-50) or skip rerank on simple queries.

Impact — the killer stat: on standard benchmarks (BEIR, MTEB), adding a cross-encoder rerank stage after dense+sparse+RRF typically improves nDCG@10 by 15-30 percentage points. That's the difference between a helpful RAG and a frustrating one.

When can you skip stages? — the decision tree

The retrieval quality ladder — L4 to L7

• L4 (starter): Dense-only. Cosine similarity over one embedding model. Works up to ~10K docs on a homogeneous corpus. This is your A3 journey RAG.

• L5 (professional): Dense + sparse + RRF fusion. Add hybrid the moment your users complain about missing exact terms or CVE-style identifiers. Elasticsearch or Qdrant handle both natively.

• L6 (advanced): Full 3-stage pipeline — dense + sparse → RRF → cross-encoder rerank. Standard for production RAG systems in 2025. Perplexity, Anthropic contextual retrieval, and enterprise Q&A all live here.

• L7 (frontier): Query understanding + query expansion + multi-index retrieval + learned rerank + LLM reflection. This is the layer where Perplexity Pro Search, ChatGPT with Search, and Google's SGE operate. Rewrites queries, retrieves from multiple indexes (web, docs, code, tables), reranks with a domain-specific model, and uses the LLM itself to critique retrieval quality mid-flight.

The most common mistake (and the cheap fix)

Teams ship a dense-only RAG, get complaints about "the AI can't find things I know are in the docs," and their instinct is to fine-tune the embedding model or swap it for a bigger one.

Do neither first. Add sparse retrieval + RRF. It takes ~30 lines of code with Elasticsearch, Qdrant, or Weaviate. It typically doubles recall on identifier-heavy queries. Only THEN consider embedding upgrades — most teams find they don't need to.

What's next in this journey:

• Chapter 1: The BM25 tuning parameters (k1, b) that Elasticsearch defaults get wrong for short-doc corpora
• Chapter 2: SPLADE and ColBERT — the "learned sparse" evolution beyond BM25
• Chapter 3: How constant k=60 in RRF was chosen (empirical study by Cormack 2009) and when to override it
• Chapter 4: Cross-encoder economics — when Cohere Rerank pays for itself vs. running BGE-reranker on your own GPU
• Chapter 5: Query rewriting with an LLM before retrieval — HyDE, step-back prompting, and multi-query generation
• Chapter 6: Retrieval evals — nDCG@10, MRR, hit rate, and building the golden dataset that lets you A/B test retrieval changes

Sources cited in this chapter:

• Cormack, Clarke, Buettcher (2009). "Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods." (paper defining RRF)
• Cohere Rerank v3.5 launch: cohere.com/blog/rerank
• BGE-reranker-v2-m3: huggingface.co/BAAI/bge-reranker-v2-m3
• SPLADE paper (Formal et al. 2021): arxiv.org/abs/2107.05720
• ColBERT (Khattab & Zaharia 2020): arxiv.org/abs/2004.12832
• Anthropic Contextual Retrieval (Sep 2024): anthropic.com/news/contextual-retrieval
• Elasticsearch hybrid search docs: elastic.co/guide/en/elasticsearch/reference/current/hybrid-search.html
• Qdrant hybrid search: qdrant.tech/documentation/tutorials/hybrid-search-fastembed
• BEIR benchmark (Thakur et al. 2021): arxiv.org/abs/2104.08663