Search Engine (Elasticsearch, OpenSearch, Meilisearch)

Historical timeline

The inverted index — foundation of full-text search

An inverted index flips the data model on its head. Instead of "doc → words," it's word → docs. Look up any term → get the list of docs containing it. O(1) per term. Try searching:

Analyzer pipeline: from raw text to searchable tokens

Before text hits the index, it goes through an analyzer: tokenize → normalize → stem → filter. Same pipeline runs on both indexed docs and queries — so "Running" matches "runs." Type in text and watch each stage:

Scoring: BM25, the industry default

"Matches" isn't enough — you need to rank. Lexical search uses BM25 (Best Match 25), a probabilistic model that scores each doc by:

score(D, Q) = Σ IDF(q_i) · (f(q_i, D) · (k1 + 1))
                        ─────────────────────────────
                        f(q_i, D) + k1·(1 - b + b·|D|/avgdl)

Vector search: understanding meaning, not just keywords

Lexical search finds documents that share words with the query. Vector search finds documents that share meaning. The trick: represent each document as a high-dimensional vector (typically 384-1536 dims) computed by an embedding model. Similar meanings → nearby vectors → cosine similarity ranks them.

Scaling: shards for capacity, replicas for throughput

A single Lucene index maxes out around 100M-1B docs. Beyond that: shard — split the index across nodes. Add replicas to serve more queries in parallel. Elasticsearch/OpenSearch default: 1 primary + 1 replica.

Product comparison

How to choose: Enterprise general search → Elasticsearch or OpenSearch. Site search managed → Algolia or Typesense. Vector-heavy RAG → Pinecone (managed) or Qdrant (OSS). Small budget or already on Postgres → pgvector. Yahoo-scale + ML ranking → Vespa.

12 real-world search deployments

Key takeaways

• 1Search is ranked retrieval, not filtered SELECT. Get 10 best-matching docs, ranked by relevance.
• 2The inverted index (term → posting list of doc IDs) is the core data structure. O(1) per term.
• 3The analyzer pipeline (tokenize → lowercase → stem → filter) must run identically on docs and queries. Change it = reindex.
• 4BM25 is the industry-default lexical scorer. Beats TF-IDF. Tunable via k1 and b.
• 5Vector search (HNSW + cosine similarity) finds documents by meaning. Standard for RAG.
• 6Hybrid search (BM25 + vector, fused) beats either alone. Production RAG uses this.
• 7Scale via shards + replicas. Shards for capacity, replicas for query throughput.
• 8For most workloads: Elasticsearch or OpenSearch for general. Pinecone/Qdrant/Weaviate for pure vector. Algolia for managed site search. pgvector when you're already on Postgres.

References & further reading

• • Salton, G. et al. (1975). "A Vector Space Model for Automatic Indexing." CACM. Foundation of modern search.
• • Salton, G. (1988). Automatic Text Processing. Addison-Wesley. The bible.
• • Robertson, S. et al. (1994). "Okapi at TREC-3." The BM25 paper.
• • Manning, C., Raghavan, P., Schutze, H. (2008). Introduction to Information Retrieval. Cambridge. Free online. The modern textbook.
• • Malkov, Yu. A. & Yashunin, D. A. (2018). "Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs." The HNSW paper.
• • Devlin, J. et al. (2018). "BERT: Pre-training of Deep Bidirectional Transformers." The paper that made neural rerankers mainstream.
• • Elasticsearch Docs: "A Practical Introduction to the Vector Space Model" blog + "Query DSL" reference.
• • Lucene JavaDoc: Actually reading the source is faster than reading blogs about it.
• • Doug Turnbull, John Berryman (2016). Relevant Search. Manning. Excellent practical scoring guide.
• • Vespa Documentation: The best hybrid + ML-ranking reference in the OSS world.
• • Pinecone Learning Center: Best vector-search-for-devs content anywhere.