Build your first RAG system

I'm going to tell you what your CTO said last Monday.

"We need to add ChatGPT to our internal docs so employees can ask questions and get answers. Two months. Ship it."

You nod. You know how to call the OpenAI API — you finished the A1 journey. You've got streaming working. You've got the cost dashboard. You're ready.

Then you actually try it. You POST your first prompt: "What is our company's PTO policy?" GPT-4o responds with a completely made-up answer that sounds authoritative. It has never seen your PTO policy. It cannot see it. The model's training cutoff was 18 months ago and you weren't in the training set.

Welcome to the reason RAG exists.

What RAG is, in one sentence

RAG (Retrieval-Augmented Generation) is the pattern where you fetch relevant documents from YOUR data at query time, stuff them into the LLM prompt as context, and let the LLM answer based on those documents.

You didn't fine-tune the model. You didn't retrain anything. You just gave the LLM your data at the moment of the query, so it can ground its answer in facts it can actually see.

Read that again. This is the entire pattern.

• Question → "What is our PTO policy?"
• Retrieval → search your Confluence/Notion/Google Drive, find the top 3 relevant snippets
• Augmentation → stuff those snippets into the prompt: "Given the following docs: [snippet 1], [snippet 2], [snippet 3] — answer: What is our PTO policy?"
• Generation → LLM produces an answer grounded in the actual snippets, with citations

That's it. Everything else in this 12-chapter journey — chunking, embeddings, vector databases, reranking, hybrid search, streaming, evaluation, cost — is a refinement of this 4-step loop.

The whole journey at a glance

Every 10× in corpus size forces a fundamentally different RAG architecture. Same discipline as the URL Shortener journey — SCALE → BOTTLENECK → ARCHITECTURE EVOLUTION:

Chapters that cover each tier: L4 → Chapter 5 (naive FAISS MVP). L5 → Chapter 6 (vector-DB migration). L6 → Chapter 7 (hybrid search + reranker). L7 → Chapter 8 (enterprise federation).

Key insight: RAG's bottleneck moves FROM "generation quality" (L4) TO "retrieval quality" (L5-L6) TO "access control & data governance" (L7). The LLM is not the hard part after L5.

Read this chart before diving in. Notice the pattern — it is the same shape as URL Shortener's L4→L7 evolution: each 10× in scale kills something you were relying on, and forces a new architectural primitive.

What's fundamentally different from a base LLM call

If A1 taught you to call the LLM API, RAG adds these five properties that base LLM calls cannot deliver:

The single most important property is the third one — citations that are real. Users trust an AI feature not because it's smart but because they can check its work. A ChatGPT response with a made-up citation is worse than no citation. A RAG response with three real citations from your Confluence is a professional tool your CEO will pay for.

When RAG wins vs when fine-tuning wins vs when neither wins

You will be asked this question in every AI architecture interview:

Rule of thumb: if the answer changes over time or per-user, use RAG. If the behavior is stable and shared across users, use fine-tuning. If neither, use the base model. The default answer for enterprise AI features in 2025 is RAG.

Where the rest of this journey goes

Chapters 1-4 — foundations:
- Ch 1 Requirements — the 5 clarifying questions before you touch code
- Ch 2 Estimation — corpus size, query volume, embedding budget, latency budget
- Ch 3 API surface — the 3 endpoints: /ingest, /query, /admin
- Ch 4 Data model — chunks, embeddings, metadata, ACL

Chapters 5-8 — the scale evolution shown above.

Chapters 9-12 — failures, evaluation, production, defense:
- Ch 9 When RAG breaks — hallucination-with-citations, wrong-chunk-retrieved, stale index
- Ch 10 Evaluation — how to actually measure if your RAG is good (RAGAS, LLM-as-judge, golden datasets)
- Ch 11 Production checklist — the 12 items a senior AI architect checks before shipping
- Ch 12 Defense — the 8 questions your interviewer will ask about your RAG design

Chapters queued (session note)

Chapters 1-12 of this journey are queued for follow-on authoring sessions. This Chapter 0 exists so you can see the whole arc and decide if RAG is right for your current problem. Come back as the remaining chapters ship.

Newbie mentor commentary — what to internalize NOW

1. RAG is not a technology. It is a pattern. The pattern is: retrieve → augment → generate. The technology (FAISS, Pinecone, Qdrant, Weaviate, pgvector) is substitutable. Do not fall in love with a vendor.

1. Retrieval quality dominates generation quality above L5. Once you have GPT-4o class models, the LLM will produce a good answer IF you give it good context. The hard problem is finding the right context — and that's what chapters 6-8 are about.

1. You will over-invest in the vector DB and under-invest in the chunker. Every RAG project I have seen ships with a fancy vector database and a naive chunking strategy. The chunker (Ch 4) determines whether you retrieve useful snippets or useless fragments. Get it right first.

1. You will forget the evaluation step until it's too late. Ship a RAG system without evals and you'll spend 6 months chasing "why is the answer wrong sometimes" with no way to measure improvement. Build the evaluation harness in Chapter 10 BEFORE you optimize retrieval.

1. RAG is 80% of the enterprise AI budget in 2025. Every internal Q&A bot, customer support agent, sales enablement tool, and knowledge platform ships with RAG. If you master this journey, you are qualified for the majority of production AI work being done today.

References (8 items)

• Lewis et al. (2020) — "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" (arxiv.org/abs/2005.11401) — the paper that coined "RAG" and formalized the retrieve-then-generate pattern.
• Karpukhin et al. (2020) — "Dense Passage Retrieval for Open-Domain Question Answering" (arxiv.org/abs/2004.04906) — the DPR paper that made dense retrieval competitive with BM25.
• Gao et al. (2024) — "Retrieval-Augmented Generation for Large Language Models: A Survey" (arxiv.org/abs/2312.10997) — the definitive survey covering naive RAG → advanced RAG → modular RAG.
• Anthropic's Contextual Retrieval (Sep 2024) — anthropic.com/news/contextual-retrieval — how prepending chunk context reduced retrieval failure by 49%. Reference for L6 chunking strategy.
• OpenAI Cookbook: Q&A with RAG — github.com/openai/openai-cookbook — the canonical working example most teams start from.
• Pinecone Learning Center: RAG — pinecone.io/learn/retrieval-augmented-generation — solid vendor-neutral introduction.
• Chapter 6.5 of URL Shortener journey — the "why Redis" cache decision framework applies to vector DBs too (glob "cache" for "embeddings", glob "Redis" for "Pinecone/Qdrant/pgvector").
• RAGAS framework — docs.ragas.io — the standard evaluation framework covered in Chapter 10.

You're back. Your PM handed you the "add ChatGPT to our internal docs" project. You read Chapter 0. You know RAG is the pattern.

Now — do not open your editor yet.

I have watched dozens of engineers ship broken RAG systems because they skipped this step. They immediately reached for LangChain, wired up a vector database, chunked the docs, and shipped a demo. Three weeks later: users complain the answers are wrong, the finance team asks why the OpenAI bill is $12K/mo, and security says the bot leaked the CEO's salary spreadsheet.

Every single one of those failures traces back to a question that was not asked at requirements time.

Here are the 5 questions a Staff-level AI architect asks BEFORE writing any code. Each question has an architectural consequence. If you skip a question, you either over-build (waste money) or under-build (ship a broken product). The decision map at the end shows exactly how the answers compose into a starting architecture.
## The 5 questions, at a glance

Rule: EVERY question changes what appears in your architecture. If a question does not change architecture, do not ask it.

Now let me walk through each — the "why," the options, the architectural consequence, and the anti-pattern to avoid.

Question 1: Corpus type — what are you retrieving from?

Why it matters: the parser and chunker are the FIRST failure mode of any RAG system. Retrieval quality is bounded by what your parser could extract. If your parser drops the tables from a PDF, no vector DB in the world will find that data.

Options and trade-offs:
- All-text PDFs → cheapest path. pdfplumber handles 95% of cases. Budget: 1 engineer-week for the ingestion pipeline.
- Scanned PDFs / images → OCR required. Adds 3-10× cost and 30% error rate on top of retrieval. Budget: 3 engineer-weeks. Consider outsourcing to Azure Document Intelligence or AWS Textract.
- Code → tree-sitter parses AST per language. Chunk by function/class, NOT by line count. Cursor.sh and Sourcegraph do this. Budget: 2 engineer-weeks per language you support.
- Mixed corpus (the real answer at any enterprise) → build a router that sniffs mime-type or file extension and dispatches to per-type parsers. This is where 60% of enterprise RAG ingestion complexity lives.

Anti-pattern: using LangChain's default PyPDFLoader on scanned PDFs, getting empty strings, and blaming the vector DB. Test your parser on 10 random docs BEFORE touching retrieval.

Interview soundbite: "First I asked what the corpus was. If it's text-based PDFs I use pdfplumber and heading-aware chunking. If it's scanned I add an OCR gate with a confidence threshold and route low-confidence docs to a human review queue. If it's code I use tree-sitter and chunk by function. The parser is the L1 failure mode — I test it on 10 representative docs before touching anything downstream."

Question 2: Freshness SLA — how stale can the answer be?

Why it matters: freshness determines whether you can batch-reindex nightly (cheap, simple) or need a real-time ingestion pipeline (expensive, complex). This decision alone drives 10× cost differences.

Options and trade-offs:
- "Never re-indexed" → the docs are static (e.g., legal contracts frozen at signing). Cheapest RAG possible. Reindex is a manual quarterly job.
- "Daily is fine" → cron job at 2AM runs the full ingest. Simplest production RAG. This is 80% of enterprise use cases.
- "Hours" → still a cron, but every 4 hours. Same simplicity, more compute.
- "Minutes" → webhook from the source system (Confluence page saved → HTTP POST to your ingestion queue → worker embeds and upserts). 4× the operational complexity.
- "Real-time / seconds" → CDC (Change Data Capture) from the source DB. Kafka Connect, Debezium, or similar. 10× cost, 10× complexity. Only pick this if the business truly needs it.

Anti-pattern: the PM says "well ideally real-time." You build a Kafka pipeline. Six months later you learn 99% of your queries were about docs that hadn't changed in 2 years. You wasted $200K.

The right question: "if we're 24 hours stale, does the user care?" 90% of the time the honest answer is no — pick daily batch.

Interview soundbite: "Freshness question. Batch nightly gets you 80% of the value at 10% of the cost. I default to daily unless there's a specific business reason for tighter — like legal docs that must reflect the latest amendments within an hour. Real-time is a $200K/yr decision I make people justify."

Question 3: Query volume — 100/day or 100M/day?

Why it matters: volume determines your L-tier from the Chapter 0 chart. The architecture at 100 queries/day (single-node FAISS) has almost nothing in common with 100M queries/day (federated multi-tenant).

Options and trade-offs:
- L4 (<100 QPS): in-mem FAISS rebuilt every deploy. NO vector DB. Store embeddings in a JSON file if you must. Total cost: your LLM API bill + $50 in cloud storage. This is the correct starting point for 90% of internal tools.
- L5 (1-100 QPS): dedicated vector DB (Pinecone Starter or Qdrant on a single VM). Metadata filtering. Embedding cache. The workhorse tier for company-wide RAG.
- L6 (100-10K QPS): hybrid search (BM25 + dense) with cross-encoder reranking. Sharded vector index. Streaming responses. Real budget engineering starts here.
- L7 (>10K QPS): federated per-tenant, ACL-native retrieval, multi-model ensemble. This is Perplexity / Notion AI territory.

Anti-pattern: starting at L6 because you "might grow to 1M queries." You will not. Your first version will get 200 queries/day and you will have spent 4× more time on infra than product. Ship L4, evolve to L5 when your metrics tell you to.

Interview soundbite: "Query volume drives the L-tier. I default to L4 with FAISS and a single embed model — cheapest, fastest to ship, sufficient for any tool serving <1000 queries/day. I evolve to L5 (dedicated vector DB) when p95 retrieval latency exceeds 100ms or corpus grows past 10K docs. I only touch L6 when actual traffic proves the need — never speculatively."

Question 4: Access control — who sees what?

Why it matters: ACL is the most commonly-missed requirement. If your corpus contains HR data, salary spreadsheets, board decks, or per-customer data, retrieval must filter by the querying user's permissions BEFORE the LLM sees the snippets. Otherwise the LLM will happily leak everything.

Options and trade-offs:
- Public → the docs are public. No filter. This is a marketing bot or public-help RAG. Simplest possible ACL story.
- All-employees → single global index. Filter only checks "user is authenticated." This is your internal wiki RAG. 60% of enterprise cases.
- Role-based → each chunk carries a required_roles metadata array. At query time filter retrieved.metadata.required_roles ⊆ user.roles. This is your engineering-vs-HR-vs-legal case.
- Per-user → chunks carry owner_id. Filter chunk.owner_id == user.id. This is Notion-AI-for-personal-notes.
- Per-tenant (SaaS) → SEPARATE VECTOR INDEX PER TENANT. Do not share indexes across tenants and try to filter at query time — it's a compliance and performance nightmare. This is a design mistake that Pinecone's namespaces and Qdrant's collections were built to prevent.
- Regulatory-siloed → PII residency, HIPAA, GDPR. Federated retrieval across per-jurisdiction indexes. Every query logged for audit. This is bank / healthcare territory.

Anti-pattern: "we'll add ACL later." You will not. Retrofitting ACL onto a running RAG system means re-embedding your entire corpus with metadata columns you didn't add originally. Metadata schema is decided at requirements time, not later.

The one thing to remember: the LLM sees EVERY snippet you pass to it. If a snippet leaks past your filter into the prompt, the LLM will use it in the answer. Filter at retrieval, not at generation.

Interview soundbite: "ACL is the requirement most engineers miss. I define it at requirements time because the metadata schema is baked into the embedding table. For all-employees I use a single index. For role/team/per-user I add metadata filters at query time. For per-tenant SaaS I use separate indexes per tenant — never shared. For regulated data I federate across per-jurisdiction indexes with query audit logging."

Question 5: Evaluation — how do you know it's working?

Why it matters: without an eval harness, you cannot tell whether your last change made retrieval better or worse. You will spend 6 months tuning prompts based on vibes. Ship evals BEFORE optimization.

Options and trade-offs:
- None → guaranteed to fail. You will not know when things break. Do not ship without at least user thumbs up/down.
- User thumbs → cheapest signal. Ship this on day 1 of production. Reviewable weekly. Insufficient alone.
- LLM-as-judge → GPT-4o rates every response for faithfulness (does the answer match the retrieved snippets?) and relevance (does the answer address the question?). ~$0.001 per eval. Reasonable production default.
- Gold dataset → 100 canonical Q/A pairs authored by SMEs. Test suite for regressions. This is what production teams actually use.
- RAGAS — the standard open-source framework. Combines retrieval metrics (context precision, context recall) and generation metrics (faithfulness, answer relevance). Free.
- Human-in-loop — for regulated / high-stakes RAG (medical, legal), SME reviews a sample every week. Adds $50-500/week in labor.

Anti-pattern: shipping without evals, then trying to add them 6 months later when quality complaints hit. Your gold dataset needs to be built collaboratively with SMEs — that's a 3-4 week social process, not a 1-hour code task.

Interview soundbite: "Evals ship on day 1. Minimum viable: user thumbs + LLM-as-judge on every response. Production standard: a 100-item gold dataset built with SMEs, plus RAGAS for retrieval + generation metrics. Without evals I cannot tell if my last commit made things better or worse. Everyone regrets skipping this — no exceptions."

The composite — how the 5 answers become an architecture

Here is what those 5 answers actually compose into. This is your starting architecture. Chapter 5-8 will evolve it as scale demands.

text
   ═══════════ YOUR ANSWERS → YOUR ARCHITECTURE ═══════════

    Question           Your answer example      Architecture piece
    ─────────────────  ─────────────────────    ────────────────────
    1. Corpus type     Confluence + Google       BeautifulSoup + Google
                       Docs (all-text HTML       Docs API + heading-
                       and Docx)                 aware chunker

    2. Freshness       Daily is fine             Nightly cron ingestor
                                                 (Airflow or GitHub Actions)

    3. Query volume    1K queries/day            L5 tier
                       peak 5 QPS                Pinecone starter tier
                                                 + embedding cache

    4. ACL             All-employees             Single global index
                                                 SSO-gated /query endpoint
                                                 No metadata filter

    5. Evaluation      LLM-as-judge + user       GPT-4o eval endpoint
                       thumbs on every           + weekly SME review
                       response

   ─── COMPOSITE ARCHITECTURE ───

    Confluence + Google Docs
              │
              ▼
    ┌─────────────────────┐
    │ Nightly ingestion   │ (cron @ 2AM)
    │ - fetch changed     │
    │ - parse (BS4/Docs)  │
    │ - chunk (heading)   │
    │ - embed (OpenAI     │
    │   text-embedding-   │
    │   3-small)          │
    │ - upsert → Pinecone │
    └──────────┬──────────┘
               │
    ┌──────────▼──────────┐    ┌─────────────────┐
    │ Pinecone starter    │    │  SSO / OIDC     │
    │ single index        │◀───│  authentication │
    │ all-employees ACL   │    │  gate           │
    │ (no metadata filter)│    └────────┬────────┘
    └──────────┬──────────┘             │
               │                        │
               │             ┌──────────▼──────────┐
               │             │  User query         │
               │             │  (from Slack bot,   │
               │             │  Teams bot, or web  │
               │             │  UI)                │
               │             └──────────┬──────────┘
               │                        │
               └──────────┬─────────────┘
                          │
                 ┌────────▼──────────┐
                 │  /query endpoint  │
                 │  1. embed query   │
                 │  2. retrieve top5 │
                 │  3. stuff prompt  │
                 │  4. GPT-4o call   │
                 │  5. return answer │
                 │     + citations   │
                 └────────┬──────────┘
                          │
                 ┌────────▼──────────┐
                 │  Eval hooks       │
                 │  - user thumbs    │
                 │  - GPT-4o judge   │
                 │  - log to BigQuery│
                 └───────────────────┘

    Total infra cost: ~$400/month
    Total dev time  : 3-4 weeks
    Team size       : 1-2 engineers

And here's the same composite as a proper architecture flowchart — the version you'd sketch on an interview whiteboard:

How to read the composite: each colored subgraph maps directly to one of the 5 requirements questions. Blue = corpus type (Q1). Green = freshness (Q2, nightly cron). Amber = query volume (Q3, Pinecone starter = L5-tier). Pink = ACL (Q4, single-index + SSO gate). Purple = evaluation (Q5, thumbs + LLM-as-judge). If any color is missing from your architecture, you either skipped a requirement question or your answer to that question was "not applicable" (which should also be documented).

This is L4-going-on-L5. Simple. Correct. Cheap. Ships fast. That's the goal at Chapter 1.

The interview close

When the interviewer asks "how would you build our RAG feature?" your first response is:

"Before I design, five questions. What's the corpus type — that drives the parser. What's the freshness SLA — daily batch or streaming. What's the query volume — that picks the L-tier. What's the ACL model — single-tenant, per-tenant, or role-based. What's the eval strategy — I ship user thumbs and LLM-as-judge on day 1. Once I have those five answers I can draw an architecture in 2 minutes and defend every component. Without them I'd just be guessing."

That opening is worth 15 minutes of interview time by itself. It shows discipline. It shows you have shipped RAG in production. It shows you know the failure modes. Every senior AI interviewer respects it.

References (6 items)

• LangChain document loaders — python.langchain.com/docs/integrations/document_loaders — the largest open collection of parsers for every corpus type (use as reference, not necessarily as production code).
• LlamaIndex node parsers — docs.llamaindex.ai/en/stable/module_guides/loading/node_parsers — the chunking strategy library that Chapter 4 will draw from.
• RAGAS — docs.ragas.io — the eval framework Chapter 10 covers.
• Debezium — debezium.io — CDC platform for real-time freshness pipelines.
• Pinecone namespaces docs — docs.pinecone.io/guides/indexes/using-namespaces — the multi-tenant ACL pattern for Question 4.
• Airbnb's "How We're Building an LLM-Powered Search Engine" — medium.com/airbnb-engineering — real-world enterprise RAG with all 5 questions answered.

You got the 5 answers from Chapter 1. Now the second discipline: derive every number before you write any code.

Estimation is where junior engineers waste 3 months. They pick "some vector DB" without knowing corpus size. They pick "some LLM" without knowing per-query cost. Six months later they present to the CFO and cannot answer "why is this costing $30K/month?" — because they never did the math on day one.

Here is what a senior AI architect does in the 30 minutes AFTER requirements gathering:

1. Corpus funnel — docs → tokens → chunks → vectors → storage bytes
2. Query funnel — DAU → queries/day → peak QPS → cost per query
3. Latency funnel — parser + embed + retrieve + rerank + LLM = end-to-end
4. The economics — infra cost / month + LLM cost / month = defensible P&L line

If you skip any of the four, you either over-build (waste money) or under-build (system falls over). This chapter walks all four.
## Funnel 1: Corpus size — what am I storing?

Every RAG system starts with "we have N documents." That number is useless until you translate it into tokens, chunks, vectors, and storage bytes.

text
    ══════════ THE CORPUS FUNNEL ══════════

    ┌───────────────────────────────────────────────────────┐
    │  Documents in corpus                        1,000     │  ← ASK for this
    │  (from Confluence export / Google Drive / repo)       │
    └────────────────────────────┬──────────────────────────┘
                                 │ × 5 pages/doc average
                                 ▼
     ┌────────────────────────────────────────────────────┐
     │  Total pages                           5,000       │
     └────────────────────────────┬───────────────────────┘
                                  │ × ~500 words/page
                                  ▼
       ┌───────────────────────────────────────────────┐
       │  Total words                    2,500,000     │
       └────────────────────────────┬──────────────────┘
                                    │ × 1.3 tokens/word (OpenAI tokenizer)
                                    ▼
         ┌───────────────────────────────────────────┐
         │  Total tokens                3,250,000    │
         └────────────────────────────┬──────────────┘
                                      │ ÷ 500 tokens/chunk (chunk size)
                                      ▼
           ┌───────────────────────────────────────────┐
           │  Total chunks                    6,500    │
           └────────────────────────────┬──────────────┘
                                        │ × 1536 dims × 4 bytes/vector
                                        ▼
             ┌───────────────────────────────────────┐
             │  Embedding storage:                    │
             │  6,500 × 6,144 bytes = ~40 MB          │
             └────────────────────────────┬──────────┘
                                          │ × ~2x metadata + index overhead
                                          ▼
               ┌─────────────────────────────────────┐
               │  Total vector DB storage: ~80 MB    │ ← trivial for FAISS
               │  Compressed on disk: ~30 MB (Parquet)│   or Pinecone free
               └─────────────────────────────────────┘

    Corpus tier boundary check:
      < 100 MB     → L4 (in-mem FAISS, JSON file, no vector DB)
      < 10 GB      → L5 (single-node Pinecone starter, Qdrant on 1 VM)
      < 1 TB       → L6 (sharded Qdrant, Weaviate, multi-node vector DB)
      > 1 TB       → L7 (federated per-tenant, tiered storage)

    Rule: every 100x in corpus size forces a new vector-DB architecture.

Same corpus funnel as a proper flowchart — this is what you'd sketch on the whiteboard when the interviewer asks "walk me through your sizing":

Assumptions you need to defend:
- 1.3 tokens/word — English averages this via the OpenAI tokenizer (BPE with ~50K vocab). Code averages ~2 tokens/word due to symbols. Chinese/Japanese: ~2-3 tokens per character. Adjust per corpus.
- 1536 dimensions — that's OpenAI's text-embedding-3-small default. Options: 512 (Cohere embed-english-light), 768 (Sentence-BERT), 1024 (E5-large), 1536 (OpenAI small), 3072 (OpenAI large). Trade dimensionality for quality; storage cost scales linearly.
- 500 tokens/chunk — standard "semantic chunk" size. Options: 128 (very focused, more chunks), 500 (default), 1000 (broader context, fewer chunks). Trade granularity for retrieval precision. Deep dive in Chapter 4.
- ~2x overhead — index structures (HNSW graph, product quantization codebook) plus metadata (source, chunk_id, page_num, ACL fields) roughly double raw vector bytes.

Interview soundbite: "I converted the corpus to tokens at 1.3 tokens/word, chunked at 500 tokens, embedded at 1536 dims × 4 bytes = 6.1KB per vector. With 2× overhead for index + metadata I get ~80MB total for a 1K-doc corpus — trivially L4 in-mem FAISS territory. Every 100× in size crosses a tier boundary."

Funnel 2: Query volume — what am I processing?

You know the corpus. Now: how many queries hit the system, and what does each cost?

text
    ══════════ THE QUERY FUNNEL ══════════

    ┌───────────────────────────────────────────────────────┐
    │  Daily Active Users (DAU)                       500   │  ← ASK for this
    │  (from PM: expected users of the RAG feature)         │
    └────────────────────────────┬──────────────────────────┘
                                 │ × 4 queries/user/day (realistic for
                                 │   internal Q&A bots)
                                 ▼
     ┌────────────────────────────────────────────────────┐
     │  Queries per day                       2,000       │
     └────────────────────────────┬───────────────────────┘
                                  │ ÷ 86,400 seconds/day
                                  ▼
       ┌───────────────────────────────────────────────┐
       │  Average QPS                          0.023   │
       └────────────────────────────┬──────────────────┘
                                    │ × 4 (peak factor — 9-11AM
                                    │   spike for internal tools)
                                    ▼
         ┌───────────────────────────────────────────┐
         │  Peak QPS                       ~0.1     │ ← DESIGN for this
         └────────────────────────────┬──────────────┘
                                      │
                                      │  Now: what does one query cost?
                                      ▼
           ┌───────────────────────────────────────────────────┐
           │  Per-query breakdown:                             │
           │                                                   │
           │  Step 1: Embed the query                          │
           │    ~50 tokens × $0.02 / 1M tokens (embed-3-small) │
           │    = $0.000001 = 1 microcent                      │
           │                                                   │
           │  Step 2: Vector search                            │
           │    Local FAISS = free                             │
           │    Pinecone Starter = free tier                   │
           │    (paid: ~$0.00003 per query at scale)           │
           │                                                   │
           │  Step 3: LLM generation                           │
           │    Input:  5 chunks × 500 tokens + query + system │
           │         ≈  2,800 input tokens                     │
           │    Output: ~300 tokens (typical answer length)    │
           │    Cost: 2,800 × $2.50/1M + 300 × $10/1M          │
           │         = $0.007 + $0.003 = $0.010 per query      │
           │  (GPT-4o rates as of 2025)                        │
           │                                                   │
           │  Total per query: ~$0.010 (~1 cent)               │
           └────────────────────────────┬──────────────────────┘
                                        │ × 2,000 queries/day
                                        ▼
             ┌───────────────────────────────────────────┐
             │  Daily LLM cost: $20                      │
             │  Monthly LLM cost: ~$600                  │
             └────────────────────────────┬──────────────┘
                                          │ + infra
                                          ▼
               ┌─────────────────────────────────────────┐
               │  Total monthly cost:                    │
               │  LLM: $600  +  infra: $50  +            │
               │  eval (LLM-as-judge): $600 (yes,        │
               │  evals double LLM cost - budget for it) │
               │  = ~$1,250 / month                      │
               └─────────────────────────────────────────┘

   Sanity: cost per user per month = $1,250 / 500 = $2.50/user
   Business ratio: $2.50 cost vs typical SaaS $20+ revenue/user = 12% GM impact - healthy

Query funnel as a flowchart — the shape that walks the interviewer from "N users" to "$1,250/mo":

Assumptions you need to defend:
- 4 queries/user/day — this is the realistic internal-tool number. Consumer AI features (Perplexity, Copilot) see 10-30× higher. Adjust per product.
- Peak factor 4× — internal tools spike hard at 9-11AM local. Consumer tools spike at 8-10PM. Peak factor 2-5× is the range.
- 5 chunks × 500 tokens — the standard "top-5 semantic retrieval" input. Some products stuff 10-20 chunks for better answers; some use 3 for cost.
- GPT-4o rates — $2.50/M input, $10/M output as of 2025. Other options: GPT-4o-mini ($0.15/M input) at 1/17th the cost, Claude 3.5 Sonnet ($3/$15), Gemini 2.5 Flash ($0.30/M input).

The counterintuitive insight: at low query volume, LLM cost dominates. At high volume, infra dominates. For an internal tool with 2K queries/day, $600 LLM > $50 infra. For Perplexity at 100M queries/day, $500K infra > $200K LLM. Design for the crossover.

Interview soundbite: "500 DAU × 4 queries × 30 days = 60K queries/month. At $0.01/query with GPT-4o that's $600/month LLM. Plus $600 for LLM-as-judge eval on every response. Plus $50 for infra. Total $1,250/month = $2.50 per user per month. On $20 SaaS revenue that's 12% GM impact — healthy. If we swap to GPT-4o-mini we cut LLM cost 17× at the price of some quality — that's the fine-grained lever I own."

Funnel 3: Latency budget — what's my end-to-end?

Users hit "ask" and start counting seconds. You have ~3 seconds before frustration kicks in. Where do those seconds go?

text
   ══════════ END-TO-END LATENCY BUDGET ══════════

   User hits "ask" button
        │
        ▼
   ┌─────────────────────────────────────────────────┐
   │  Step 1: Embed the query                         │
   │   OpenAI embed-3-small API call                  │
   │   50 tokens, small model                          │
   │   Typical: 50-100 ms                              │
   │   ▄▄▄▄▄  ~80 ms                                   │
   └─────────────────────────────────────────────────┘
        │
        ▼
   ┌─────────────────────────────────────────────────┐
   │  Step 2: Vector retrieval                        │
   │   FAISS local: ~5 ms                              │
   │   Pinecone p95: ~50 ms                            │
   │   Sharded Qdrant p95: ~100 ms                     │
   │   ▄▄▄▄  ~50 ms typical                            │
   └─────────────────────────────────────────────────┘
        │
        ▼
   ┌─────────────────────────────────────────────────┐
   │  Step 3: Optional reranking (L6+)                │
   │   Cross-encoder rerank of top-20 → top-5         │
   │   ~100 ms for BGE-reranker-base                   │
   │   ▄▄▄▄▄▄▄  ~100 ms  (skip at L4, add at L6)      │
   └─────────────────────────────────────────────────┘
        │
        ▼
   ┌─────────────────────────────────────────────────┐
   │  Step 4: LLM generation (the killer)             │
   │   First-token latency (TTFT): ~500 ms             │
   │   Full response (300 tokens, ~80 tok/s):          │
   │     3.75 seconds if not streamed                  │
   │   ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄  ~3,750 ms   │
   └─────────────────────────────────────────────────┘
        │
        ▼
   ┌─────────────────────────────────────────────────┐
   │  Step 5: Response assembly + citations           │
   │   String templating + citation lookup             │
   │   ▄  ~10 ms                                       │
   └─────────────────────────────────────────────────┘
        │
        ▼
   ┌─────────────────────────────────────────────────┐
   │  END-TO-END (non-streaming):                     │
   │   80 + 50 + 100 + 3,750 + 10 = 3,990 ms         │
   │   ~4 seconds — USER FRUSTRATED                    │
   ├─────────────────────────────────────────────────┤
   │  END-TO-END (STREAMING to first token):          │
   │   80 + 50 + 100 + 500 + 10 = 740 ms             │
   │   Users see first word in <1s — USER HAPPY       │
   └─────────────────────────────────────────────────┘

    LATENCY LEVER PRIORITIES (in order):
      1. STREAM the LLM response - reduces perceived latency 5-6x
      2. Skip reranking at L4-L5 - saves 100 ms
      3. Cache embeddings for repeat queries - saves 50-80 ms
      4. Use cheaper/smaller LLM (GPT-4o-mini, Gemini Flash) -
         cuts TTFT to ~200 ms and streaming to ~40 tok/s
      5. Parallelize embed + auth check - saves 30 ms

Latency funnel as a proper stage flowchart — the shape that makes the streaming lever obvious:

The single most important insight in this chapter: users measure "time to first token," not "time to full answer." If your first word arrives in <1 second, users feel the system is fast — even if the full answer takes 5 seconds. If nothing renders for 3 seconds and then a wall of text appears, users think it's broken. Streaming is not optional above L4.

Latency budget you commit to: p95 time-to-first-token < 1 second. p95 full-answer time < 4 seconds. These are the SLAs your on-call gets paged on.

Interview soundbite: "The user's perceived latency is dominated by LLM generation. I stream the response — TTFT ~500ms with GPT-4o + ~80ms embed + 50ms retrieval = ~700ms to first word. Full answer streams in over 3-4 seconds. If TTFT breaks 1s I'd swap to GPT-4o-mini (halves TTFT) before I'd touch retrieval. The retrieval budget is 200ms total (embed + search + rerank) — I only spend more if evals show quality gains."

Funnel 4: The 4-tier economics — how it composes

Everything above rolls up into the 4 scale tiers. Read this table and know EACH row's derivation cold:

The moves that unlock each tier:
- L4 → L5: dedicated vector DB replaces in-mem FAISS (embedding cache helps too)
- L5 → L6: hybrid search + reranker replace vanilla vector; separate embed model per corpus
- L6 → L7: federated per-tenant indexes replace shared; regional deployment for latency and compliance

Newbie mentor commentary — memorize these:

1. The LLM bill is the biggest line item until L7. Compute and vector DB together are usually <30% of monthly cost. Optimize LLM first — swap model, cache prompts, batch requests — before you optimize infra.

1. Prompt caching cuts LLM cost 50-90%. Anthropic Claude and OpenAI both cache prompt prefixes (Sep 2024 launches). If your system prompt is 2,000 tokens and you send it on every query, caching turns 2,000 tokens/query into 200 tokens/query for the same prompt. This is a $500/mo → $100/mo lever for a mid-scale L5 RAG.

1. Every doubling of corpus size roughly doubles storage but does NOT double query cost. Vector search is sub-linear (O(log N) with HNSW). LLM cost stays constant per query (independent of corpus size). This is a key differentiator vs "put it all in the prompt" alternatives.

1. Do not forget eval cost. LLM-as-judge on every response DOUBLES LLM cost. Budget for it explicitly. If you sample 10% instead of 100% for judge eval, you save 90% of eval cost while still catching most quality regressions.

1. The break-even DAU is the number to defend to your CFO. If it takes 100 users to hit break-even at L5 pricing, you must project 500+ users in 12 months or the RAG feature is a P&L drag. This is the number PMs care about.

Interview close

When the interviewer asks "how much will this RAG system cost at 1M queries/day?" your answer is:

"1M queries × $0.01/query = $10K/day = $300K/month for LLM alone. Plus another $300K if I run LLM-as-judge on every response — I'd probably sample at 10% to cut that to $30K. Plus $5-10K vector DB and $10K compute. Total ~$350K/month at L6. Break-even at 30K users assuming $10/user revenue. If we're not projecting that within 18 months, we should look at GPT-4o-mini which cuts LLM cost 17× to ~$25K/month total — that changes break-even to 4K users. The lever I own is model choice more than infra."

That is the answer that gets a Staff-level offer.

References (8 items)

• OpenAI pricing (openai.com/api/pricing) — text-embedding-3-small $0.02/M, GPT-4o $2.50/$10, GPT-4o-mini $0.15/$0.60. Verify at time of design; prices shift quarterly.
• Anthropic pricing (anthropic.com/pricing) — Claude 3.5 Sonnet $3/$15, Haiku $0.25/$1.25. Prompt caching launched Sep 2024.
• Anthropic prompt caching docs — docs.anthropic.com/en/docs/build-with-claude/prompt-caching. 90% write cost + 90% read cost reduction for cached prefixes.
• OpenAI prompt caching — platform.openai.com/docs/guides/prompt-caching. Free tier automatically active on prompts > 1024 tokens.
• Vector DB pricing pages — pinecone.io/pricing, cloud.qdrant.io/pricing, weaviate.io/pricing. Free tiers exist at small scale.
• BGE-reranker performance — huggingface.co/BAAI/bge-reranker-base. ~100ms per query for top-20 rerank on CPU.
• HuggingFace token estimator — huggingface.co/spaces/openai/tokenizer. Verify your specific corpus's token ratio empirically.
• RAG cost calculator template — build your own in a spreadsheet; treat the numbers above as starting hypotheses to verify against your actual corpus and traffic.

Every RAG API I have reviewed has the same 3 endpoints. The reason is that a RAG system does exactly 3 things: it ingests documents into an index, it answers queries against the index, and it lets an admin manage what's in there. Everything else is a UI concern.

But — and here is where senior engineers separate — every one of those endpoints has 5 subtle design decisions inside it. Get them right on day one and your API survives 5 years of product evolution. Get them wrong and every one of those decisions costs you a migration in production later.

I have seen teams migrate their query endpoint TWICE because they didn't decide streaming protocol on day one. I have seen ingest endpoints DDoS-attack themselves because idempotency wasn't specified. Do the work here. It pays back 100×.
## The 3 endpoints at a glance

text
    ═════════ THE RAG API SURFACE — 3 ENDPOINTS ═════════

    ┌──────────────────────────────────────────────────────────┐
    │  POST /v1/documents         — INGEST a doc into index    │
    │    - Client uploads doc (or provides a URL/reference)    │
    │    - Server parses, chunks, embeds, upserts               │
    │    - Returns: doc_id + chunk count                       │
    │    - Async for large docs (returns job_id)               │
    │    - Idempotency-Key header required                     │
    └──────────────────────────────────────────────────────────┘

    ┌──────────────────────────────────────────────────────────┐
    │  POST /v1/query             — ASK a question             │
    │    - Client sends query + optional session_id + ACL ctx  │
    │    - Server embeds, retrieves top-k, generates answer    │
    │    - Response streams via Server-Sent Events (SSE)       │
    │    - Includes structured citations                       │
    │    - Idempotency-Key optional (helps for retries)        │
    └──────────────────────────────────────────────────────────┘

    ┌──────────────────────────────────────────────────────────┐
    │  GET/PATCH/DELETE /v1/documents/{doc_id}                 │
    │                            — ADMIN operations             │
    │    - List docs (paginated)                               │
    │    - Update metadata (rarely; not the doc content)       │
    │    - Delete doc + all its chunks + all its embeddings    │
    │    - Includes eval-signal endpoints (thumbs, feedback)   │
    └──────────────────────────────────────────────────────────┘

    Rule: EVERY production RAG system has EXACTLY these 3.
          If you have 8 endpoints, 5 of them are UI features
          calling one of these 3 endpoints. Design the 3
          right and everything else falls out.

Now let me walk through each endpoint with its subtle decisions.

Endpoint 1: POST /v1/documents (INGEST)

The request contract:

text
    POST /v1/documents
    Authorization: Bearer <jwt-or-api-key>
    Idempotency-Key: <client-supplied-uuid>       ← REQUIRED

    {
      "source_type": "url" | "text" | "s3" | "upload",
      "source": "https://docs.example.com/handbook.pdf" | "text..." | "s3://...",
      "metadata": {
        "author": "engineering-team",
        "tags": ["engineering", "onboarding"],
        "acl": {
          "required_roles": ["employee"],
          "tenant_id": "acme-corp"
        }
      },
      "processing": {
        "chunk_size": 500,             ← optional overrides
        "chunk_overlap": 50,
        "embedding_model": "text-embedding-3-small"
      }
    }

Response:

text
    HTTP/1.1 201 Created (synchronous, small docs)
    HTTP/1.1 202 Accepted (async, large docs)

    {
      "doc_id": "doc_01H2XY3Z...",
      "status": "processing" | "indexed" | "failed",
      "chunk_count": 47,
      "job_id": "job_..." (only if 202)   ← poll GET /v1/jobs/{job_id}
    }

The 5 subtle decisions:

1. Sync vs async processing?

Small doc (<10 pages, <500KB): process synchronously, return 201 with the full result. User pastes a page from Confluence and sees the confirmation immediately.

Large doc (>10 pages OR >500KB): process asynchronously. Return 202 with a job_id. Provide GET /v1/jobs/{job_id} for polling. Otherwise the client HTTP connection times out and the client retries — and now you have a double-ingest problem.

The rule: the 30-second HTTP timeout is your line. If parsing + chunking + embedding takes more than 30s, go async.

2. Idempotency-Key — mandatory, not optional

The problem it solves: a client uploads a 200-page PDF. Network drops mid-transfer. Client retries. Server has no way to know it's a retry. You now embed the doc twice. You spend $10 twice on OpenAI embeds. You double every chunk in your vector DB. This happens every day at every real RAG deployment without idempotency.

The pattern: client supplies Idempotency-Key header with a UUID (or hash of content + source). Server stores {key → doc_id, response, expires_at} for 24 hours. Retry with same key returns the same response — no re-processing.

Reference: RFC-draft "The Idempotency-Key HTTP Header Field" (github.com/mroth/http-idempotency-key, IETF standard track) + Stripe's implementation guide (stripe.com/docs/api/idempotent_requests).

3. Metadata schema — freeze it on day one

The metadata you commit to on day one lives in every embedded chunk. Adding a metadata field 6 months later requires re-embedding your entire corpus. That is a $10K+ mistake at any real scale.

Minimum viable metadata:
- author (who created this doc)
- source_url (where it came from)
- created_at (recency for LLM answer freshness)
- updated_at (for eviction decisions)
- tags (list of strings — flexible search axis)
- acl.required_roles (Ch 1 - filter at retrieval time)
- acl.tenant_id (per-tenant isolation - see Ch 1 Q4)
- chunk_position (which chunk of the doc; useful for re-assembly)

Add anything you might need. It's much cheaper to have an empty field than to re-embed.

4. Processing overrides — allow but do not require

Sensible defaults do 95% of the work: 500-token chunks, 50-token overlap, text-embedding-3-small. But your document types differ — a Python codebase wants 200-token chunks, a legal contract wants 1000-token chunks. Allow overrides via the processing field. Do NOT force the client to specify them.

5. ACL enforcement — at ingest, not at query

The client passes ACL metadata WITH the doc. You store it AS chunk metadata. You do NOT rely on the query-time filter alone. If a doc is public, its metadata says so. If it's per-tenant, metadata says so. The retrieval filter then applies boolean logic on the SERVER SIDE against the querying user's identity. Never trust client-side ACL claims at query time.

Anti-pattern: letting the caller specify "search only these tenant's docs" as a query parameter. If a curious/malicious user changes the query param, they see cross-tenant data. Filter by session identity, not query claim.

Endpoint 2: POST /v1/query (ASK)

The request contract:

text
    POST /v1/query
    Authorization: Bearer <jwt-or-api-key>
    Idempotency-Key: <optional-client-uuid>       ← optional but recommended

    {
      "query": "What is our PTO policy?",
      "session_id": "sess_..." (optional),        ← conversation continuity
      "options": {
        "top_k": 5,                                ← how many chunks to retrieve
        "stream": true,                            ← SSE stream vs single JSON
        "include_citations": true,
        "temperature": 0.2                        ← LLM randomness
      }
    }

Response — the STREAMING contract (recommended):

text
    HTTP/1.1 200 OK
    Content-Type: text/event-stream
    Cache-Control: no-cache

    event: retrieval
    data: {"chunks": [{"id":"chunk_1","source":"handbook.pdf","score":0.87}, ...]}

    event: token
    data: {"token": "Our"}

    event: token
    data: {"token": " PTO"}

    event: token
    data: {"token": " policy"}

    ... (many more token events streamed as generated) ...

    event: citation
    data: {"citation_id": 1, "chunk_id": "chunk_1", "text": "Section 4.2..."}

    event: done
    data: {"total_tokens": 234, "latency_ms": 3200, "cost_cents": 1}

The 5 subtle decisions:

1. Streaming protocol — SSE vs WebSocket vs long-poll

Server-Sent Events (SSE) is the correct default:
- Works over standard HTTP/1.1 — no proxy issues
- One-way (server → client) which is exactly what streaming answers need
- Simpler than WebSocket (no message framing, no ping/pong, no reconnect protocol)
- Every browser supports EventSource natively
- Standard: WHATWG HTML Living Standard §9.2 (html.spec.whatwg.org/multipage/server-sent-events.html)

When WebSocket wins: truly bidirectional (client interrupts mid-generation with "stop that, ask something else"), or high-frequency two-way (voice interaction). Overkill for typical RAG.

When long-poll wins: never at L4+; use for legacy client-side environments that block SSE.

Anti-pattern: returning a single JSON response with the full answer. Users see nothing for 3-4 seconds, then a wall of text. See Ch 2 latency funnel — streaming is not optional above L4.

2. Session state — stateless vs conversation-turn

Stateless (recommended for L4): every query is independent. No history. Simplest possible. Users ask fresh questions each time.

Conversation-turn (upgrade at L5+): client supplies session_id; server maintains conversation history (in Redis with TTL) and sends full history to the LLM on each query. Enables "and what about that?" follow-ups.

The design decision here is not "yes vs no session" — it's "who owns the history?":

• Client owns: client passes entire prior context in each request. Server is stateless. Simpler server, larger request payload.
• Server owns: server persists session state, client sends session_id. Smaller request, more complex server. Preferred for long conversations.

At L4 you can skip conversation state entirely. Add it in Ch 6 when users demand follow-ups.

3. Citation format — the 3 options

Option A: Inline text markers — "Our PTO policy is 20 days per year [Source 1]. Employees over 5 years get an additional [Source 2]." Simple; downstream can render as links. Winner for chatbots.

Option B: Footer with hyperlinks — plain answer, then a "Sources:" block at the end. Cleaner reading; harder to attribute each claim. Winner for report-style responses.

Option C: Structured citations array in response — answer as text, citations as JSON array with chunk IDs, source URLs, quote snippets. Winner for custom UI (React apps can render citations however they want).

The event: citation events in the SSE example above are Option C. Client can render as A or B based on UX preference.

4. Idempotency for queries — optional but recommended

Queries are semi-idempotent by nature (same query returns same answer if the corpus is unchanged). Idempotency-Key helps for retries during LLM slow-down: user submits, LLM is slow, client times out and retries, server has already spent $0.01 on the LLM call. With Idempotency-Key, server serves the cached in-flight response instead of doubling the cost.

Trade-off: requires a 5-minute in-memory idempotency store (Redis).

5. ACL passthrough — token to filter, filter to answer

The client's JWT (or session token) carries their identity + roles. The server:
1. Validates the JWT (signature + expiry)
2. Extracts user_id + roles + tenant_id
3. Applies these as retrieval filter: chunk.metadata.acl.required_roles ⊆ user.roles AND (chunk.metadata.acl.tenant_id == user.tenant_id OR chunk.metadata.acl.tenant_id == "public")
4. Only feeds matching chunks to the LLM

The critical property: the LLM never sees a chunk the user should not see. If a chunk leaks past the filter, the LLM will use it. Filter at retrieval, not at generation.

Endpoint 3: GET/PATCH/DELETE /v1/documents (ADMIN)

Most tutorials skip this. Real production RAG systems have 3 of these:

text
    GET /v1/documents?tag=engineering&page=1&limit=50
      → paginated list of docs (metadata only, no chunks)

    GET /v1/documents/{doc_id}
      → single doc metadata + chunk count + last-indexed timestamp

    PATCH /v1/documents/{doc_id}
      → update metadata (tags, ACL). NOT the doc content.
      → doc content updates = new POST /v1/documents (re-ingest)

    DELETE /v1/documents/{doc_id}
      → CRITICAL: delete doc + ALL its chunks + ALL its embeddings
      → GDPR/compliance requirement (right to erasure)

    POST /v1/documents/{doc_id}/reindex
      → trigger re-embed of an existing doc (new chunk strategy or
        new embedding model without changing source)

    POST /v1/query/{query_id}/feedback
      → user thumbs up/down on a specific answer
      → Ch 10 (evaluation) uses these signals

The 5 subtle decisions:

1. Deletion must be transactional across the vector DB + metadata store

If the doc is deleted from the metadata store but the vector DB still has the chunks, retrieval returns "orphan chunks" with no source. Users see [Source 5] but no way to click through — worst possible experience.

Pattern: two-phase soft-delete then hard-delete:
1. Mark doc + chunks as deleted: true in metadata (immediate)
2. Filter deleted chunks at retrieval
3. Nightly cron actually removes them from the vector DB

Reference: this is the same pattern Notion uses for deleted pages (github.com/notion — soft delete then GC).

2. GDPR / right to erasure requires TRUE hard delete within 30 days

You cannot just soft-delete forever. Regulators want real removal. Add a hard_delete_after timestamp on soft-delete; the nightly cron enforces the 30-day window.

3. Pagination — cursor-based, not offset-based

At L4 you can use ?page=1&limit=50. Above 10K docs, offset queries slow down (Postgres has to skip N rows). Switch to ?cursor=<opaque_token>&limit=50 — the cursor is the last-seen doc_id + sort field. Now every page is O(1) regardless of position.

4. Feedback endpoints are load-bearing infrastructure

The thumbs up/down + free-text feedback from users is not a "nice to have" — it is the eval signal you cannot get any other way. Ship it in the same release as /query. Ch 10 uses these signals as the first evaluation harness.

5. Admin APIs need separate rate limiting

Admin ops can be expensive (list all docs, reindex all, cascade delete). Rate-limit them separately (5 req/sec per user, not 100 req/sec like /query). Otherwise a curious admin runs list_all_docs in a loop and your DB cries.

The API design cheat sheet

For interview recall:

text
    ┌─────────────────────────────────────────────────────────┐
    │  Endpoint     Method    Idempotency    Stream    ACL    │
    │  ───────────  ────────  ──────────────  ────────  ─────  │
    │  /documents   POST      REQUIRED        no        cred   │
    │  /documents   GET       n/a             no        cred   │
    │  /documents/  PATCH     REQUIRED        no        cred   │
    │  /documents/  DELETE    REQUIRED        no        cred   │
    │  /query       POST      recommended     SSE       cred   │
    │  /jobs/{id}   GET       n/a             no        cred   │
    │  /feedback    POST      REQUIRED        no        cred   │
    └─────────────────────────────────────────────────────────┘

    STREAMING PROTOCOL PICK: SSE (never single-JSON for query)
    SESSION STATE PICK:      stateless at L4; server-owned at L5+
    CITATION FORMAT PICK:    structured array (client renders)
    ACL PATTERN PICK:        chunk metadata + server-side filter
    DELETION PICK:           soft-delete + 30-day hard-delete GC

Interview close

When the interviewer asks "walk me through your RAG API surface":

"Three endpoints. POST /v1/documents for ingest with Idempotency-Key required, sync for small docs, async with job_id for large. POST /v1/query with SSE streaming, structured citations in the response events, ACL filter applied server-side from JWT claims. Admin CRUD on /v1/documents with cursor-based pagination for scale. Two-phase soft-then-hard delete for GDPR. Feedback endpoint on /v1/query/{id}/feedback for eval signal. Every design decision defensible with a specific failure mode I've avoided — idempotency prevents double-ingest cost bloat, streaming prevents perceived-latency complaints, chunk metadata + server-side filter prevents cross-tenant leaks."

That's the answer that shows you've shipped RAG in production, not just read about it.

References (14 items)

• SSE standard — WHATWG HTML Living Standard §9.2 (html.spec.whatwg.org/multipage/server-sent-events.html) — the browser API you use for streaming.
• OpenAI streaming docs — platform.openai.com/docs/api-reference/streaming — for how OpenAI's own API streams (mirror this pattern).
• Idempotency-Key IETF draft — datatracker.ietf.org/doc/html/draft-ietf-httpapi-idempotency-key-header — the standard the industry converged on.
• Stripe idempotency guide — stripe.com/docs/api/idempotent_requests — production-proven implementation.
• Notion soft-delete architecture — engineering.notion.com — how they solve GDPR-safe deletes at billions-of-blocks scale.
• Cursor-based pagination — slack.engineering/evolving-api-pagination-at-slack — Slack's writeup on why cursors beat offsets at scale.
• Chapter 4 (upcoming) — the data model that these APIs read/write. Explains where doc_id lives, chunk_id relationships, and vector storage layout.

You have finished Chapters 0-2. You know the pattern. You know the questions. You know the math.

Now — build it.

This chapter separates the engineers who read AI blog posts from the ones who ship AI features. The right L4 MVP for a 100-doc corpus is the simplest possible thing that works. Not the most impressive. Not the most scalable. The simplest.

Here is what that looks like:

• 1 Python script (~120 lines total)
• 1 in-memory FAISS index (rebuilt every deploy)
• 1 OpenAI embedding call per doc chunk (ingested once at build time)
• 1 OpenAI GPT-4o call per user query (streamed to the browser)
• Zero vector databases
• Zero Kubernetes
• Zero LangChain

Total cost: ~$50/month. Total dev time: 2 weekends. Handles ~2K queries/day comfortably. Ships from your laptop.

I know this hurts. You spent 6 months learning LangGraph and Pinecone and reranking. And I'm telling you the right first RAG system is 120 lines of Python with FAISS. But here is the thing — every RAG system at every scale from every AI-native startup started as this exact shape. Cursor started as a script. Perplexity started as a script. Notion AI started as a script. The reason to introduce complexity is that a real problem forces you to. Never before.
## The L4 MVP topology

Here is what the whole system looks like on one whiteboard:

text
   ══════════ RAG L4 MVP — THE COMPLETE SYSTEM ══════════

    OFFLINE (once per deploy)                    ONLINE (every query)

    ┌────────────────────────────┐               ┌────────────────────────────┐
    │ /docs directory            │               │  User query                │
    │  - handbook.md             │               │  "What's our PTO policy?"  │
    │  - onboarding.pdf          │               └──────────────┬─────────────┘
    │  - engineering-guide.md    │                              │
    │  - product-roadmap.md      │                              │
    │  (100 files, ~2 MB total)  │               ┌──────────────▼─────────────┐
    └──────────────┬─────────────┘               │  1. Embed query            │
                   │                             │     OpenAI text-embedding- │
                   │                             │     3-small                │
                   ▼                             │     (~50 tokens, ~80 ms)   │
    ┌────────────────────────────┐               └──────────────┬─────────────┘
    │  1. Parse                  │                              │
    │  - md → mistune            │                              │
    │  - pdf → pdfplumber        │                              │
    │  - Extract text            │                              │
    └──────────────┬─────────────┘               ┌──────────────▼─────────────┐
                   │                             │  2. Search FAISS in-mem    │
                   ▼                             │     np.dot() over ~1000    │
    ┌────────────────────────────┐               │     vectors                │
    │  2. Chunk                  │               │     Get top-5 chunks       │
    │  - RecursiveCharacter-     │               │     (~5 ms)                │
    │    TextSplitter            │               └──────────────┬─────────────┘
    │  - 500 tokens per chunk    │                              │
    │  - 50-token overlap        │                              │
    │  - ~2000 chars per chunk   │               ┌──────────────▼─────────────┐
    └──────────────┬─────────────┘               │  3. Stuff into prompt      │
                   │                             │     "Given docs: [chunk1]  │
                   │                             │     [chunk2]... Answer:   │
                   ▼                             │     What's PTO policy?"   │
    ┌────────────────────────────┐               └──────────────┬─────────────┘
    │  3. Embed each chunk       │                              │
    │  - OpenAI text-embedding-  │               ┌──────────────▼─────────────┐
    │    3-small                 │               │  4. Generate                │
    │  - 1536 dims               │               │     OpenAI GPT-4o           │
    │  - $0.02 per 1M tokens     │               │     Streamed response       │
    │  - ~2M tokens = $0.04      │               │     ~700 ms to first token  │
    │  - one-time cost           │               │     ~3-4s to full answer   │
    └──────────────┬─────────────┘               └──────────────┬─────────────┘
                   │                                            │
                   ▼                                            ▼
    ┌────────────────────────────┐               ┌────────────────────────────┐
    │  4. Build FAISS index      │               │  5. Return + cite          │
    │  - IndexFlatIP (exact      │               │  - Stream answer text      │
    │    inner product)          │               │  - Append '[Source 1: ..]' │
    │  - Save to embeddings.faiss│               │  - Show chunk IDs          │
    │  - Load at server startup  │               └────────────────────────────┘
    │  - ~5 MB on disk           │
    └────────────────────────────┘

    Total cost: $50/mo (LLM API + hosting) - handles 2K queries/day
    Total code: ~120 lines of Python
    Deploy: one flask/fastapi container on Railway, Render, or Fly.io

The same L4 topology as a proper offline-and-online flowchart — this is what you sketch in an interview:

Why this shape works at L4: the offline pipeline is a script you run once, not a service. The online pipeline is 5 function calls, not a distributed system. If a service goes down, restarting it rebuilds everything from source. That's the definition of "the simplest thing that works."

The complete L4 MVP — actual code

Here is what all of the above looks like in Python. Read it top to bottom; ignore no line.

python
# =========================================================
# rag_mvp.py - the ENTIRE L4 RAG MVP in one file
# =========================================================
import os
import faiss
import numpy as np
from openai import OpenAI
from pathlib import Path
import pdfplumber
import mistune

client = OpenAI()

# ---------- OFFLINE INGESTION ----------

def parse_doc(path):
    if path.suffix == ".pdf":
        with pdfplumber.open(path) as pdf:
            return "\n".join(p.extract_text() or "" for p in pdf.pages)
    elif path.suffix in (".md", ".txt"):
        return path.read_text()
    return ""

def chunk_text(text, size=500, overlap=50):
    words = text.split()
    chunks = []
    for i in range(0, len(words), size - overlap):
        chunks.append(" ".join(words[i : i + size]))
    return chunks

def embed_batch(texts):
    resp = client.embeddings.create(
        model="text-embedding-3-small",
        input=texts,
    )
    return np.array([d.embedding for d in resp.data], dtype="float32")

def build_index(docs_dir):
    all_chunks = []
    all_sources = []
    for path in Path(docs_dir).rglob("*"):
        if not path.is_file():
            continue
        text = parse_doc(path)
        for chunk in chunk_text(text):
            all_chunks.append(chunk)
            all_sources.append(str(path))
    vectors = embed_batch(all_chunks)  # batch API call - one round-trip
    index = faiss.IndexFlatIP(vectors.shape[1])
    faiss.normalize_L2(vectors)
    index.add(vectors)
    return index, all_chunks, all_sources

# Run once at build time OR at server startup
index, chunks, sources = build_index("./docs")

# ---------- ONLINE QUERY ----------

def retrieve(query, k=5):
    q_vec = embed_batch([query])
    faiss.normalize_L2(q_vec)
    _, ids = index.search(q_vec, k)
    return [(chunks[i], sources[i]) for i in ids[0]]

def answer(query):
    top_chunks = retrieve(query, k=5)
    context = "\n\n".join(
        f"[Source {i+1}: {src}]\n{chunk}"
        for i, (chunk, src) in enumerate(top_chunks)
    )
    stream = client.chat.completions.create(
        model="gpt-4o",
        stream=True,
        messages=[
            {"role": "system", "content":
              "Answer based ONLY on the provided sources. "
              "Cite sources like [Source 1]. "
              "If sources do not contain the answer, say so."},
            {"role": "user", "content":
              f"Sources:\n{context}\n\nQuestion: {query}"},
        ],
    )
    for chunk in stream:
        content = chunk.choices[0].delta.content or ""
        yield content

# ---------- FASTAPI ENDPOINT ----------

from fastapi import FastAPI
from fastapi.responses import StreamingResponse

app = FastAPI()

@app.get("/query")
def query_endpoint(q: str):
    return StreamingResponse(answer(q), media_type="text/plain")

That is the entire system. ~85 lines including boilerplate. You can deploy this to Railway or Fly.io in a single docker run and it will handle 2K queries/day at $50/month.

Notice what is NOT in this code:
- No vector database (FAISS is a library, not a service)
- No LangChain, LlamaIndex, or any framework
- No async worker queue
- No authentication (add it when you have users)
- No eval harness (Ch 10)
- No prompt caching (Ch 6.5)
- No reranking (Ch 6.5)
- No streaming to a fancy React UI (works with curl for now)

You add each of those when a real problem forces you to. Not before.

The "what you did NOT build" visual

Every L4 MVP is defined equally by what you chose to skip. The senior engineer skill is defending each rejection. Here is the L4 rejection table:

text
    WHAT WE BUILT (L4)               WHAT YOU MIGHT HAVE OVER-BUILT
    ────────────────                 ──────────────────────────────
    120 lines Python                 500+ lines + 15 npm packages
    $50/mo                           $2K+/mo
    Ships in 2 weekends              Ships in 3 months

    ┌───────────┐                    ┌──────┐ ┌──────┐ ┌──────┐ ┌──────┐
    │ FastAPI   │                    │Vect  │ │Rerank│ │Cache │ │Query │
    └─────┬─────┘                    │DB    │ │model │ │layer │ │rewriter│
          │                          │(Pin/ │ │(BGE  │ │(Redis│ │(LLM   │
    ┌─────▼─────┐                    │Qdrant│ │Cohere│ │+ tuned│ │chain) │
    │ In-mem    │                    │)     │ │)     │ │TTLs)  │ │        │
    │ FAISS +   │                    └──┬───┘ └──┬───┘ └──┬───┘ └──┬───┘
    │ OpenAI    │                       │        │        │         │
    └───────────┘                    ┌──▼────────▼────────▼─────────▼───┐
                                     │  Complex ingestion pipeline:     │
                                     │  Airflow + Debezium CDC + custom │
                                     │  chunk-quality classifier +      │
                                     │  metadata extraction ML +        │
                                     │  ACL enrichment + provenance     │
                                     │  tracking + PII redaction        │
                                     └──────────────────────────────────┘

The same comparison as two clean flowcharts — print these mentally before adding any component:

BUILT · L4 correct (120 lines · $50/mo · 2 weekends):

OVER-BUILT · L4 wrong (500+ lines · $2K+/mo · 3+ months):

Read the two together: each additional box in the over-built version has a specific threshold at which it becomes correct — spelled out in the table below. Ship the L4 shape first; add a box only when a metric crosses its threshold.

Why the over-built version is wrong at L4

RULE: every box you add must delete a problem you actually have. If you add a box to prevent a hypothetical problem, you added complexity you MUST maintain today for zero gain.

Interview soundbite for this visual: "At L4 I ship 120 lines of Python with FAISS in-memory, one OpenAI embed batch call at build time, and one GPT-4o call per query with streaming. I explicitly omit vector DB, reranker, cache, query rewriter, LangChain, ingestion orchestration, ML-based chunking, and Kubernetes — each has a metric-based threshold at which I'd add it (L5 for cache and vector DB, L6 for reranker and rewriter). This handles 2K queries/day at $50/mo. Ships in 2 weekends by one engineer."

Where each dollar goes at L4

The unit economics you should burn into your brain:

Per-query cost breakdown (2,000/day = 60,000/month):

Monthly totals:

Sanity check: cost per user per month at 500 DAU = $1.28. On $20/user/month SaaS revenue = 6% GM impact — very healthy. The BIG dial: swap gpt-4o for gpt-4o-mini ($0.15 in / $0.60 out per M tok) → LLM cost drops from $600 to $50/mo (12× cut). Quality drops ~5-8 points on tricky questions. Net: consider mini as the default; escalate to gpt-4o only on flagged low-confidence answers (Ch 10 evaluation).

Newbie mentor commentary — read this before you code

1. Start with EXACTLY this code. Do not add abstractions. Do not add Django Rest Framework. Do not add SQLAlchemy. Do not extract a "Retriever" class or a "PromptBuilder" class. The 85 lines above ARE the abstraction. Every "improvement" you add before you have real production usage is technical debt you will pay for.

1. The ingestion pipeline is a function, not a service. build_index("./docs") is called at server startup. If docs change, restart the server. Restarting takes 30 seconds. That is fine at L4. At L5 you move ingestion to a separate cron job; at L6 you add a webhook queue.

1. The first thing to add is streaming. Notice stream=True in the OpenAI call. First-token latency (~700ms) is what users perceive as speed, not full-answer latency (~3 seconds). See A2 Ch 0 for why perceived latency is different from wall-clock.

1. The first thing NOT to add is authentication. If you deploy this behind a Cloudflare Access rule (SSO) or a shared secret in the URL, you have auth. Do not build a users table + JWT + refresh flow until you have users. When you do have users, use Clerk or Auth0 - do not roll your own.

1. Ship this to production before you ship anything fancier. The L4 MVP running in prod with 20 real users teaches you MORE than 3 months of L6 architecture on paper. What you learn: which chunks users actually retrieve, which queries fail, what latency they tolerate, what answer format they trust. That signal drives every L5+ decision.

When to evolve to L5

The precise triggers that move you from this L4 to the L5 architecture (dedicated vector DB, real ingestion pipeline, embedding cache):

• Corpus grows past ~1 GB of raw text. FAISS in-memory still works, but server restart on doc change takes >5 minutes. Move to Pinecone starter or Qdrant on a single VM.
• Traffic exceeds ~10 QPS peak. FAISS on the request thread starts blocking other requests. Move retrieval to a background service (still FAISS, but separate process).
• Users report freshness lag. "I added a doc 4 hours ago and the bot still says it does not exist." Move to an ingestion webhook + queue (Ch 6).
• You start caching queries. Same query 10× per day = 10× wasted LLM calls. Add an embedding+response cache. (Ch 6.5 covers this - it's the same pattern as URL Shortener's cache-aside.)
• Your eval harness catches a regression in retrieval quality. You now need reranking. (Ch 6.)

If NONE of the above is true, you do not need L5. Stay at L4 as long as it works.

The 2-weekend build plan

text
    ┌─────────────────────────────────────────────────────────┐
    │  WEEKEND 1 (offline pipeline)                           │
    ├─────────────────────────────────────────────────────────┤
    │  Sat AM: Set up repo + OpenAI + FastAPI skeleton        │
    │  Sat PM: Write parse_doc + chunk_text for your corpus   │
    │           types. Test on 5 real docs.                   │
    │  Sun AM: Write embed_batch + build_index. Verify        │
    │           embeddings.faiss saves + loads.               │
    │  Sun PM: Write retrieve(). Test on 10 real questions.   │
    │           Read the top-5 chunks by hand for each.       │
    ├─────────────────────────────────────────────────────────┤
    │  WEEKEND 2 (online + ship)                              │
    ├─────────────────────────────────────────────────────────┤
    │  Sat AM: Write answer() with streaming.                 │
    │  Sat PM: Add citation formatting. Test end-to-end.      │
    │  Sun AM: Deploy to Railway/Fly.io. Set env vars.        │
    │  Sun PM: Share with 5 real users. Watch what they ask. │
    └─────────────────────────────────────────────────────────┘

    2 weekends. 120 lines. $50/month. Real users.
    That is the L4 milestone.

References (8 items)

• FAISS documentation — faiss.ai — the library reference. Facebook AI Research original; still the fastest CPU-based vector search.
• FAISS IndexFlatIP tutorial — github.com/facebookresearch/faiss/wiki/Getting-started — the exact "brute force cosine similarity" index we use.
• OpenAI embeddings API — platform.openai.com/docs/guides/embeddings — text-embedding-3-small at $0.02/M tokens.
• OpenAI Python SDK streaming docs — platform.openai.com/docs/api-reference/streaming — the stream=True pattern used above.
• FastAPI StreamingResponse — fastapi.tiangolo.com/advanced/custom-response — how to plumb OpenAI stream to HTTP.
• Railway (railway.app), Fly.io (fly.io), Render (render.com) — the 3 easy Python deployment platforms at L4 scale.
• The A3 RAG Ch 6-8 chapters (upcoming) — cover the L5, L6, L7 evolutions of this exact code.
• Chapter 5 of URL Shortener journey — the SAME 'what you did NOT build' rejection pattern applied to a URL shortener. Read both together to internalize the senior-engineer skill of principled rejection.

Your L4 RAG is live. Users are complaining. When you check the failures, one pattern dominates:

The right document was in the top-100 results — but not the top-5. So it never made it to the LLM's context. Wrong answer.

This is the recall problem. And it's the #1 quality bug in production RAG. It's why you need reranking + hybrid search — the two techniques that turn a "kind of works" RAG into a production-grade one WITHOUT rebuilding your stack.
## Why pure vector search misses good docs

Vector embeddings compress semantic meaning into ~1500 dimensions. They're good at "kind of like." They're TERRIBLE at:

1. Exact terminology — "customer_id" vs "customer identifier" — semantically identical, embed-similarly-but-not-identically. Rare terms get lost in the noise.
2. Numbers and dates — "revenue in 2023" vs "revenue in 2024" — embeddings barely distinguish. BM25 nails it.
3. Rare tokens — "SM05137 CodeQL alert" — the specific alert code is high-signal for text search, low-signal for semantics.
4. Acronyms — "S3 encryption" vs "Amazon Simple Storage Service encryption" — embeddings correlate them, but often demote the exact-match one.
5. Negation — "does NOT support X" retrieves docs that DO support X. Embeddings just see the topic tokens.

Reading the top-5 vector results feels like reading close synonyms of your query. The document with the exact answer is often at position 27.

The two-stage retrieval pattern

The industry solution — used by every production RAG worth trusting — is two-stage retrieval:

```
1. RETRIEVAL → pull top-N candidates (N = 50-100) cheaply
- Vector search (dense) → catches semantic matches
- BM25/keyword search (sparse) → catches exact terms
- Merge with reciprocal rank fusion (RRF)

2. RERANKING → re-order the N candidates with a heavier model
- Cross-encoder like BGE-Reranker or Cohere Rerank
- Takes (query, doc_i) pairs and outputs a relevance score
- Returns top-5 after re-ranking

3. GENERATION → pass the reranked top-5 to the LLM as context
```

Stage 1a: Vector search (already have this)

Same as your L4: embed the query, cosine-similar top-100 from the vector store.

Stage 1b: BM25 keyword search (NEW)

BM25 (Best Matching 25) is a statistical ranking function from 1994. It scores docs by term-frequency × inverse-document-frequency with document-length normalization. It has been the industry standard for keyword search for 30 years, and it still beats naive vector search on exact-term matches.

Implementation options:
- Elasticsearch — the gold standard. Real-time indexing, distributed, mature. ~$500/mo for a small cluster.
- Postgres full-text search (`tsvector`) — built-in. Free if you already have Postgres. Slower to index, but good enough for corpora <10 GB.
- Meilisearch / Typesense — modern, embeddable, easy setup. ~$0-100/mo.
- BM25 in the vector DB itself — Weaviate, Qdrant, and Vespa now support hybrid retrieval natively. One less service to run.

The rule: if your vector DB supports hybrid, use it. Otherwise pick the search engine your team already knows. Do NOT run Elasticsearch just for hybrid RAG unless you're at L6+ scale.

Stage 1c: Reciprocal Rank Fusion (RRF) merge

You have two rank lists — 100 vector results, 100 BM25 results. Some docs appear in both. Some in one. How do you combine?

The naive answer: normalize scores + average. Wrong. Vector scores are cosine similarity (0-1); BM25 scores are unbounded. Mixing them is apples-and-oranges.

The industry answer: Reciprocal Rank Fusion. Simple, no score normalization, works surprisingly well.

python
def rrf(rank_lists, k=60):
    scores = defaultdict(float)
    for rank_list in rank_lists:
        for rank, doc_id in enumerate(rank_list):
            scores[doc_id] += 1.0 / (k + rank + 1)
    return sorted(scores.items(), key=lambda x: -x[1])

The k=60 is a smoothing constant. Higher k means the top-1 and top-2 positions carry similar weight; lower k means position 1 dominates. 60 is the tuned default from the original RRF paper.

Doc at position 1 in list A + position 5 in list B gets 1/61 + 1/65 = 0.0318. Doc at position 3 in list A + position 100 in list B gets 1/63 + 1/160 = 0.0221. The first wins.

Stage 2: Cross-encoder reranking (the game-changer)

Here's the trick that separates production RAG from prototype RAG:

Regular embeddings encode a document independently. Cross-encoders encode (query, document) as a PAIR. They see both texts together and can attend across them. The relevance score is directly optimized.

Cost/benefit:
- Bi-encoder (regular embeddings): ~50ms for 100 docs. Cheap.
- Cross-encoder (reranker): ~500ms for 100 docs. 10x slower. But 10-30% better recall.

The math: cross-encoders can't PRE-COMPUTE document embeddings, because the query is part of the input. So they must run at query time for every (query, doc) pair. This is why you pre-filter with cheap vector search to a smaller candidate set (top-100), then rerank.

Popular rerankers:

The full retrieval code

python
# STAGE 1a: Vector search
query_embedding = embed(query)  # OpenAI text-embedding-3-small
vector_hits = vector_db.search(query_embedding, top_k=50)

# STAGE 1b: BM25 keyword search
keyword_hits = elastic.search(query, top_k=50)

# STAGE 1c: RRF merge
candidate_ids = rrf([
    [h.id for h in vector_hits],
    [h.id for h in keyword_hits]
])[:100]

# Deduplicate + fetch full docs
candidate_docs = doc_store.get_batch(candidate_ids)

# STAGE 2: Cross-encoder reranking
pairs = [(query, doc.text) for doc in candidate_docs]
rerank_scores = cross_encoder.score(pairs)
reranked = sorted(zip(candidate_docs, rerank_scores), key=lambda x: -x[1])

# STAGE 3: Take top-5, pass to LLM
top_5 = [doc for doc, score in reranked[:5]]
answer = llm.generate(query, context=top_5)

Total added latency: ~500ms at 100 candidates. Total quality improvement: 15-30% recall in production benchmarks. This is the single highest-ROI change you can make to a working RAG system.

The trade-off matrix

The right stopping point depends on your use case. Customer-support chatbot: rerank. Legal research assistant: multi-hop. Codebase Q&A: hybrid without rerank (code has exact-match signal).

When to skip reranking

Yes, there's a case:
- Corpus is small (<10K docs) — vector search gets everything. Reranking is pure latency cost.
- Query is highly specific — "when did Q3 2023 revenue drop?" already matches the right doc as #1.
- Latency budget is tight (<500ms end-to-end) — reranking eats 500ms.
- You have another quality-boosting technique in your pipeline — for example, if you're using query rewriting + multi-hop, adding rerank may be redundant.

Common failure modes

1. RRF k too low — position-1 dominates, so a bad top-1 in either list ruins the merge. Tune k=60 first.
2. Reranking with a stale embedding model — the reranker was trained on modern embeddings. If your embeddings are 2 years old, mismatch causes weird orderings.
3. Cross-encoder not aligned with domain — an English reranker on medical text is 5-10 pts worse than an in-domain fine-tune. Consider fine-tuning if your domain is specialized.
4. Latency amplification — reranking 500 candidates instead of 100 = 5x latency. Keep candidate set small.

The interview soundbite

"Pure vector RAG misses ~30% of good matches on exact terminology, numbers, and rare tokens. Fix: hybrid retrieval — vector + BM25 merged with reciprocal rank fusion — plus a cross-encoder reranker as stage 2. Adds ~500ms per query for 15-30% recall improvement. This is the single highest-ROI change you can make to a working RAG. Use BGE-Reranker-Base for self-host or Cohere Rerank v3 for hosted."

Prerequisites for Chapter 7

Chapter 7 (evaluation) builds on this. Once you've added reranking, you need a way to MEASURE the recall improvement. That means eval harnesses, ground-truth question sets, and metrics like MRR / nDCG / RAGAS scores. See you in Ch 7.

You added reranking (Ch 6). Users report it's better. But is it? By how much? What if you regress a metric silently on your next deploy?

RAG evaluation is where 90% of teams fail. They ship changes based on "feels better" and 3 months later realize their bot has been hallucinating in production. The eval harness is the LAST line of defense between your code changes and your users' trust.

The 4 dimensions of RAG quality

Every RAG failure lives in one of these 4 buckets:

1. Retrieval quality — did we FIND the right context?
2. Faithfulness — does the answer stick to the retrieved context (no hallucination)?
3. Answer relevance — does the answer address the question?
4. Context precision — is the retrieved context USEFUL or just topical noise?

You need to measure all 4. Metrics that measure only one lie to you.

The metric zoo

Let's decode the alphabet soup:

Retrieval metrics (offline, needs ground truth)

• Recall@K — of the docs a human labeled as relevant, what fraction did we retrieve in our top-K? Higher = we didn't miss good docs. Simple + universally understood.
• MRR (Mean Reciprocal Rank) — 1/rank_of_first_relevant_doc, averaged over queries. Rewards putting good docs at the TOP, not just in the top-K. MRR of 1.0 = perfect (relevant doc always at position 1).
• nDCG@K (Normalized Discounted Cumulative Gain) — weighted-by-position sum of relevance scores, normalized. Handles graded relevance ("very useful" vs "somewhat useful"). Standard in search.

End-to-end metrics (RAGAS-style, LLM-as-judge)

• Faithfulness — is EVERY factual claim in the answer traceable to the retrieved context? Prompts an LLM to break the answer into claims + check each against context. Score 0-1. Low = hallucination.
• Answer relevance — does the answer address the question? Prompts an LLM to generate hypothetical questions from the answer + check similarity to original. Low = off-topic.
• Context precision — of the retrieved chunks, what fraction were actually useful for the answer? Measures wasted context tokens.
• Context recall — of the ground-truth answer's facts, what fraction were IN the retrieved context? Measures retrieval gaps.

Latency + cost (production observability)

• P50 / P95 / P99 end-to-end latency — always measure. Median is a bad summary — P99 is where users churn.
• $/query — sum of embed + retrieve + rerank + LLM costs. Ties directly to unit economics.
• Cache hit rate — for embeddings + LLM responses. Critical at scale.

The eval infrastructure — 3 tiers

Tier 1: Golden Question Set (mandatory, day 1)

50-200 labeled questions covering:
- Common queries (80% of prod volume)
- Long-tail queries (20%)
- Edge cases (numeric, negation, comparative, multi-hop)
- Known-hard queries from user complaints

Each entry: { question, ground_truth_answer, ground_truth_context_ids, difficulty }

Where do the labels come from? Domain experts spend a day labeling. Do NOT skip this step. Auto-generated evals from LLMs give you a plausible number that doesn't reflect reality.

Tier 2: LLM-as-Judge (scalable, RAGAS-style)

For the questions that don't have expensive human labels, use an LLM (typically GPT-4o) as a judge. Prompt:

```
Given a question, a candidate answer, and the retrieved context,
score the candidate answer on 4 dimensions from 0 to 5:
- Faithfulness (does it stick to context?)
- Relevance (does it address the question?)
- Completeness (does it cover the answer?)
- Fluency (is it well-written?)

Return JSON: { "faithfulness": ..., "relevance": ..., ... }
```

RAGAS is the popular open-source implementation (github.com/explodinggradients/ragas). Ships with faithfulness + answer_relevance + context_precision + context_recall metrics. Uses GPT-4o by default. Cost: ~$0.02 per question evaluated.

LLM-as-judge caveats:
- LLMs favor longer, more confident answers even when wrong. Correct with baseline comparisons.
- Same LLM used for generation + judging is biased. Use a DIFFERENT model as judge (e.g., generate with GPT-4o-mini, judge with Claude Opus).
- Judge instability: run 3 samples per question and average. Increases eval cost 3× but reduces noise.

Tier 3: Real user feedback (production, ongoing)

• Thumbs up/down on every answer. Simplest signal, works well.
• "Did this answer your question?" binary + optional freeform. Higher-quality signal.
• Session-level metrics: did the user rephrase after this answer? Did they leave the product? Late signals but powerful.

The evaluation loop

Every model + prompt + retrieval config change goes through this:

```
1. RUN golden question set through the new config
2. COMPARE per-metric to previous run (regression check)
3. If any metric drops by >5%: AUTO-FAIL the deploy
4. If all metrics improve or stay flat: proceed
5. DEPLOY to canary (5% of traffic)
6. MEASURE production feedback for 24h
7. Full rollout OR rollback based on production metrics
```

This turns RAG from "vibes-based deploys" into "measured deploys." Every senior AI engineer runs some version of this loop.

Common eval mistakes

1. Optimizing recall alone. High recall + low faithfulness = well-retrieved hallucinations. Balance metrics.
2. Evaluating only on "easy" questions. Your golden set should include the ones that BREAK your bot. Hard questions expose bugs.
3. No baseline. "Our accuracy is 87%" means nothing without "baseline: 82%" or "previous version: 89%".
4. Judging with the same LLM you generate with. Biased. Use a different family (OpenAI vs Anthropic vs Gemini).
5. Ignoring long-tail queries. 80% of your prod volume comes from 20% of query patterns. But long-tail is where users feel the failures.
6. No canary. Ship changes to 100% of users at once and pray. Recipe for a bad Friday.

RAGAS implementation snippet

python
from ragas import evaluate
from ragas.metrics import (
    faithfulness,
    answer_relevancy,
    context_precision,
    context_recall,
)
from datasets import Dataset

# Prepare your eval dataset
data = {
    "question": ["What is X?", "How does Y work?"],
    "answer": ["X is ...", "Y works by ..."],
    "contexts": [["chunk1", "chunk2"], ["chunk3"]],
    "ground_truth": ["X is ... (label)", "Y works by ... (label)"],
}
dataset = Dataset.from_dict(data)

# Run all 4 metrics
result = evaluate(
    dataset,
    metrics=[faithfulness, answer_relevancy, context_precision, context_recall],
)

print(result)
# { faithfulness: 0.87, answer_relevancy: 0.92, ... }

Cost + latency for a production eval

50 golden questions × 4 RAGAS metrics × 3 samples each = 600 LLM calls per eval run.
At $0.02/call = $12/eval run.
Latency: ~5 minutes end-to-end.
Frequency: run on every PR that touches the RAG stack (or nightly).

The eval dashboard

Every production RAG team has some version of this:

• Time-series graph of the 4 RAGAS metrics + latency + cost across deploys
• Table of "hard" questions — ones the eval regressed on
• Per-tag breakdown — questions from category X trending down = category-specific regression
• User-feedback correlation — do RAGAS scores correlate with thumbs-up? If not, your labels are wrong.

The interview soundbite

"RAG eval has 4 dimensions: retrieval recall, faithfulness, answer relevance, context precision. Measure with RAGAS + LLM-as-judge on a 50-question golden set + ongoing user feedback in prod. Every PR runs the eval; regressions block merge. Cost ~$12/run, 5 min latency. This is what separates hobby RAG from production RAG."

Prerequisites for Chapter 8

Chapter 8 (multi-tenant) assumes you have a working eval pipeline. Multi-tenant RAG is where evaluation gets 10× harder — every tenant has different quality expectations, different domain, different acceptable-failure profile. See you in Ch 8.

You've built a great single-tenant RAG. Now the CTO says: "We're going enterprise. 10,000 companies. Each with their own docs. Complete isolation. Custom retention policies. Per-tenant metrics. Ship in Q3."

Welcome to multi-tenant RAG. This is where 90% of AI startups die. The single-tenant patterns don't scale, don't isolate, and don't attribute costs. Let me show you the 5 patterns that actually work.

The 4 hard problems of multi-tenant RAG

1. Data isolation

Every embedding, every chunk, every retrieval result must be attributable to exactly one tenant. Cross-tenant leaks are:
- A GDPR violation (Article 32 - security of processing)
- A HIPAA violation (technical safeguards)
- A trust-destroying incident (Slack Connect leak: fireable in enterprise sales)

Isolation must be defense-in-depth. Application filtering alone is not enough - one SQL injection and the tenant boundary is gone.

2. Cost attribution

Every LLM call, every embedding, every vector-DB query has a cost. You must know which tenant caused each cost, in real time. Otherwise:
- One noisy tenant burns 90% of your API budget
- You can't invoice fairly (per-usage pricing = table stakes at enterprise)
- Budget alerts fire on aggregate, not per-tenant

3. Per-tenant customization

Different tenants want:
- Different LLM (GPT-4o vs Claude Opus vs open-source)
- Different embedding model (multilingual vs English-only)
- Different retention (30 days vs forever)
- Different search behavior (BM25 weight vs vector weight)
- Different compliance (HIPAA on some, none on others)

Hard-coding tenant behavior is a maintenance nightmare. Config-driven per-tenant policy is the answer.

4. Noisy neighbor

One tenant with 100M docs shouldn't slow down the tenant with 1000 docs.
One tenant sending 1000 QPS shouldn't rate-limit the tenant sending 1 QPS.
One tenant's bad prompt shouldn't spike LLM latency for everyone.

Resource isolation, rate limits per-tenant, workload prioritization - all required.

Pattern 1: Physical isolation (per-tenant database)

Each tenant gets its own vector DB, its own document store, its own embedding index.

When to use: enterprise contracts, HIPAA/FedRAMP compliance, tenants with >$1M/year contract value.

Pros:
- Zero cross-tenant leak risk (physical isolation)
- Per-tenant scaling (each DB sized independently)
- Per-tenant retention policies (drop a DB = wipe a tenant)
- Compliance-friendly (per-region DBs for data residency)

Cons:
- Expensive: minimum $50-200/mo per tenant for infrastructure
- Operationally heavy: 10K tenants = 10K DBs to monitor
- Cold-start cost: new tenant needs full provisioning workflow

Real example: Notion Enterprise, Slack Enterprise Grid, Salesforce Data Cloud. All enterprise-tier products use per-tenant physical isolation for the biggest customers.

Pattern 2: Logical isolation with tenant_id partitioning

One shared vector DB with mandatory tenant_id filter on every query.

When to use: SaaS-tier product with <$50K/year customers, high tenant count (>10K).

Implementation:

python
# EVERY vector search
results = vector_db.search(
    query_embedding,
    filter={"tenant_id": {"$eq": current_tenant_id}},
    top_k=50
)

# EVERY document fetch
docs = doc_store.get_batch(
    doc_ids,
    tenant_id=current_tenant_id  # DB row-level policy enforces this
)

The critical infrastructure:
- Row-level security (RLS) on Postgres: tenant_id policy enforced at DB level
- Vector DB namespaces: Pinecone namespaces, Qdrant collections, Weaviate classes
- Application-tier guard: every request context carries validated tenant_id; every query includes it; unit tests block queries without it

Pros:
- Cost efficient (~$5-15/mo/tenant at scale)
- Easy to provision new tenants (just insert tenant_id)
- Shared operational surface (one DB to monitor)

Cons:
- Higher risk of cross-tenant leaks (bug in filter = data leak)
- Noisy neighbor: one tenant's giant index slows queries for others
- Retention policy per-tenant is manual (need custom purge jobs)

Pattern 3: Hybrid - cells of tenants

Groups of tenants share infrastructure; groups are physically isolated from each other.

When to use: medium-scale (100-10K tenants) with mixed compliance requirements.

Design:
- Group tenants into "cells" of 100-500 tenants each
- Each cell has its own vector DB, LLM API key, application tier
- Cell routing at API gateway based on tenant_id → cell_id mapping

Advantages:
- Compliance-sensitive tenants get their own cell (per-region for GDPR, dedicated cell for HIPAA)
- Noisy neighbors capped at cell size (not global)
- Cost scales as cell_count, not tenant_count

Real example: Salesforce "orgs", AWS "accounts", Google Workspace "domains". All are cell-based isolation.

Pattern 4: Bring-your-own-key + endpoint

Enterprise tenants provide their own LLM API keys and often their own vector DB endpoint.

Design:
- Tenant config table stores API keys (encrypted at rest with per-tenant KMS)
- LLM router: request looks up tenant → tenant.llm_config → makes call with tenant's key
- Same for vector DB, embedding provider, external services

Advantages:
- Perfect cost attribution (tenant's key = tenant's bill)
- Tenant controls their own compliance (their data, their API keys)
- Tenant can pick their own LLM (GPT-4o vs Claude vs on-prem)

Disadvantages:
- Support burden: tenant's API errors are yours to diagnose
- Rate limiting: tenant's rate limits affect their experience
- Feature parity: not all LLMs support all your prompts

Pattern 5: Cost attribution + budget guards

Every request emits a per-tenant cost event.

python
def rag_query(tenant_id, query):
    with cost_tracker(tenant_id) as tracker:
        query_embedding = embed_provider.embed(query)
        tracker.add("embed", cost=0.0001, tokens=len(query))

        vector_results = vector_db.search(query_embedding, tenant_id=tenant_id)
        tracker.add("vector_search", cost=0.0002)

        reranked = reranker.rerank(query, vector_results)
        tracker.add("rerank", cost=0.0005)

        answer = llm.generate(query, context=reranked[:5])
        tracker.add("llm", cost=0.05, input_tokens=..., output_tokens=...)

    # Persist per-tenant, per-minute for real-time cost dashboards
    metrics_db.emit_cost(tenant_id, tracker.total())
    return answer

Every tenant gets:
- Real-time cost dashboard (last hour, day, month)
- Budget alerts (soft warning at 80%, hard cut-off at 100%)
- Cost-per-query metric visible in-app

The multi-tenant eval problem

Chapter 7's eval harness assumes one tenant. In multi-tenant, quality varies drastically per-tenant:
- Medical customer: hard queries, high stakes, needs 95%+ accuracy
- Marketing customer: fuzzy queries, low stakes, 80% accuracy is fine

Solution: per-tenant golden question sets + per-tenant baseline. Every deploy runs eval against EACH tenant's set. Regression per-tenant blocks deploy for THAT tenant only (blue-green per-tenant).

Rate limits + fair queuing

At scale:
- Per-tenant rate limit (Nginx or app-tier token bucket)
- Global rate limit (protect the LLM provider)
- Priority queue: paying tenants > free tier > best-effort

The classic bug: one tenant blows their rate limit; retries with backoff; retries interfere with other tenants' queries. Fix: circuit breaker per-tenant, not global.

The interview soundbite

"Multi-tenant RAG has 4 hard problems: isolation, cost attribution, per-tenant customization, noisy neighbors. Solutions: physical isolation for enterprise (per-tenant DB), logical isolation with tenant_id partitioning for SaaS, cells of 100-500 tenants as a hybrid, BYOK for enterprise. Every request emits per-tenant cost events with real-time budgeting. Eval is per-tenant with per-tenant golden sets. This is where AI SaaS scales - or dies."

The three-tier architecture at 10K tenants

• Free tier (7000 tenants): shared cell, tenant_id-filtered vector search, rate-limited to 100 queries/day, best-effort SLO
• Pro tier (2500 tenants): shared cell with priority queue, dedicated namespace in vector DB, 1000 queries/day, 99.5% SLO
• Enterprise tier (500 tenants): per-tenant cell with dedicated DB, BYOK, unlimited queries, 99.95% SLO with 24/7 support

This gives you 3 different unit economics + 3 different compliance postures with ONE codebase. It's the architecture Notion, Slack, Salesforce, and every enterprise SaaS platform runs.