Every component, deep.

One server can only do so much. When traffic exceeds the capacity of a single machine, or when you need redundancy so a single crash doesn't take you down, you put a load balancer in front and add servers behind it. The LB becomes the single entry point and turns a fleet into what looks like one big machine.

Layer 4 (TCP/UDP) load balancers route packets by connection tuple; they're fast and lightweight but see none of the HTTP semantics. Layer 7 (HTTP) balancers terminate the connection, inspect headers/paths/cookies, and route intelligently — sticky sessions, header-based routing, canary splits, TLS termination. Modern LBs also do health checks (removing dead nodes automatically), connection draining (letting existing requests finish before a node is retired), and rate limiting.

A single LB is horizontally scalable via DNS round-robin, Anycast IP, or a chained LB (L4 in front of L7). Cloud LBs scale automatically; self-managed (Nginx, HAProxy, Envoy) scale via a shared VIP or Anycast. LB throughput is measured in packets per second (L4) or requests per second (L7); a modern Envoy on commodity hardware handles ~50-100K RPS per instance.

• LB itself becomes a single point of failure — mitigate with 2+ LBs behind Anycast or a shared VIP
• Health check misconfiguration marks healthy nodes as dead (or vice versa), cascading to full outage
• Sticky sessions pin traffic to failed nodes; when the node dies, sessions are lost
• Connection storm on LB restart — clients reconnect in a synchronized wave, saturating the new LB

• DNS-based load balancing (round-robin A records) — simplest, but no health checks, TTL-limited failover
• Client-side load balancing (gRPC does this) — no middleman hop, but clients need discovery + retry logic
• Service mesh sidecars (Envoy/Linkerd per pod) — great for east-west traffic; still often need an ingress LB

• AWS Application Load Balancer / Network Load Balancer
• Google Cloud Load Balancing
• Cloudflare (global Anycast + intelligent routing)
• Nginx, HAProxy, Envoy for self-managed deployments

1. L4 vs L7 — when would you use each?
2. How does your LB handle a viral spike that saturates connections?
3. What does a health check look like — active vs passive, and what interval?
4. What happens when the LB itself goes down?
5. How do you drain connections before retiring a node?
6. What breaks first at 100K RPS through a single LB?

In a microservices world, you have dozens of backend services and one thing every request needs to do consistently: authenticate, authorize, rate-limit, log, trace, and route to the right service. If you do these in each service, you end up with N inconsistent implementations of the same logic. An API gateway centralizes cross-cutting concerns at the edge — so services stay focused on business logic.

The gateway sits between clients and services. On every incoming request it: validates the API key or JWT (authN), checks scopes/roles (authZ), enforces per-caller rate limits, does request/response transformation (protocol translation, response shaping), fans out to backend services (aggregation for BFF patterns), and adds observability headers (request ID, tracing context). Modern gateways (Kong, Envoy Gateway, AWS API Gateway) offer plugins so you don't have to fork the code to customize behavior.

Stateless — scales horizontally like any service. Modern gateways handle 10K-100K RPS per instance. The bottleneck is usually the auth check (crypto verification of tokens) and the downstream fanout (if it's aggregating). Latency overhead is typically 1-5ms on cache hits; 20-50ms on JWT verification with an external identity check.

• Gateway becomes a single point of failure — always run 2+ instances behind an LB
• Auth service upstream goes down → gateway can't validate tokens → cache token validation with short TTL as a fallback
• Bad rate-limit config accidentally blocks legitimate traffic — feature-flag rate limits so you can disable per-tier fast
• Plugin/filter chain adds unbounded latency — set per-stage timeouts
• Response aggregation timeout when one downstream is slow — set aggressive per-hop timeouts and return partial results

• Service mesh (Istio, Linkerd) — auth/observability at every hop via sidecars; no central bottleneck but heavier operational model
• Reverse proxy (Nginx, HAProxy) — cheap for routing + TLS, but you'll build auth/rate limiting yourself
• Serverless per-endpoint (Lambda + API Gateway) — great for spiky, uneven load; expensive at sustained scale
• GraphQL federation (Apollo Router) — if your API is GraphQL, the federation gateway subsumes the traditional gateway role

• Netflix Zuul — famous internal gateway that inspired the industry
• Uber Edge Gateway — routes ~10M RPS at peak
• Stripe uses a homegrown gateway for their public API (idempotency + versioning at the edge)
• AWS API Gateway + Lambda authorizers is a common serverless pattern

1. Where do you enforce auth — at the gateway, at each service, or both?
2. How do you handle a global rate limit across 10 gateway instances?
3. Your gateway is at 90% CPU because JWT verification is expensive. What now?
4. How would you canary a new gateway version without breaking clients?
5. The gateway needs a header from an upstream service (e.g., user tier). How do you get it?
6. What happens when a downstream service returns a 5xx to the gateway?

The speed of light is a fixed cost. If your origin is in Virginia and your user is in Sydney, a round trip is 200ms of pure network — before you've done anything useful. CDNs solve this by pushing bytes to hundreds of PoPs around the world; the user's request terminates at the nearest edge, not at your origin. Result: 20ms instead of 200ms, and the origin is protected from 90%+ of traffic.

A user's DNS lookup resolves to a nearby edge PoP (via Anycast or geo-DNS). The edge checks its cache; if hit, it serves the response directly. If miss, it fetches from origin, caches per the response headers (`Cache-Control`, `Surrogate-Control`), and serves. Modern CDNs support edge functions (Cloudflare Workers, Lambda@Edge) so you can run code at the PoP, not just serve files. Purge/invalidate happens via API when content changes.

CDN capacity is measured in bandwidth (Tbps) and cache hit ratio (higher = less origin load). Adding PoPs adds capacity linearly. Hit ratio is a function of cacheability (static: 99%+, dynamic HTML: 30-70%) and cache eviction policy (LRU with configurable TTL). Cost model: pay per GB egress, cheaper as volume grows.

• Cache stampede on eviction — if a hot key expires, N PoPs all fetch from origin simultaneously. Mitigation: origin-shield tier + coalesced fetches.
• Cache poisoning via header manipulation — cache pollution can serve wrong content to millions
• Purge lag — API says purged, but cached copies serve for seconds; mitigation: cache-busting query strings + versioned URLs
• PoP outage routes traffic to farther PoPs, raising latency and origin miss rate

• Reverse proxy at your origin (Varnish, Nginx) — cheap for single-region, no distribution benefit
• Origin-only with heavy Redis cache — works for smaller scale but doesn't help TCP round-trip latency
• Peer-to-peer (BitTorrent-style) — worked for large downloads, mostly irrelevant now

• Cloudflare (largest by PoP count; edge Workers)
• AWS CloudFront (integrated with S3, Lambda@Edge)
• Fastly (fast purge, Varnish-based, Compute@Edge)
• Akamai (oldest, enterprise strength)
• Netflix Open Connect (custom CDN embedded in ISPs)

1. How does the CDN decide which PoP to serve from?
2. What happens when your origin goes down but the CDN still has cache?
3. How would you prevent a cache stampede on a hot key?
4. How would you version cacheable assets so purges are instant?
5. What's your cache hit ratio target for static vs dynamic content, and why?
6. How does the CDN protect you from a DDoS?

For the last 40 years, SQL databases have been the standard answer for storing structured data with strong consistency. When you need to atomically move money from one account to another, when you need referential integrity across tables, when you want ad-hoc queries with JOINs — you reach for SQL. It's the boring, dependable, well-understood choice, and most projects reach for NoSQL prematurely.

A SQL database organizes data into tables with typed columns and enforces constraints (primary keys, foreign keys, unique, NOT NULL). Modern engines use B-tree indexes for point/range queries and MVCC (multi-version concurrency control) so readers don't block writers. Transactions provide ACID guarantees. Under the hood: a page cache in RAM, WAL (write-ahead log) for durability, and a query planner that turns SQL into an execution plan.

A well-tuned Postgres or MySQL primary handles ~10K QPS read and ~1K QPS write on a single m5.2xlarge. Read scaling: read replicas (async replication, ~ms lag). Write scaling: partition tables OR shard across multiple databases (application-level or via Vitess/Citus/Aurora). Storage: single-instance limit ~64 TB on managed services; sharded fleets scale to petabytes.

• Primary DB down → automatic failover to a synchronous replica (~30s outage window)
• Replication lag → read replicas serve stale data; write-then-read from the same session fails
• Long-running transactions → lock contention → cascading queue depth
• Bad query without index → full table scan pegging CPU and blocking short queries
• Connection storm → 'too many connections' errors; mitigate with PgBouncer/ProxySQL
• MVCC bloat → table gets slow over time from dead-tuple accumulation; VACUUM must run

• NoSQL wide-column (Cassandra, DynamoDB) for horizontal write scale with weaker consistency
• Document store (MongoDB) for schemaless flexibility
• NewSQL (Spanner, CockroachDB, TiDB) — SQL API with horizontal scale + strong consistency, at a latency cost
• Time-series DB (InfluxDB, TimescaleDB) for append-heavy metric data

• Stripe uses Postgres for the ledger (money movement)
• GitHub uses MySQL for issues, PRs, and repository metadata
• Airbnb uses MySQL sharded via Vitess
• Notion uses Postgres for document storage

1. You're at 10K writes/sec on a single Postgres primary. What breaks first?
2. How would you shard by user_id, and what queries become hard?
3. How do you handle a schema migration on a 100 GB table with zero downtime?
4. What's the difference between REPEATABLE READ and SERIALIZABLE, and when does it matter?
5. Read replica lag is 30 seconds. What now?
6. You have a 2-column index on (a, b) — will a query on b alone use it? Why or why not?

SQL databases scale vertically well but struggle horizontally at massive write throughput. NoSQL databases were built to trade some of SQL's guarantees (strict schema, cross-row transactions, complex JOINs) for horizontal scalability and specific workload optimizations. They come in four flavors: document (MongoDB), wide-column (Cassandra, DynamoDB), key-value (Redis-style but persistent), and graph (Neo4j) — each optimized for a different access pattern.

Document stores store JSON-like blobs with flexible schemas; queries navigate the document structure. Wide-column stores organize data as row keys → many columns, with each column optionally in a different family (physical disk layout); great for time-series and high-write workloads. Key-value stores are simple hashmaps at planetary scale. Underneath: LSM trees for write efficiency, consistent hashing for shard placement, and quorum reads/writes for tunable consistency.

Wide-column and KV stores scale linearly by adding nodes; write throughput per node ~10K-100K ops/sec depending on payload. Read amplification (LSM compactions) can spike disk IOPS. Consistency is tunable: strong reads cost latency + coordination; eventual reads are cheap. Storage is often much cheaper per GB than SQL because there's no need for a page cache large enough to fit indexes.

• Hot partitions: a poorly-chosen partition key concentrates traffic on one node — throughput plateaus even though the cluster looks idle
• Silent data loss during quorum-write with insufficient replicas configured
• Read-after-write inconsistency confuses application logic — must design for eventual visibility
• LSM compaction storms cause read latency spikes; needs headroom in disk IOPS
• Schema migration is per-document (or per-row) — takes a long time on large datasets

• SQL DB with sharding (Vitess for MySQL, Citus for Postgres) — SQL semantics with horizontal write scale
• NewSQL (Spanner, CockroachDB) — SQL + horizontal scale, with the consistency trade-off
• Object storage (S3) for very large blob payloads — cheaper than any database
• Search engine (Elasticsearch) for full-text + faceted query patterns

• DynamoDB powers Amazon's shopping cart and orders
• Cassandra runs Netflix's viewing history and Instagram's user feeds
• MongoDB is behind many SaaS products (Notion once, still many startups)
• HBase powers Facebook Messages (historically)

1. Choose a partition key for a chat app that stores 100K messages/sec. What's the risk?
2. Explain quorum reads and writes. How would you tune R and W for a write-heavy vs read-heavy load?
3. Your Cassandra cluster has one node at 90% disk usage while others are at 40%. Diagnosis?
4. How do you do a schema evolution (add a new field) in DynamoDB?
5. What are the trade-offs between DynamoDB provisioned and on-demand capacity?
6. You need strong consistency on a write-then-read from the same client. How do you achieve it in Cassandra?

RAM is 100,000× faster than disk. When your workload is hot-set-bound (a small fraction of your data is 99% of the traffic), moving that hot set into memory drops your read latency from milliseconds to microseconds and takes load off your primary database. Redis is the industry's default in-memory cache — but it's also a Swiss army knife for anything that needs sub-millisecond coordination.

Redis is a single-threaded, in-memory key-value store with pluggable data types (strings, lists, sets, sorted sets, hashes, streams, HyperLogLog, bitmaps, geospatial). Because it's single-threaded, individual commands are atomic — huge win for correctness. Persistence is optional (RDB snapshots + AOF logs). Replication is leader-follower with async replication by default. Redis Cluster shards keys across nodes using hash slots (16384 slots hashed via CRC16).

A single Redis node on modern hardware handles ~100K QPS at sub-millisecond latency (~200 KB payloads). Redis Cluster scales linearly to millions of QPS across nodes. Memory is the ceiling: at ~100 bytes per key, 100M keys need ~10 GB. Cost: memory is the expensive resource; the CPU is barely used.

• Node down → cache miss stampede on the database. Mitigate with circuit breakers + load shedding.
• Hot key on a single node → CPU pegged even though the cluster is fine. Mitigate with local in-process caching or key splitting.
• Big-key writes block the single-threaded server. Mitigate with SCAN + chunked writes.
• Persistence disabled + power loss = full cache loss on restart. Mitigate with warm-up policy or AOF.
• Redis Cluster resharding is disruptive on massive datasets — plan capacity in advance.

• Memcached — simpler, multi-threaded, no persistence, no advanced data types. Great for pure hot-set caching.
• In-process LRU (Guava, Caffeine, node-lru-cache) — zero-hop but shared across replicas; adequate for smaller scale
• DynamoDB DAX / ElastiCache — managed alternatives with less operational overhead
• Local NVMe SSD with mmap — for cache sets larger than RAM but too small for a cluster

• Twitter uses Redis for the timeline cache
• GitHub uses Redis for job queues and session storage
• Instagram uses Redis for the feed hot cache
• Discord uses Redis (and Elixir) for presence and pub/sub

1. You have a hot key that saturates one Redis shard — how do you handle it?
2. How does Redis Cluster distribute keys, and what happens when you add a node?
3. What are the trade-offs of Redis persistence (RDB vs AOF vs both vs none)?
4. How would you build a rate limiter in Redis for 1M users?
5. How does Redlock work, and what are its known limitations?
6. What happens when Redis fails over — writes, reads, replicas?

Some work doesn't need to happen in the request/response cycle — sending a welcome email, resizing an image, computing an analytics rollup. If you do it inline, the user waits. If you drop it, you lose reliability. A message queue is the middle path: the producing service enqueues the work and returns immediately; a pool of workers pulls from the queue and processes at their own pace. This decoupling absorbs bursts and lets each side scale independently.

A queue is FIFO by default (with per-message-ACK semantics — the message stays visible until a worker explicitly acknowledges completion). Producers publish messages to a queue; workers poll the queue, process one message at a time, and ACK on success. On failure or timeout, the message becomes visible again for another worker (visibility timeout). Messages that fail N times land in a dead-letter queue for manual inspection. This differs from Kafka: queues do per-message ACK, not offset-based; consumption removes the message, not just advances a pointer.

A single queue can handle 10K-100K messages/sec depending on message size and durability guarantees. Horizontal scale by adding partitions (like Kafka) or by splitting hot queues into topic hierarchies (per-user queues). Worker pools scale independently: 10 workers vs 1000 depending on the depth of the queue and processing time per message. Queue depth is your primary alarm — a growing queue means consumers are behind.

• Worker crashes mid-message: visibility timeout returns the message to the queue; but if the worker committed a side effect (email sent) before crashing, you get a duplicate. Mitigate with idempotent workers.
• Poison message stalls the queue: one bad message keeps failing and blocks workers. Mitigate with DLQ + retry limit.
• Slow consumers create a backlog: queue depth grows, memory pressure at queue broker. Mitigate with autoscaling consumers + circuit breaker on producer.
• Message loss on broker failure if durability isn't configured — always enable replication + fsync for critical work.
• Fan-out amplification: one message triggers 10 downstream messages, each triggers 10 more — quickly overwhelms the queue. Mitigate with backpressure and per-tier rate limits.

• Kafka — for event streaming with replay + higher throughput; different semantics (offset-based, not per-message ACK)
• Cron + database polling — dead simple for low volume; doesn't scale but zero moving parts
• In-process job queue (Celery, Sidekiq) — piggybacks on Redis; great for smaller scale, harder to isolate failure
• Serverless (AWS SQS + Lambda) — managed queue + managed workers; cost-effective for spiky loads

• AWS SQS — the workhorse of async workloads on AWS (trillions of messages/day)
• RabbitMQ — flexible routing (topic/direct/fanout exchanges), popular in enterprise
• GCP Cloud Tasks + Cloud Pub/Sub
• Sidekiq (Redis-backed) is behind Shopify's background jobs

1. Your queue depth is climbing. What are the possible causes and how do you diagnose?
2. A message fails 5 times. Where does it go, and what should you do about it?
3. How would you guarantee exactly-once processing for a queue where each message triggers a payment?
4. You're at 100K messages/sec on a single queue. When do you start sharding?
5. How do you handle a message that requires a service that's currently down?
6. Design a delayed-execution queue (message becomes visible after 24 hours).

When you want to decouple producers from consumers, absorb bursts, replay history, and fan out the same event to N different downstream systems — traditional message queues (RabbitMQ, SQS) run out of gas. Kafka reframes the problem as an ordered, durable, replayable log. It's the backbone of most modern event-driven architectures.

Kafka stores messages in topics, which are split into partitions. Each partition is an append-only log on disk (fast sequential writes). Producers append to a partition (chosen by key hash for ordering per key); consumers read from an offset they track themselves. Replication is via leader-follower per partition. Consumers organize into groups; each partition is consumed by exactly one consumer in the group, providing parallelism. Retention is time- or size-based, not consumption-based — messages stay until they age out.

A single Kafka broker handles ~100 MB/s per partition on commodity SSD, scaling to ~1 GB/s per broker with tuning. Partitions are the unit of scale: 100 partitions × 4 brokers gives ~400 MB/s throughput. Latency is ~5-10ms end-to-end for producers with acks=all. Storage is cheap — Kafka is often used for months of history.

• Consumer lag grows unbounded → hot topic backs up. Mitigate with backpressure + scaling consumers.
• Partition rebalance on consumer join/leave causes brief processing pause. Mitigate with cooperative rebalancing.
• Broker down → partition replicas re-elect leader (~30s). Messages may need re-send from producer.
• Bad message poisons a consumer → all consumers on that partition stall. Mitigate with a dead-letter topic.
• Producer retries + acks=all can cause duplicates. Mitigate with idempotent producer (enable.idempotence).
• Under-replicated partitions during a broker outage risk data loss if another broker fails.

• RabbitMQ / ActiveMQ — traditional queues, per-message ACK, no replay
• SQS — managed, simple, no ordering guarantees on standard queues
• AWS Kinesis — Kafka-like managed offering
• Redpanda / Pulsar — Kafka-compatible or Kafka-like with different trade-offs
• Pub/Sub (Google) — managed pub-sub, at-least-once, no partition ordering

• LinkedIn built Kafka; it processes trillions of messages/day
• Netflix uses Kafka for the entire event backbone (Keystone pipeline)
• Uber uses Kafka for dispatch and analytics
• Airbnb uses Kafka for CDC out of MySQL

1. How does Kafka guarantee ordering, and when is that guarantee violated?
2. What's a consumer group, and what happens when a consumer joins?
3. How do you deliver exactly-once with Kafka? (Hint: it's tricky.)
4. Your consumer lag is 10 minutes and growing. Walk me through your debugging.
5. How would you use Kafka for CDC from a Postgres primary?
6. Producer with acks=all + idempotence — what does that guarantee, and what does it cost?
7. What if you need cross-partition ordering (e.g., a global sequence number)?

Once you have a Kafka pipeline, you often need to compute things over that stream: 'count events per user per 5-minute window', 'join clicks with impressions', 'detect fraud in real time'. A batch job runs every N hours; a stream processor runs continuously with sub-second latency. This is what turns event streams into actionable data products.

The processor reads events from one or more sources (Kafka topics), applies operators (map, filter, group-by, window, join, aggregate), and writes results to sinks (Kafka topics, databases, dashboards). State (like running counts) is stored in a local key-value store (RocksDB embedded); checkpointed to durable storage so recovery is exactly-once. Windows can be tumbling (fixed intervals), sliding (overlapping), or session (activity-based). Watermarks handle late-arriving events.

Throughput scales with parallelism (partition count). Flink handles millions of events/sec per cluster; Kafka Streams runs embedded in your JVM app. State size can be tens of GB per operator instance. Latency is typically ~100ms end-to-end for simple pipelines; complex joins can push to seconds.

• Checkpoint failures on slow storage cause processing to stall — monitor checkpoint duration
• State grows unbounded (missing TTL on aggregations) — RocksDB gets huge and compaction dominates CPU
• Watermark stalls when one partition has slow events — computation waits for the laggard
• Backpressure from a slow sink cascades upstream — always monitor source lag
• Rebalance during a scale-up moves state across nodes — brief pause in processing

• Kafka Streams — embedded in your JVM app; simplest deployment; state lives with your app
• Spark Structured Streaming — great if you already have Spark; higher latency (~seconds)
• Materialize / RisingWave — SQL-first streaming DBs; easier ergonomics for SQL teams
• Custom Kafka consumer with in-app aggregation — good for simple cases; complex windowing is painful
• Batch job every N minutes — pragmatic when 'real-time' can mean 5 minutes

• Uber uses Flink for their real-time surge pricing pipeline
• Netflix uses Flink for the Keystone real-time analytics pipeline
• LinkedIn built Samza (predecessor to Kafka Streams)
• Alibaba uses Blink (their fork of Flink) for real-time e-commerce features

1. Explain the difference between tumbling and sliding windows. Give an example use case for each.
2. Your stream processor's state is 500 GB and growing. How do you diagnose and fix?
3. How do watermarks work, and what happens if events arrive out of order?
4. Compare exactly-once semantics in Kafka Streams vs Flink.
5. Design a fraud-detection feature: 'flag any card that had a login in one country and a purchase in another within 5 minutes.' What operators?
6. Your stream processor is doing 100ms latency; PM wants 10ms. What levers do you have?

Storing large blobs (>1MB — images, videos, PDFs, backups) in a database is a bad idea: databases are optimized for structured queries and page-cache-fitting rows, not for streaming megabytes. Object storage is purpose-built for exactly this: cheap disk-backed storage with an HTTP API, eleven 9s of durability (S3 default), and a flat namespace (no filesystem hierarchy). It's the default answer for 'where do I store user uploads?'

Objects are stored as key → blob + metadata in buckets. There's no directory structure — 'folders' are just prefixes in the key (e.g., users/123/avatar.jpg). Data is replicated across ≥3 availability zones. Access is via HTTP GET/PUT/DELETE. Consistency is strong per-object (S3 as of 2020) — a PUT is visible to subsequent GETs immediately. There's no partial updates: you always overwrite the whole object or upload a new version.

Effectively infinite storage and throughput per bucket (S3 auto-partitions internally). Practical limits: ~5,500 GETs/sec/prefix and ~3,500 PUTs/sec/prefix before AWS starts throttling — mitigate with random prefixes for high-write workloads. Latency is 20-200ms per request depending on size. Bandwidth is cheap; egress across regions is expensive.

• Bucket misconfiguration exposes data publicly — S3 default is private, but IAM policies can leak. Mitigate with default-deny + linters + org SCP.
• Hot prefix throttling: too many requests hit one auto-partition. Mitigate with random-hash prefixes.
• Cost surprise: cross-region egress or Glacier retrieval bills you didn't plan for. Set billing alarms.
• Multi-object updates aren't atomic. If you write image + metadata separately, they can diverge on failure.
• Data corruption on partial write is not automatically detected — always verify ETag / checksum.

• Filesystem (NFS, EFS) — POSIX semantics, works for legacy apps; expensive at scale
• Block storage (EBS) — for databases and stateful workloads that need low-latency random IO
• Database BLOB columns — works for <10MB files; usually a mistake at scale
• Peer-to-peer / IPFS — for content-addressed storage; niche use cases

• S3 is the industry default — Netflix, Airbnb, Pinterest all use it heavily
• Instagram stores user photos on Facebook's Haystack (internal object store)
• Dropbox built and later replaced their own Magic Pocket to replace S3
• Backblaze B2 — cheapest per-GB object storage

1. How do you design a file-upload service? Where does the file live? What's the write flow?
2. You need to serve 1TB/sec of video. Where does the video live, and how do you deliver it?
3. How would you migrate from local filesystem to S3 with zero downtime?
4. Explain multi-part upload. When do you use it, and what's the failure story?
5. How do you prevent your app from becoming a proxy for object downloads? (Hint: presigned URLs)
6. You have 100M objects in one prefix. Reads are getting throttled. What's the fix?

SQL LIKE '%foo%' is O(N) — a full table scan. At any real scale, full-text search on a relational database is unusable. Search engines flip the model: they build an inverted index (word → list of document IDs containing it) so queries are O(matches), not O(rows). They also handle relevance scoring, faceted filtering, fuzzy matching, and aggregation in a way that's painful in SQL.

At index time, each document is analyzed: tokenized into terms, lowercased, stemmed, and possibly filtered (stopwords). Each term is added to the inverted index pointing to the doc ID. At query time, the search engine tokenizes the query the same way, retrieves candidate docs from the index, scores them (BM25 by default), and returns the top-K. Sharding splits the index across nodes; queries fan out and results are merged. Elasticsearch uses Lucene under the hood.

A single Elasticsearch node handles ~10K QPS for typical queries; complex aggregations are 10-100× slower. Sharding scales writes and index size linearly. Replication provides read scale + fault tolerance. Index size dominates memory: keep the working set in RAM for fast queries. Rule of thumb: 20-30 GB per shard, ~1 shard per CPU core.

• Split-brain during network partition (Elasticsearch quorum matters) — set minimum_master_nodes carefully
• Index corruption during power loss without proper flush settings — always enable translog fsync
• Query overload from complex aggregations — set search timeouts + circuit breakers
• Reindexing during schema changes takes hours to days — plan capacity and use aliases for zero-downtime cutover
• Cluster overload from too many small shards — every shard has overhead; keep them large

• PostgreSQL full-text search (pg_search, tsvector) — good enough for <10M docs; keeps you on one datastore
• Meilisearch / Typesense — simpler, developer-friendly, great for typo-tolerance out of the box
• Algolia — hosted, best-in-class relevance out of the box, expensive at scale
• Vector search (pgvector, Pinecone, Qdrant) — for semantic search, replaces or complements keyword search

• GitHub uses Elasticsearch for code and issue search
• Uber uses Elasticsearch for trip-record search
• Netflix uses Elasticsearch for their internal metadata search
• Slack uses Elasticsearch for message search

1. Explain the difference between an inverted index and a B-tree. When would you prefer which?
2. How do you handle a schema change on a 1TB Elasticsearch index?
3. What are the trade-offs of using Elasticsearch as your source of truth vs an index derived from Postgres?
4. How would you build autocomplete for 100M product names with typo tolerance?
5. Your Elasticsearch cluster is running out of heap. What's your diagnostic sequence?
6. How does BM25 scoring differ from TF-IDF, and when does it matter?

Every service has a capacity ceiling. Without a rate limiter, one runaway client — buggy retry loop, aggressive scraper, DDoS — takes the whole service down. Rate limiting enforces fairness (no one caller consumes disproportionate capacity) and protection (the system stays healthy for the many even when one caller misbehaves). It's a required layer in any public API and many internal ones.

The rate limiter maintains a counter per caller key (API key, IP, user ID). On each request, it increments the counter and checks against the budget. If over, it returns 429 (Too Many Requests). Algorithms vary: fixed window (simplest, has burst issues at window boundaries), sliding window (smoother, more accurate), token bucket (allows bursts up to bucket size, refills at a rate), leaky bucket (smooths bursts to a constant output rate). Distributed rate limiters use Redis with INCR + EXPIRE.

Local (per-instance) rate limiters are cheap but inaccurate at scale — 10 instances each allowing 100 RPS = 1000 RPS actual. Distributed rate limiters (Redis-backed) are accurate but add 1-2ms per request. Modern designs approximate globally with periodic sync: each instance has a local quota, syncing with a central store every N seconds — accurate to within a few percent, near-zero latency.

• Redis outage → local fallback allows unlimited traffic (better than blocking everything). Design for this.
• Legitimate spike gets throttled (e.g., a product launch) — always feature-flag rate limits so you can raise them fast
• Global rate limiter becomes a hot key — shard by hash prefix
• Bucket bookkeeping drifts across instances — use approximate token bucket with periodic sync
• Rate limit response body is expensive to generate — use a static 429 payload

• API Gateway with built-in rate limiting (Kong, AWS API Gateway) — cheapest
• Envoy / service mesh with rate-limit service (envoy-rls) — for internal East-West traffic
• Circuit breaker — different tool, complementary; circuit breaker protects YOU from downstream failures, rate limiter protects downstream FROM you
• Load shedding at the load balancer — coarse but works when you're already overloaded

• Stripe rate-limits API keys per tier — free tier ~100 RPS, higher tiers unlimited
• GitHub rate-limits API by IP + user (visible in x-ratelimit-* headers)
• Twitter (X) uses aggressive per-user rate limits on their public API
• Cloudflare rate-limits at the edge for DDoS protection

1. Compare token bucket, leaky bucket, and sliding window. When would you use each?
2. Design a distributed rate limiter for 1M API keys at 1M RPS. Where's the state?
3. Your rate-limit Redis is down. What happens to traffic?
4. You want to allow bursts up to 100 requests but sustain only 10/sec per user. Which algorithm?
5. How would you implement per-endpoint rate limits (different budgets for different APIs) efficiently?
6. How does rate limiting interact with retries? What can go wrong?

Some work must happen periodically: daily reports, weekly billing runs, hourly cache warmups, monthly cleanup. cron on a single machine works for small scale — but at scale, you need distributed schedulers that survive node failure, guarantee at-least-once (or exactly-once) execution, retry on failure, and give you observability (did the job run? how long? what did it produce?).

A scheduler reads job definitions (schedule + task) and triggers execution when the time arrives. For distributed schedulers, one node holds the 'is scheduler' lease and is the sole trigger (avoids double-fire). Complex schedulers (Airflow, Prefect) support DAGs — jobs that depend on other jobs. Advanced schedulers (Temporal, Cadence) support durable execution: the workflow's state is checkpointed to a database so it survives worker crashes and can wait days for a subsequent step.

Simple schedulers (cron on one box) handle any single-node workload; the limit is worker fan-out, not the scheduler itself. Distributed schedulers scale by adding workers, keeping the scheduler thin. Airflow can trigger millions of jobs/day; Temporal handles millions of long-running workflows concurrently. Cost: scheduler state (DB) is small; worker pools scale linearly with job load.

• Double execution: two nodes both think they're the scheduler. Mitigate with distributed lock + heartbeat.
• Missed execution: scheduler node was down when the trigger time passed. Mitigate with catch-up mode (Airflow) or run-once semantics.
• Long-running job blocks the next scheduled run — always set a timeout + concurrency limit per job
• Job depends on data that isn't ready yet — Airflow has sensors; Temporal has waitFor
• Cron drift on a single machine (system clock issues) causes silent misfires

• cron on a single box — simplest for tiny scale
• Kubernetes CronJob — cron with pod isolation and retry; great for containerized workloads
• Airflow — for DAG-based ETL; heavy but powerful
• Temporal — for long-running, stateful workflows with retries built in
• AWS EventBridge / GCP Cloud Scheduler — managed cron for cloud native

• Airbnb built Airflow for data pipelines
• Uber built Cadence (now open-sourced as Temporal); powers billions of workflow executions
• GitHub Actions cron triggers — used for scheduled CI
• Stripe uses Temporal for money movement workflows (payment settlement, refunds)

1. How do you prevent a scheduled job from running twice when you have 3 scheduler nodes?
2. A daily job takes 26 hours some days and overlaps with the next run. What's your fix?
3. Design a system to send 'follow-up in 24 hours' emails. Where does the scheduling state live?
4. How would you retry a failed job — immediately, with backoff, or from a specific checkpoint?
5. Explain how Airflow handles a DAG where one task fails.
6. What's the difference between an idempotent job and one that just retries safely?

Some processes are hard: 'Charge the card, then update the shipping order, then send a confirmation email — and if any step fails, roll back the previous ones.' Doing this in application code means writing retry logic, compensation logic, state persistence, timeout handling, and observability yourself — every time. Workflow engines factor all of this out. You write the business logic; the engine handles durability, retries, and compensation.

A workflow is defined as code (Temporal, Cadence) or as a JSON state machine (Step Functions). Each step is an activity — a call to a service or a piece of business logic. The engine persists the workflow's state after every step. On worker failure, the engine reruns the workflow from the last checkpoint. Because activities may re-execute, they must be idempotent. Workflows can wait for external signals (user confirmation), sleep for days, and roll back via explicit compensation actions.

Temporal / Cadence handle millions of concurrent workflows in production; state size per workflow is bounded (~100 KB) but total state can be terabytes. Latency per step is a few ms of engine overhead + the activity itself. Throughput scales with worker fleet + engine cluster; typical clusters run 100K workflow starts/sec.

• Non-idempotent activities cause corruption on retry — always design activities to be idempotent
• Workflow history grows unbounded on long-running workflows — use continueAsNew to reset
• Poison workflow: a bug in the code causes infinite retry — set retry limits + alerts
• Engine cluster overloaded from too many workflows — capacity plan carefully
• Debugging is harder than synchronous code — invest in observability early

• Choreographed sagas via events — no central engine; more moving parts to reason about
• A queue with retry — good for 2-step workflows; falls apart at 5+ steps
• Airflow — better for scheduled DAGs (batch); worse for event-triggered long-running flows
• Custom code + database state — reinventing the wheel; usually a mistake at any scale

• Uber built Cadence (now Temporal) — used for millions of workflows/day
• Stripe uses Temporal for payment settlement workflows
• Snap uses Temporal for user onboarding orchestration
• AWS Step Functions is behind many serverless architectures

1. Explain the Saga pattern. When would you use a workflow engine to implement it vs. choreographed events?
2. How do you handle a workflow that fails halfway through — resume from where it stopped or start over?
3. Design a subscription-renewal workflow that runs monthly, retries on payment failure, and grace-periods before cancellation.
4. What does 'idempotent activity' mean in Temporal, and why is it critical?
5. Your workflow started before you deployed a code change. What happens when it resumes?
6. How would you monitor the health of 10M concurrent workflows?

Some things must happen exactly once across a fleet: 'only one scheduler triggers the nightly job', 'only one worker processes this order', 'only one leader accepts writes'. On a single machine this is a mutex. Across N machines, you need a distributed lock — a coordination primitive backed by a consensus system (or clever cache tricks). It's one of the hardest primitives in distributed systems, and one of the easiest to get subtly wrong.

The lock is a key in a distributed store; acquiring the lock means atomically writing your ID with a TTL if no one else holds it. Consensus-based locks (ZooKeeper, etcd) use Paxos/Raft — a lock is a znode/kv with a lease; the client renews the lease periodically; if the client dies, the lease expires and the lock is available. Redis Redlock uses a majority-vote across N independent Redis nodes with a short TTL; the algorithm has known issues but is fast. Fencing tokens (monotonic numbers issued at lock acquisition) prevent 'zombie' locks from corrupting downstream state.

Locks are inherently sequential — throughput is bounded by lock acquire+release rate. ZooKeeper handles ~10K lock ops/sec per cluster; etcd is similar. Redis Redlock is faster (~50K/sec) but with weaker guarantees. Lock contention (many nodes wanting the same lock) kills throughput; design to minimize contention.

• Split-brain: two nodes both think they hold the lock (e.g., holder's GC pause exceeds TTL). Mitigate with fencing tokens.
• Lock service outage stops all coordination. Design for graceful degradation.
• TTL too short → holder loses lock while still working. Too long → dead holder blocks progress.
• Redis Redlock consistency issues under clock skew are famously debated (Martin Kleppmann vs antirez).
• Watchdog process fails to renew lease → holder loses lock silently.

• Optimistic concurrency (CAS) at the database — often what you actually want
• Partition-and-process (each node owns its shards; no lock needed) — the right answer if you can arrange it
• Kafka consumer groups — implicit leader election for partition assignment
• Cloud-native leader election (Kubernetes lease objects) — clean if you're on K8s

• ZooKeeper is behind Kafka (broker coordination), HBase, Solr
• etcd runs Kubernetes control plane, CoreOS, and countless service discovery use cases
• Redlock is used by many startups; hotly debated for correctness
• AWS DynamoDB conditional writes are the 'poor man's distributed lock' — reliable and simple

1. Explain why a naive Redis lock (SETNX with TTL) can allow two clients to think they hold the lock. Fix?
2. What's a fencing token, and how does it prevent zombie locks?
3. Compare ZooKeeper and etcd for a distributed lock use case.
4. Design a distributed cron system where each of 10 nodes might try to run the job.
5. Your lock holder is GC-paused for 30 seconds. The TTL is 20 seconds. What happens, and what's the fix?
6. When would you use optimistic concurrency instead of a distributed lock?

In a static environment, service-A calls service-B at 10.0.0.42:8080 — configured once. In a dynamic environment (containers, autoscaling, rolling deploys), IPs change constantly. A service discovery layer solves the question: 'what are the current healthy instances of service-B?' — with automatic registration on startup, deregistration on shutdown, and health checking. It's the plumbing that makes microservices possible.

Services self-register on startup by writing (name, IP, port) into the discovery store. Clients query the store to find current instances. A health check either polls each instance or receives heartbeats; unhealthy instances are removed. Some systems (Kubernetes) use DNS as the front — you look up 'service-b.default.svc.cluster.local' and get a list of current IPs. Others (Consul, etcd) expose a richer API for service metadata + tags.

Discovery stores are read-heavy — millions of queries/sec is common. Write load (register/deregister/heartbeat) is bounded by fleet size. DNS-based discovery scales trivially via caching (TTL trade-off vs freshness). Consul/etcd scale to thousands of services; beyond that you shard by domain.

• Stale registrations: a dead service is still returned to clients. Mitigate with aggressive health checks + short TTLs.
• Split-brain in Consul during partition — some clients see one set of services, others another
• Cascading heartbeat failures: discovery store overload causes it to mark healthy nodes as dead
• DNS TTL mismatch: clients cache old IPs for minutes, seeing 'dead' services after failover
• Client-side caching without invalidation causes routing to gone instances

• Kubernetes Service + DNS — batteries-included for K8s workloads
• Client-side load balancing with a service registry (Ribbon, gRPC name resolvers)
• Envoy service mesh with xDS — dynamic config push from a control plane
• Hardcoded LB + rolling deploys — works for stable environments; falls apart with autoscale

• Consul is widely used at HashiCorp customers + Airbnb
• etcd runs the Kubernetes control plane
• Netflix Eureka is a JVM-centric discovery service
• Google's internal Borg uses a proprietary discovery service that inspired Kubernetes

1. Compare DNS-based service discovery with a purpose-built registry (Consul/etcd). Pros and cons?
2. Your service dies but stays registered for 30s. What's the client-side impact and mitigation?
3. How does Kubernetes service discovery work under the hood (Endpoints, kube-proxy)?
4. Design a multi-region service discovery that keeps regional latency low.
5. What happens if the service discovery cluster itself goes down?
6. How would you route traffic to services during a rolling deployment without dropping requests?

A single-node database has a ceiling: RAM for the working set, CPU for query planning, network for connections, disk for durability. At extreme scale (petabytes, millions of RPS, global users), no single node is big enough. Distributed databases spread data across dozens or thousands of nodes with automatic sharding, replication, failover, and (in some cases) global consistency. They pay for this with operational complexity and (often) weaker consistency semantics.

Data is partitioned across nodes by a partition key (usually hashed via consistent hashing). Each partition is replicated (typically 3× within a region, plus cross-region for global systems). Reads and writes go to any replica; quorum protocols (R + W > N) tune consistency vs availability. Some systems (DynamoDB, Cassandra) are leaderless — any node can accept a write and gossip it. Others (Spanner, CockroachDB) are leader-per-partition using Raft for consensus, giving stronger consistency at the cost of more coordination.

Linear horizontal scale: add nodes, get more capacity. DynamoDB serves trillions of requests/day. Cassandra clusters handle millions of writes/sec. Latency: single-digit ms for local reads/writes; tens of ms cross-region. Storage is essentially unlimited (petabytes). Cost scales with capacity — pay per RCU/WCU on DynamoDB, per node on Cassandra.

• Hot partition: bad key choice causes one node to be pegged while others idle
• Silent tombstone accumulation in Cassandra (deletes that never actually delete) — hurts read latency
• Rebalancing during scale-up disrupts throughput temporarily
• Eventual consistency confuses application logic — write, read from another node, get old value
• Cross-region replication lag causes stale reads during regional failover
• Cost overrun from misconfigured provisioned capacity (DynamoDB)

• Sharded SQL (Vitess, Citus) — familiar SQL semantics + horizontal scale, more operational care needed
• NewSQL (Spanner, CockroachDB, TiDB) — SQL + horizontal scale + global consistency (at latency cost)
• Object storage + database index — for very large blobs; S3 + Postgres index is a common combo
• Time-series DB (InfluxDB, Timescale) for time-series workloads specifically

• Amazon runs DynamoDB internally for the shopping cart
• Netflix stores viewing history on Cassandra
• Google runs Spanner globally for Ads, YouTube, Photos
• Apple runs FoundationDB (~massive Cassandra-like) internally

1. Compare DynamoDB and Cassandra. When would you pick each?
2. Design a partition key for a chat system. What's the risk with a bad choice?
3. Explain quorum reads and writes. When would you use strong quorum vs local quorum?
4. How does Spanner achieve external consistency? What's TrueTime?
5. Your DynamoDB cost tripled last month. Common causes and how to diagnose?
6. How would you migrate 100 TB from Postgres to Cassandra with zero downtime?