1:1 and group messaging at scale — WebSocket + MQTT, E2E encryption, multi-device sync.
The scenario
3B MAU messaging — where Erlang, 2M-connections-per-server, and end-to-end encryption converge
Same startup, same engineer #4. Sixth Monday.
Your CTO drops by. "Users love everything you've built. We want to add private 1:1 chat. Real-time. Reliable. End-to-end encrypted so we NEVER see the message content. Ship an MVP in 12 weeks."
She pauses. "For context — WhatsApp has 3 billion monthly active users on 500-ish engineers total. They serve 100 billion messages a day. Half of humanity uses this thing. If we're going to compete, we need to understand why their architecture is fundamentally different from Slack."
That's the whole game.
Here's the paradigm shift:
Slack was team chat — bounded fanout (~100 members per channel), rich features (threads, reactions, search, integrations), enterprise controls.
WhatsApp is personal messaging — 1:1 dominant, small groups (up to 1,024 in 2024), E2E encrypted so the server literally cannot read messages, and the same infrastructure serves both a teenager in Brazil and a small business in Bangladesh. The design pressures are completely different.
The famous 2014 fact: WhatsApp reached 450M users with 32 engineers. Reference: Facebook acquisition disclosure, 2014. How? They picked Erlang — a language designed at Ericsson in 1986 for telephone switches — and got 2 million concurrent WebSocket-like connections on a single server. Reference: Rick Reed's talk "The WhatsApp Architecture" at Erlang Factory 2014. If your interview answer doesn't mention Erlang, you'll fail.
The real 2024 numbers
- ~3B MAU (Meta Q4 2024 earnings)
- ~100B messages/day at peak (Meta 2024 events)
- ~2M concurrent connections per server (2014 Rick Reed; still the reference number)
- ~1TB+ Erlang binary distribution across all servers globally
- Signal Protocol (X3DH + Double Ratchet) for E2E — cannot decrypt messages server-side
- ~1,024 members max group size (upped from 512 in 2024)
- ~100M voice/video calls/day — WebRTC-based, also E2E encrypted
Interview soundbite: "WhatsApp is the counter-example to microservices maximalism. 32 engineers ran 450M users. The core message-router is still a small Erlang codebase. The lesson: language choice + protocol choice matters more than architecture buzzwords."
The whole journey at a glance
Every 10× in user scale surfaces a different bottleneck:
text═══════════ WHATSAPP ARCHITECTURE ACROSS 4 SCALES ═══════════ L4 (10K users) L5 (10M users) L6 (500M users) L7 (3B MAU, global) Node.js + Postgres Erlang router Erlang cluster · Signal 200+ regions · 2M conn/srv 12 weeks · $500/mo 6 months · $50K/mo 18 months · $500K/mo ongoing · $500M/yr infra ┌────────┐ ┌────────┐ ┌── Anycast IPs ─────┐ ┌── Meta backbone + XLB ┐ │ Web │ │ Web │ │ region routing │ │ 200+ regions edge │ │Mobile │ │Mobile │ └─┬──┬──┬──┬─────────┘ │ BGP anycast │ │Desktop │ │Desktop │ │ │ │ │ └──┬──┬──┬──┬──────────┘ └───┬────┘ └───┬────┘ ┌─▼──▼──▼──▼─────┐ │ │ │ │ │ │ │ Regional ALB │ ┌───▼──▼──▼──▼──────────┐ ┌──▼───┐ ┌──▼──┐ │ WebSocket/ │ │ chatd (Erlang) │ │ ALB │ │ ALB │ │ MQTT-like │ │ 2M concurrent conn/srv │ │ │ │ WSS │ │ proto (custom │ │ ~50K servers globally │ └──┬───┘ └──┬──┘ │ by WhatsApp) │ └──┬──────┬──────┬───────┘ │ │ └───┬──────┬─────┘ │ │ │ ┌──▼──────┐ ┌───▼─────────┐ ┌───▼──────▼───┐ ┌──▼──────▼──────▼──────┐ │ Node.js │ │ Erlang │ │ Erlang │ │ Signal Protocol layer │ │ / Go │ │ router │ │ message │ │ X3DH key agreement │ │ API + WS│ │ ejabberd │ │ router │ │ Double Ratchet msgs │ │ 5K conn │ │ variant │ │ clustered │ │ Client-side only │ └──┬──────┘ │ 40K conn/nod│ │ Mnesia (memory) └──┬────────────────────┘ │ └───┬─────────┘ └───┬──────────┘ │ │ ┌───▼──┐ ┌──▼───────────┐ ┌───▼──────────────────┐ │ │Redis │ │Cassandra │ │Cassandra (multi-DC) │ │ │offline │offline queue │ │offline queue │ │ │queue │ │+ presence │ │+ receipts + presence │ │ └───┬──┘ └───┬──────────┘ └──┬───────────────────┘ │ │ │ │ ┌──▼───┐ ┌───▼─┐ ┌──▼──────────┐ ┌──▼──────────────────┐ │Postgr│ │Postgr│ │FreeBSD-tuned│ │FreeBSD hosts + 1M │ │(users│ │+ shar│ │Erlang hosts │ │Erlang procs each │ │+ groups │d by │ │(Meta's kern-│ │(BEAM VM) │ │+ pending │user │ │el work saved│ │+ Mnesia dist. │ │ msgs) │_id │ │ 30%+ CPU) │ │+ E2E media Signal │ └──────┘ └──────┘ └─────────────┘ └─────────────────────┘ ↑ ↑ ↑ ↑ Boring Erlang enters the Erlang + Signal Proto WhatsApp reality: 2M Node.js works. picture (500M ppl E2E encryption. Multi-DC conn per FreeBSD host. Postgres + used ejabberd at Cassandra offline queue. ~50K servers globally. Redis + WS. Facebook's ask). Half of humanity. Chapter 5 Chapters 6+6.5 Chapter 7+7.5 Chapter 8 walks walks through walks through the walks through Meta through Erlang router + Signal Protocol backbone, Erlang L4 MVP Cassandra offline (X3DH + Double Ratchet) kernel patches, and queue 2M-conn engineering Key insight: WhatsApp works because Erlang's process model = 1 lightweight green-thread per connection. 2 MILLION connections per server. This is 100x what Node.js can do. Language + runtime choice IS the architecture. That's the entire lesson.
The same 4 tiers as clean architecture diagrams
The ASCII compressed all four. Here are the same shapes as Mermaid:
L4 · 10K users · Node.js + Postgres · $500/mo · 12 weeks:
flowchart TD
W([Mobile / Web / Desktop]) -->|WSS| LB[ALB · WebSocket sticky<br/>$25/mo]
LB --> API[1 Node.js server<br/>c5.large · 5K concurrent WS]
API --> RD[(Redis<br/>online-connection registry<br/>presence)]
API --> PG[(Postgres Multi-AZ<br/>users + groups + pending<br/>+ delivery receipts)]
API -->|presigned| S3[(S3<br/>media binaries)]
classDef n fill:#dbeafe,stroke:#2563eb,color:#1e3a8a
class LB,API,RD,PG,S3 nL5 · 10M users · Erlang router + Cassandra · $50K/mo · 6 months:
flowchart TD
W([Global mobile]) -->|WSS| LB[Regional ALB · WSS]
LB --> R{Consistent-hash<br/>router by user_id}
R --> E1[Erlang chatd 1<br/>40K conn/node]
R --> E2[Erlang chatd 2<br/>40K conn/node]
R --> E3[Erlang chatd 3<br/>40K conn/node]
E1 --> M[(Mnesia in-memory<br/>presence + routing table)]
E2 --> M
E3 --> M
E1 --> C[(Cassandra<br/>pending msgs + receipts)]
E2 --> C
E3 --> C
E1 -->|E2E media only| S3[(S3 + Cloudfront<br/>E2E-encrypted media blobs)]
classDef n fill:#dbeafe,stroke:#2563eb,color:#1e3a8a
classDef m fill:#fef3c7,stroke:#d97706,color:#78350f
class LB,R,E1,E2,E3 n
class M,C,S3 mL6 · 500M users · Erlang cluster + Signal Protocol · $500K/mo · 18 months:
flowchart TD
W([Global mobile / WA Web]) -->|WSS + Signal| CDN[Anycast IPs<br/>Meta edge]
CDN --> LB[Regional ALB]
LB --> ER[Erlang chatd cluster<br/>~40K conn/node<br/>clustered via distributed BEAM]
ER --> M[(Mnesia distributed<br/>presence · routing)]
ER --> C[(Cassandra multi-DC<br/>pending queue + receipts)]
W -->|X3DH + Double Ratchet<br/>ALL keys on device| SIG[Signal Protocol<br/>client-side ONLY<br/>server sees encrypted opaque]
ER -->|blob passthrough| S3[(S3 + Cloudfront<br/>E2E ciphertext only)]
classDef n fill:#dbeafe,stroke:#2563eb,color:#1e3a8a
classDef m fill:#fef3c7,stroke:#d97706,color:#78350f
classDef s fill:#dcfce7,stroke:#16a34a,color:#14532d
class CDN,LB,ER n
class M,C,S3 m
class SIG sL7 · 3B MAU · 2M conn/server + Meta backbone · $500M/yr:
flowchart TD
W([Half of humanity]) -->|WSS + Signal| CDN[Meta anycast backbone<br/>200+ edge regions]
CDN --> LB[Regional XLB]
LB --> ER[Erlang chatd hosts<br/>FreeBSD-tuned<br/>2M conn/server × ~50K servers]
ER --> M[(Mnesia distributed<br/>presence + routing table)]
ER --> C[(Cassandra multi-DC<br/>pending + receipts<br/>3× replication)]
W -->|Signal Protocol<br/>X3DH + Double Ratchet<br/>Forward secrecy per msg| SIG[Client-side E2E<br/>server NEVER sees plaintext]
ER -->|blob storage| MF[(Meta Manifold<br/>+ CDN<br/>E2E-encrypted media)]
ER -->|voice/video| WRT[WebRTC + Selective Forwarding<br/>~100M calls/day<br/>E2E encrypted]
classDef n fill:#dbeafe,stroke:#2563eb,color:#1e3a8a
classDef m fill:#fef3c7,stroke:#d97706,color:#78350f
classDef s fill:#dcfce7,stroke:#16a34a,color:#14532d
class CDN,LB,ER n
class M,C,MF,WRT m
class SIG sWhy every 10× breaks the architecture
Two dominant patterns:
- Language choice IS the architecture. L4 = Node.js works. L5+ = Erlang wins. WhatsApp's 2M conn/server number is Erlang's, not Node.js's, not Go's. Reference: Rick Reed's WhatsApp scaling talk 2014.
- E2E encryption is a data-model constraint. Once you commit to Signal Protocol (X3DH key exchange + Double Ratchet), the server literally CANNOT do server-side search, spam ML, or moderation on message bodies. You can only observe metadata (who talked to whom, when, size). Every feature request must be re-scoped through this filter.
The 3 senior insights before we start Chapter 1
- Erlang is a first-class architectural decision, not an "implementation detail." Every candidate says "we'll use WebSocket." Fewer can explain WHY that plus Erlang beats WebSocket + Node.js at 500M users. The answer: BEAM's scheduler + green-processes model + preemptive context switch. Reference: Joe Armstrong on Erlang.
- Signal Protocol IS the architecture, not a "security feature." X3DH (Extended Triple Diffie-Hellman) + Double Ratchet is a specific mathematical construction that gives forward secrecy per message + deniable authentication. Reference: Signal spec. Say "we use E2E" is a red flag — say "X3DH key agreement + Double Ratchet with new key per message" and you signal senior credibility.
- The 2014 fact (32 engineers, 450M users) is the most important data point. WhatsApp is the counter-example to "we need microservices + Kubernetes + a data mesh." A small team can beat a large one if language + protocol are chosen well. That's the interview takeaway, and it's the philosophical difference from Slack's rich-feature architecture.
Chapter map for the journey ahead
- Chapter 1 — Requirements (1:1 vs group, media, E2E, offline)
- Chapter 2 — Capacity estimation (3B MAU, 100B msg/day, 2M conn/server)
- Chapter 3 — API design (WebSocket protocol, message envelope, receipts)
- Chapter 4 — Data model (users, groups, pending queue, receipts — and what the server CAN'T store)
- Chapter 4.5 — The Signal Protocol: X3DH key agreement + Double Ratchet
- Chapter 5 — L4 MVP: Node.js + Postgres + Redis. Works up to 10K users
- Chapter 6 — L5: Erlang router + Cassandra pending queue. The migration story
- Chapter 6.5 — Why Erlang? BEAM VM + scheduler + green processes
- Chapter 7 — L6: multi-DC Erlang cluster + Signal E2E + Mnesia distributed
- Chapter 7.5 — The 2M-conn engineering: FreeBSD tuning + kernel patches
- Chapter 8 — L7: 3B MAU + Meta backbone + WebRTC calls + moderation without plaintext
- Chapter 9 — Failure modes: presence storms, offline queue overflow, key-rotation bugs
- Chapter 10 — Trade-off matrix (WhatsApp vs Slack vs Signal vs Telegram)
- Chapter 11 — Interview masterclass: 45-min mock, questions to ask
- Chapter 12 — Defense: the 20 hardest interview questions on WhatsApp
Ready? Chapter 1 next: what did the CTO actually ask for?
WhatsApp at 3B MAU runs on a small Erlang codebase that gets 2M concurrent connections per server — the counter-example to microservices maximalism. Signal Protocol (X3DH + Double Ratchet) is the architecture, not a feature — the server cannot read messages, which shapes every downstream decision. Language + protocol choice matters more than architecture diagrams.
- Why is Erlang the right language for WhatsApp-scale messaging?
- What does Signal Protocol (X3DH + Double Ratchet) actually do?
- How does WhatsApp get 2M concurrent connections per server?
- Why can't WhatsApp do server-side moderation on message bodies?
- What was the 2014 headline fact and why does it matter?
Every concept below has its own interactive, animated page in the Learning Tracks section. Read them any time you want to go deeper than the mentor prose above — they're the reusable foundation this chapter is built on.
Chapter 1 next: what did the CTO actually ask for? Messaging, groups, media, E2E encryption, offline, WebRTC — each has functional and non-functional requirements. Get these wrong and you'll design the wrong system.
Components used in this design
Study each component's deep dive to understand it in isolation.
Load Balancer
Distributes incoming traffic across a pool of servers for scale and fault tolerance.
Redis
An in-memory key-value store used for caching, pub/sub, rate limiting, distributed locks, and simple queues.
NoSQL Database
Umbrella for document, wide-column, and key-value stores optimized for horizontal scale over strict schema.
Kafka
A distributed, partitioned, replicated commit log for event streaming, high-throughput ingest, and decoupled services.
Rate Limiter
Enforces per-caller (per-user, per-IP, per-tenant) request budgets to protect downstream systems from abuse and overload.
Patterns applied in this design
Study each pattern's deep dive for the recurring solution logic.
Cache-aside (Lazy loading)
Reads dominate your workload, and every read hits a slow store (database, disk, network). Latency and load on the primary storage climb until the primary becomes the bottleneck.
Sharding (Horizontal partitioning)
Your dataset or write throughput has outgrown a single node. A single primary DB or single cache node can't hold the data or handle the QPS.
Consistent hashing
You need to shard keys across N nodes, but N changes over time (nodes added, removed, or failed). Naive hash-mod-N reshuffles almost everything on every change — cache is wiped, migration cost is huge.
Explore next — related systems
If you enjoyed this problem, these share similar patterns or challenges.