Netflix

Same startup, same engineer #4. Fifth Monday.

Your CTO drops by. "Users love shortener, chat, Instagram-style feed, and Twitter timelines. Now we want to add video-on-demand. Movies, TV series, some short-form. Adaptive bitrate. Global. 4K. Downloads for offline. Ship an MVP in 12 weeks."

She pauses. "By the way — one of our licensed titles just went viral. 5 million people started streaming it in the same hour last night. Our S3 bill was $180,000 for that one night alone."

Welcome to video streaming.

Here's the paradigm shift:

URL Shortener, Slack, Instagram, Twitter — all shared one thing: the payload was small. A short_code is 7 bytes. A tweet is ~280. A photo is ~200 KB. Even Slack messages are ~1 KB.

A movie is 5 GB. A 4K show binge is 50 GB per user per weekend. (Netflix's average bitrate is 5 Mbps at HD, 15 Mbps at 4K.) You don't design video the way you design tweets. The CDN isn't a component — the CDN IS the architecture.

That $180K S3 bill teaches the whole lesson: egress bandwidth is the entire cost. Netflix's answer, brilliantly, was to build their own CDN — the Open Connect Appliance (OCA) — and give it away for free to ISPs. It's the second-most-important architecture decision they've ever made (after "streaming, not DVD"), and if you don't mention it in the interview, you'll fail.

The real 2024 numbers

Before we design anything, know the scale you're designing against:

• ~300M paid subscribers (Netflix Q4 2024 earnings)
• ~65M peak concurrent streams (Netflix Squid Game S2 launch, Dec 2024)
• ~17,000+ Open Connect Appliances embedded in ISPs globally (Open Connect deployment map)
• ~200 Tbps peak egress across all OCAs at prime time
• ~15 billion events/day through Keystone (Netflix's Kafka pipeline) (Netflix Tech Blog: Keystone)
• ~99.99% availability during regional AWS incidents (via Chaos Monkey engineering — Netflix invented it in 2011)

Interview soundbite: "Netflix at 300M subscribers is not a video-storage company. It's a global CDN with a video catalog on top. 95%+ of bytes leaves customer devices from an OCA embedded 1-2 hops from the customer, not from AWS."

The whole journey at a glance

Every 10× in subscriber scale shifts what's expensive:

text
    ═══════════ NETFLIX ARCHITECTURE ACROSS 4 SCALES ═══════════

    L4 (100K subs)     L5 (50M subs)      L6 (200M subs)         L7 (300M subs, global)
    S3 + CloudFront    Multi-region + DRM Open Connect CDN       17K+ OCAs · Keystone
    12 weeks · $50K/mo 12 months · $2M/mo 24 months · $50M/mo    ongoing · $2B/yr

    ┌────────┐         ┌────────┐         ┌── Global CloudFront ┐  ┌── Own Open Connect ──┐
    │  Web   │         │  Web   │         │  + DRM licenses     │  │  17,000+ OCAs at ISPs│
    │TV apps │         │TV apps │         │  regional cache     │  │  95%+ traffic served │
    └───┬────┘         └───┬────┘         └─┬──┬──┬──┬─────────┘  │  from 1-2 hops away  │
        │                  │                │  │  │  │            └──┬──┬──┬──┬─────────┘
     ┌──▼───┐           ┌──▼──┐          ┌──▼──▼──▼──▼─────┐         │  │  │  │
     │ ALB  │           │ ALB │          │  Multi-region   │    ┌────▼──▼──▼──▼──────────┐
     │      │           │     │          │  ALB · Zuul     │    │ Zuul API Gateway       │
     └──┬───┘           └──┬──┘          │  API gateway    │    │ (custom Java by NF)     │
        │                  │             └───┬──────┬──────┘    │ 200K+ RPS routing       │
        │                  │                 │      │           └──┬──────┬──────┬───────┘
     ┌──▼──────┐       ┌───▼─────────┐   ┌───▼──────▼──────┐       │      │      │
     │ Monolith│       │ Microserv.  │   │  1000+ Micros   │    ┌──▼──────▼──────▼──────┐
     │ Java    │       │ 100+ svcs   │   │  Java Spring    │    │ Recommendation engine │
     │ 1 API   │       │ Spring Boot │   │  Boot / Cassandra│    │ (Cinematch → LTV pred)│
     │ + MySQL │       │ + Cassandra │   │                 │    │ + Personalization ML  │
     └──┬──────┘       │ + EVCache   │   └───┬─────────────┘    └──┬──┬──┬──┬───────────┘
        │              └───┬─────────┘       │                     │  │  │  │
        │              ┌───▼──┐               │                  ┌──▼──▼──▼──▼──────────┐
        │              │EVCache│              │                  │ Cassandra + EVCache  │
        │              │Netflix│              │                  │ + S3 for cold blobs  │
        │              │memcached           ┌─▼────────────┐    │ + Iceberg data lake  │
        │              │variant│            │ Regional S3  │    └──┬───────────────────┘
        │              └───┬──┘             │ + Push to OCA│       │
        │                  │                └───┬──────────┘    ┌──▼──────────────────┐
     ┌──▼───┐          ┌───▼─┐                  │              │ Keystone (Kafka +   │
     │MySQL │          │Cassandra                │              │ Flink + Iceberg)    │
     │(user+│          │+ EVCache│              │              │ 15B events/day      │
     │title+│          │+ S3     │              │              │ real-time analytics │
     │DRM)  │          │multi-AZ │              │              └─────────────────────┘
     └──────┘          └─────────┘              │
                                                ▼
                                     Open Connect Appliances (OCAs)
                                     Push videos here at 03:00 local time
                                     Play from 1-2 hops from user

    ↑                  ↑                  ↑                       ↑
   Boring           CloudFront + DRM   Own the CDN. Open         17K OCAs. Keystone.
   MVP: S3 +        Multi-region.      Connect Appliances.       Chaos Monkey. Zuul.
   CloudFront.      100+ Java Spring   Push content to 15,000+    95% traffic from
   Add-on: DRM.     Boot services.     ISPs at 3am.              1-2 hops away.

   Chapter 5      Chapters 6+6.5      Chapter 7+7.5              Chapter 8
   walks          walks through        walks through              walks through the
   through        multi-region CDN,   the Open Connect            Keystone pipeline,
   L4 MVP         DRM, per-title      economics + push            personalization ML,
                  encoding             algorithm                   and cost as strategy

  Key insight: video is not about STORAGE. It's about EGRESS. At 300M subscribers,
  95%+ of the bytes leaving your customer's device come from an appliance 1-2 hops
  from them, not from your cloud. That's Open Connect. That's the entire architecture.

The same 4 tiers as clean architecture diagrams

The ASCII compressed all four tiers. Here are the same shapes as clean Mermaid:

L4 · 100K subs · S3 + CloudFront · $50K/mo · 12 weeks:

L5 · 50M subs · multi-region + DRM · $2M/mo · 12 months:

L6 · 200M subs · Open Connect CDN · $50M/mo · 24 months:

L7 · 300M subs · 17K+ OCAs + Keystone · $2B/yr infra:

Why every 10× breaks the architecture

Look at the four tiers side-by-side. Three patterns dominate:

1. Compute stays roughly the same. APIs, DBs, metadata services — they scale by adding more of the same. Boring.

1. The CDN is the whole story. L4 = CloudFront. L5 = regional CloudFront. L6 = your own OCAs at ISPs. L7 = 17K OCAs pushing 200 Tbps. The "we serve videos" problem transforms into "we own the last mile of every ISP" problem.

1. The data pipeline emerges. L4 has no analytics. L5 uses simple aggregates. L6 needs a real streaming pipeline (Kafka + Flink). L7 has Keystone doing 15B events/day for personalization + billing + observability.

The 3 senior insights before we start Chapter 1

1. Open Connect is the moat, not the codec. Every candidate knows about H.264/H.265/AV1. Very few can explain how Netflix negotiated with 15,000+ ISPs to embed their CDN appliances for free. That relationship IS the architecture — anyone can add H.265; nobody can duplicate 17,000 appliances 1-2 hops from every customer. Reference: Open Connect ISP FAQ.

1. Push-at-3am, don't pull-on-demand. Traditional CDNs pull content when a customer requests it (cache miss = origin fetch). Netflix INVERSED this. At 03:00 local time to each OCA, they PUSH the next day's expected-popular titles, based on ML predictions. Cache hit rate ≈ 95%+ because the popularity model works. Reference: Open Connect predictive push.

1. Keystone is Kafka, not custom. Netflix's data pipeline (Keystone) is 90% off-the-shelf Kafka + Flink + Iceberg, custom-tuned. It handles 15 billion events/day. Do NOT say "Netflix uses a proprietary data pipeline" — that's a common tutorial mistake. Reference: Netflix Tech Blog: Keystone Real-Time Stream Processing.

Chapter map for the journey ahead

• Chapter 1 — Requirements (functional + non-functional per level)
• Chapter 2 — Capacity estimation (300M subs, 200 Tbps peak, 65M concurrent)
• Chapter 3 — API design (playback, catalog, user, billing)
• Chapter 4 — Data model (titles, users, entitlements, watch history)
• Chapter 4.5 — The video pipeline: master file → transcoding → ABR ladder → OCA push
• Chapter 5 — L4 MVP: S3 + CloudFront. Works up to 100K subscribers
• Chapter 6 — L5: multi-region + per-title encoding + DRM
• Chapter 6.5 — Per-title encoding: how Netflix saves 25% bandwidth per title
• Chapter 7 — L6: Open Connect Appliance CDN. Push at 3am.
• Chapter 7.5 — The Keystone data pipeline: Kafka + Flink + Iceberg
• Chapter 8 — L7: 17K OCAs + personalization ML + Chaos Monkey engineering
• Chapter 9 — Failure modes: OCA loss, DRM outage, viral title thundering herd
• Chapter 10 — Trade-off matrix
• Chapter 11 — Interview masterclass: 45-min mock, questions to ask
• Chapter 12 — Defense: the 20 hardest interview questions on Netflix

Ready? Chapter 1 next: what did the CTO actually ask for?