YouTube

Same startup, same engineer #4. Seventh Monday.

Your CTO drops by. "After Netflix, users want a user-generated video feature. Anyone can upload, anyone can watch, anyone can comment. A recommendation feed keeps people engaged. Ship an MVP in 12 weeks."

She pauses. "For context — YouTube processes 500+ hours of video uploaded EVERY MINUTE. 2.7 billion monthly active users. And they invented Vitess (the MySQL sharding proxy) because the read load broke every off-the-shelf database around 2010. If we're going to build this, we need to grasp WHY UGC video is a fundamentally different problem than Netflix's licensed catalog."

Here's the paradigm shift:

Netflix was 100K licensed titles, 300M subs, 200 Tbps peak egress. Solvable with Open Connect Appliances because the catalog is small and predictable — you push tomorrow's popular titles at 3am.

YouTube is 500 hours uploaded per minute — no catalog is fixed. No push algorithm can predict what will be popular. The recommendation engine has to work on ~10^10 videos, most of which have <10 views. The compute cost of encoding UGC (billions of hours of video) is so extreme that YouTube built custom silicon — the Argos Video Coding Unit — to encode 2x-33x faster than commodity CPUs. Reference: Google Cloud blog on Argos VCU.

The recommendation engine is even more consequential. It's an entire ML system that ranks candidates → ranker → then serves 720M hours of watch time per day. Reference: Covington et al., "Deep Neural Networks for YouTube Recommendations" (RecSys 2016) — the canonical two-tower ML paper.

The real 2024 numbers

• ~2.7B MAU (Statista 2024)
• ~500+ hours of video uploaded PER MINUTE (DemandSage 2024)
• ~720M hours watched per day globally
• ~1 billion hours watched per day including YouTube Shorts
• ~10 EXABYTES of stored video (10^19 bytes)
• ~200 PB CDN cache at Google's edge
• ~$32B ad revenue 2024 (Alphabet Q4 2024)
• Argos VCU: Google's custom silicon, 2x-33x encoding speed vs commodity
• Vitess (born at YouTube ~2010): now runs at Slack, GitHub, HubSpot, Etsy

Interview soundbite: "YouTube is 500 hours uploaded per minute times a recommendation engine that decides which 20 clips YOU see next. Storage is trivially exabyte-scale; the interesting problem is (a) transcoding at $/hour scale — hence Argos silicon — and (b) two-tower ML ranking a billion candidates in <200ms. Vitess was invented here because MySQL didn't scale to the read load."

The whole journey at a glance

Every 10× in upload rate shifts the bottleneck:

text
    ═══════════ YOUTUBE ARCHITECTURE ACROSS 4 SCALES ═══════════

    L4 (10 upl/day)    L5 (1M upl/day)    L6 (100M upl/day)      L7 (720M+ upl/day, 2.7B MAU)
    S3 + CloudFront    Multi-region + rec  Vitess + rec           Argos VCU + DNN ranker
    12 weeks · $50K/mo 12 months · $2M/mo  24 months · $200M/mo   ongoing · $10B/yr

    ┌────────┐         ┌────────┐         ┌── Global Cloudflare ┐  ┌── Google Edge PoPs ──┐
    │  Web   │         │  Web   │         │  or CloudFront       │  │  1000+ PoPs · 200PB   │
    │TV/mob  │         │TV/mob  │         └─┬──┬──┬──┬─────────┘  │  own network + ISPs   │
    └───┬────┘         └───┬────┘           │  │  │  │            └──┬──┬──┬──┬──────────┘
        │                  │              ┌──▼──▼──▼──▼─────┐        │  │  │  │
     ┌──▼───┐           ┌──▼──┐           │  Regional ALB   │    ┌───▼──▼──▼──▼──────────┐
     │ ALB  │           │ ALB │           │  API tier       │    │ Google Front End      │
     │      │           │     │           └───┬──────┬──────┘    │ (GFE) reverse-proxy   │
     └──┬───┘           └──┬──┘               │      │           │ + Envoy at internal   │
        │                  │                  │      │           └──┬──┬──┬──┬──────────┘
     ┌──▼──────┐       ┌───▼─────────┐    ┌───▼──────▼───┐          │  │  │  │
     │ Monolith│       │ Microserv.  │    │  Playback svc │       ┌──▼──▼──▼──▼──────────┐
     │ Django  │       │ Go + Python │    │  Upload svc  │       │ Playback / Upload /   │
     │ + MySQL │       │ + MySQL     │    │  Rec svc     │       │ Rec / Comment / Search│
     │ + Redis │       │ + Vitess    │    │  Search svc  │       │ / Shorts / Ad tiers   │
     └──┬──────┘       └───┬─────────┘    │  Ad svc      │       └──┬──┬──┬──┬──────────┘
        │                  │              └───┬───────────┘          │  │  │  │
        │              ┌───▼──┐           ┌───▼──────────┐         ┌──▼──▼──▼──▼──────────┐
        │              │Redis │           │ Vitess shards│         │ Vitess (many         │
        │              │thumbs│           │ (mySQL under)│         │ thousands of shards)  │
        │              └───┬──┘           └───┬──────────┘         └──┬───────────────────┘
        │                  │                  │                       │
     ┌──▼───┐          ┌───▼─┐            ┌───▼──────────┐         ┌──▼─────────────────┐
     │MySQL │          │MySQL│            │Sharded MySQL │         │ Two-tower deep NN  │
     │(vid  │          │+ 3  │            │+ Vitess proxy│         │ candidate gen +    │
     │meta+ │          │repl │            │Covington 2016│         │ ranker (Covington  │
     │ user)│          │icas │            │DNN ranker    │         │ 2016 + updates)    │
     └──────┘          └─────┘            │Spanner for   │         │ 720M hours/day     │
                                          │ ads billing  │         └────────────────────┘
                                          └───┬──────────┘             │
                                              │                    ┌────▼──────────────┐
                                              │                    │ Argos VCU custom  │
                                          ┌───▼──────────┐         │ silicon for       │
                                          │Google Cloud  │         │ transcoding       │
                                          │Storage + CDN │         │ 2x-33x commodity  │
                                          │(GCS multi-   │         │ CPU speed         │
                                          │region)       │         └──┬────────────────┘
                                          └──────────────┘             │
                                                                    ┌──▼──────────────┐
                                                                    │ Spanner for ads │
                                                                    │ + billing +     │
                                                                    │ strong-consist. │
                                                                    │ paid features   │
                                                                    └─────────────────┘

    ↑                  ↑                  ↑                       ↑
   Boring           Vitess enters (YT   Argos VCU custom silicon Two-tower deep NN +
   Django + MySQL   invented it 2010 to for encoding. Recommend  Argos + Spanner ads +
   works.           handle read load).  DNN goes live (Covington 200 PB edge cache +
                    Async transcode.    2016 paper).             1000+ PoPs.

   Chapter 5      Chapters 6+6.5      Chapter 7+7.5              Chapter 8
   walks          walks through Vitess walks through Argos VCU   walks through 2.7B MAU,
   through        + async transcode +  + Covington 2016 DNN      Vitess-at-scale,
   L4 MVP         Redis thumbnails     ranker + Spanner ads      Shorts vs long-form

  Key insight: YouTube is not "video storage" — it's transcoding at exabyte scale
  (hence Argos custom silicon) + ranking at billion-candidate scale (hence
  Covington 2016 two-tower DNN). Vitess was invented here because MySQL couldn't
  handle the read load. Every YouTube-scale problem has spawned an industry-wide
  tool.

The same 4 tiers as clean architecture diagrams

L4 · 10 uploads/day · Django + S3 + CloudFront · $50K/mo · 12 weeks:

L5 · 1M uploads/day · Vitess + async transcode + rec service · $2M/mo · 12 months:

L6 · 100M uploads/day · Argos VCU + Covington 2016 DNN · $200M/mo · 24 months:

L7 · 2.7B MAU · 720M hrs/day · Argos + Vitess + Spanner · $10B/yr:

Why every 10× breaks the architecture

1. Transcoding cost is the whole game at scale. L4 = FFmpeg on EC2 works. L6+ = you need custom silicon because commodity CPUs are 33x slower and $/hour math breaks. That's the Argos VCU story (Google Cloud blog 2021).

1. Recommendation isn't a feature — it IS the product. 720M hours watched per day is not because users self-select; it's because the two-tower deep NN ranks a billion candidate videos per user, choosing 20. Reference: Covington et al. 2016 — the canonical paper every ML engineer reads.

1. Vitess is the origin story you must cite. YouTube built Vitess ~2010 because MySQL didn't scale to their read load. Since then, Vitess runs at Slack, GitHub, HubSpot, Etsy. Do NOT say "YouTube uses MongoDB" or "Cassandra" — that's a common tutorial mistake.

The 3 senior insights before we start Chapter 1

1. Custom silicon (Argos VCU) is a strategic moat. Amazon has Graviton, Google has TPU + Argos VCU + custom video ASIC. If your interview answer to "how do you handle 500 hours/minute of transcoding?" is "we scale FFmpeg horizontally", you're 2x-33x more expensive than YouTube. Naming Argos signals senior-level awareness. Reference: Argos VCU Google Cloud blog.

1. Covington 2016 IS the recommendation architecture. The two-tower model (user tower + item tower → dot product) is the canonical setup. Candidate generation (fast, retrieves top 1000 from 10B) → ranker (slow, scores 1000, picks 20). Almost every consumer-scale rec system runs on this shape. Reference: Deep Neural Networks for YouTube Recommendations.

1. Vitess started at YouTube. In 2010. To handle MySQL read load. Then it was open-sourced and became the sharding proxy of choice for Slack, GitHub, HubSpot, Etsy. YouTube's problems are precursors to industry-wide tools. If you're asked "how do you shard MySQL?", the answer is "Vitess — invented at YouTube, now runs half the internet."

Chapter map for the journey ahead

• Chapter 1 — Requirements (upload, playback, comments, likes, recs, ads)
• Chapter 2 — Capacity estimation (2.7B MAU, 500hr/min upload, 720M hrs/day watch)
• Chapter 3 — API design (upload, playback, engagement, subscription)
• Chapter 4 — Data model (videos, users, watch history, comments)
• Chapter 4.5 — The Vitess origin: why YouTube invented it
• Chapter 5 — L4 MVP: Django + S3 + CloudFront. Works up to 10K users
• Chapter 6 — L5: Vitess + async FFmpeg transcode + recommendation service
• Chapter 6.5 — Two-tower deep NN ranker (Covington 2016 in depth)
• Chapter 7 — L6: Argos VCU custom silicon + Spanner ads billing
• Chapter 7.5 — The transcoding cost model: $/hour math for FFmpeg vs Argos
• Chapter 8 — L7: 2.7B MAU + Shorts + AI-generated content moderation
• Chapter 9 — Failure modes: viral upload thundering herd, rec model degradation, ad brand safety
• Chapter 10 — Trade-off matrix (YouTube vs TikTok vs Twitch)
• Chapter 11 — Interview masterclass: 45-min mock, questions to ask
• Chapter 12 — Defense: the 20 hardest interview questions on YouTube

Ready? Chapter 1 next: what did the CTO actually ask for?