Dropbox
Block-level sync, deduplication, delta upload, conflict resolution.
The scenario
File sync at exabyte scale — where migrating OFF S3 saved $75M/year
Same startup, same engineer #4. Ninth Monday.
Your CTO drops by. "Users want file sync. Personal cloud storage. Desktop app + mobile + web. Share links with permissions. Version history. Team folders. Ship an MVP in 12 weeks."
She pauses. "For context — Dropbox has ~700M registered users, stores exabytes of data, and made the boldest infrastructure decision of the 2010s: they migrated OFF S3 and built their own storage system, saving $75M+/year. That decision — Magic Pocket — is the single most-instructive story in modern infrastructure engineering. If we're going to build cloud storage, we need to understand exactly WHEN and WHY the S3-to-own economics flip."
Here's the paradigm shift:
URL Shortener, Slack, Instagram, Twitter, WhatsApp — small payloads. Even Netflix and YouTube use S3-equivalents (GCS) as source-of-truth and let CDN do the delivery lift.
Dropbox is different. The payload IS the product. Users don't come for a feed or a message — they come to have their files, exactly. Durability must be 11 nines (99.999999999%). (S3's durability guarantee is the same — it's an infrastructure floor.) At exabyte scale with 11-nine durability, you can build cheaper than S3 charges you.
Dropbox proved this in 2015-16 by migrating from S3 to Magic Pocket — their own custom storage system. Break-even point: ~100 PB. Below that, S3 wins. Above that, own it. Reference: Dropbox blog: Inside the Magic Pocket.
The real 2024 numbers
- ~700M+ registered users (Dropbox Q4 2024 earnings)
- ~18M paying subscribers
- Exabytes of stored data (last confirmed via Dropbox Tech blog 2016: ~500 PB in 2016; extrapolate)
- Magic Pocket — custom-built storage since 2016, replaces most S3 usage
- 4 MB block size — Dropbox breaks every file into 4 MB blocks + hashes for dedup
- ~95% of storage on Shingled Magnetic Recording (SMR) drives — 25% cheaper capex than perpendicular recording (PMR)
- Reed-Solomon 8+2 erasure coding — 25% overhead vs 3× replication which is 200% overhead. Same durability
- $75M+/year savings by moving off S3 (Dropbox tech blog 2016)
- ~11 nines durability — same as S3 with erasure coding + geographic distribution
Interview soundbite: "Dropbox is not S3-plus-a-client. Dropbox is Magic Pocket — an exabyte-scale storage system with 4 MB blocks, SHA-256 dedup, Reed-Solomon 8+2 erasure coding, on cheap SMR drives, distributed across regions. At 700M users and exabytes stored, they built it because S3 costs 3x their unit economics."
The whole journey at a glance
Every 10× in stored bytes shifts the cost model:
text═══════════ DROPBOX ARCHITECTURE ACROSS 4 SCALES ═══════════ L4 (10K users) L5 (10M users) L6 (100M users) L7 (700M users, exabytes) S3 + Postgres Block-level dedup Magic Pocket (built Magic Pocket + SMR 12 weeks · $500/mo 6 months · $200K/mo 24 months · $2M/mo ongoing · save $75M/yr ┌────────┐ ┌────────┐ ┌── Global CDN ──────┐ ┌── DBX own edge ──────┐ │ Web │ │ Web │ │ static assets │ │ edge caching │ │Desktop │ │Desktop │ └─┬──┬──┬──┬─────────┘ │ 200+ PoPs │ │Mobile │ │Mobile │ │ │ │ │ └──┬──┬──┬──┬──────────┘ └───┬────┘ └───┬────┘ ┌─▼──▼──▼──▼─────┐ │ │ │ │ │ │ │ Regional ALB │ ┌───▼──▼──▼──▼──────────┐ ┌──▼───┐ ┌──▼──┐ │ API + block │ │ Metadata + Sync svc │ │ ALB │ │ ALB │ │ service │ │ block placement engine │ │ │ │ │ └───┬──────┬─────┘ └──┬──┬──┬──┬───────────┘ └──┬───┘ └──┬──┘ │ │ │ │ │ │ │ │ │ │ ┌──▼──▼──▼──▼──────────┐ ┌──▼──────┐ ┌───▼─────────┐ ┌───▼──────▼───┐ │ Sync coordination │ │ Django │ │ Python + Go │ │ Metadata svc │ │ Delta sync (rsync) │ │ + MySQL │ │ + MySQL │ │ Block svc │ │ File-notifs at scale │ │ + S3 │ │ + S3 │ │ Sync svc │ └──┬───────────────────┘ └──┬──────┘ │ + Redis │ └───┬───────────┘ │ │ └───┬─────────┘ ┌───▼──────────┐ ┌──▼────────────────────┐ │ ┌───▼──┐ │Magic Pocket │ │ Magic Pocket │ │ │Redis │ │(Dropbox's │ │ (4 MB blocks) │ │ │hot │ │own storage) │ │ SHA-256 dedup │ │ │metad │ │+ RS 8+2 EC │ │ Reed-Solomon 8+2 EC │ │ └───┬──┘ │+ multi-DC │ │ 95% SMR drives │ │ │ └───┬──────────┘ │ 200 PB+ per region │ ┌──▼───┐ ┌───▼─┐ │ │ multi-region replic. │ │Postgr│ │Postgr│ │ └──┬───────────────────┘ │(meta │ │+ shar│ │ │ │+ user│ │d by │ │ ┌──▼────────────────────┐ │+ share │user │ ┌───▼──────────┐ │ Legacy S3 (5%) │ │Multi-│ │_id │ │Legacy S3 tail│ │ for edge/international│ │AZ) │ │+ RF=2│ │(migration │ │ + spillover │ └──────┘ └──────┘ │incomplete) │ └───────────────────────┘ └──────────────┘ ↑ ↑ ↑ ↑ Boring Block-level dedup Magic Pocket goes live. Magic Pocket at exabyte. S3 + Django + MySQL sharding. S3 usage drops from 95% SMR drives + 8+2 EC. works. Redis for hot 100% -> ~50%. Own network + edge PoPs. metadata. Save $75M/year. Chapter 5 Chapters 6+6.5 Chapter 7+7.5 Chapter 8 walks walks through walks through the walks through Magic through block-level dedup Magic Pocket migration Pocket at exabyte, SMR L4 MVP + sync algorithm + Reed-Solomon 8+2 EC economics, and DBX's own network buildout Key insight: Dropbox proved a rule that will define the next decade: "Own your storage when scale exceeds ~100 PB." Below that, S3 is cheaper. Above that, 3x savings on capex + operations. Magic Pocket is the archetype for building your own physical infrastructure at scale.
The same 4 tiers as clean architecture diagrams
L4 · 10K users · S3 + Postgres · $500/mo · 12 weeks:
flowchart TD
W([Web / Desktop / Mobile]) -->|HTTPS| LB[ALB<br/>$25/mo]
LB --> API[Django API<br/>c5.large]
API --> PG[(MySQL Multi-AZ<br/>files + users + shares<br/>metadata only)]
API -->|presigned PUT| S3[(S3<br/>file binaries<br/>~$100/mo)]
classDef n fill:#dbeafe,stroke:#2563eb,color:#1e3a8a
class LB,API,PG,S3 nL5 · 10M users · Block-level dedup + sharded MySQL · $200K/mo · 6 months:
flowchart TD
W([Global users]) -->|HTTPS| LB[Regional ALB]
LB --> API[100+ Python/Go services]
API --> B[Block service<br/>4 MB block dedup<br/>SHA-256 hash keys]
API --> M[(Sharded MySQL by user_id<br/>metadata + shares)]
API --> RD[(Redis hot metadata<br/>+ recent activity)]
B --> S3[(S3 for blocks<br/>dedupe-friendly)]
classDef n fill:#dbeafe,stroke:#2563eb,color:#1e3a8a
classDef m fill:#fef3c7,stroke:#d97706,color:#78350f
class LB,API n
class B,M,RD,S3 mL6 · 100M users · Magic Pocket goes live · $2M/mo · 24 months:
flowchart TD
W([Global users]) -->|HTTPS| CDN[Global CDN<br/>static assets]
CDN --> API[Regional API]
API --> META[Metadata svc<br/>+ block placement engine]
META --> M[(Sharded MySQL<br/>user + file + block-list)]
META --> RD[(Redis hot metadata)]
API --> BS[Block Service]
BS --> MP[Magic Pocket<br/>Dropbox's own storage<br/>4 MB blocks<br/>Reed-Solomon 8+2 EC<br/>multi-DC replication]
BS --> S3[(Legacy S3<br/>~50% traffic during migration)]
classDef n fill:#dbeafe,stroke:#2563eb,color:#1e3a8a
classDef m fill:#fef3c7,stroke:#d97706,color:#78350f
classDef mp fill:#dcfce7,stroke:#16a34a,color:#14532d
class CDN,API,BS n
class META,M,RD,S3 m
class MP mpL7 · 700M users · Magic Pocket at exabyte + own edge · save $75M/yr:
flowchart TD
W([Global users]) -->|HTTPS| CDN[Dropbox own edge<br/>200+ PoPs · own backbone]
CDN --> API[Regional API]
API --> META[Metadata + Sync svc<br/>delta sync · rsync-based]
META --> M[(Sharded MySQL<br/>hundreds of shards<br/>metadata + shares)]
META --> RD[(Redis hot metadata)]
API --> BS[Block Service<br/>4 MB block placement]
BS --> MP[Magic Pocket<br/>exabytes of storage<br/>4 MB blocks · SHA-256 dedup<br/>Reed-Solomon 8+2 EC<br/>95% SMR drives · 200 PB per region<br/>multi-region replication]
BS --> S3T[(Legacy S3 tail<br/>~5% edge use cases)]
classDef n fill:#dbeafe,stroke:#2563eb,color:#1e3a8a
classDef m fill:#fef3c7,stroke:#d97706,color:#78350f
classDef mp fill:#dcfce7,stroke:#16a34a,color:#14532d
class CDN,API,BS n
class META,M,RD,S3T m
class MP mpWhy every 10× breaks the architecture
- Dedup is the whole business. L4 = file-level SHA-256. L5+ = block-level (4 MB blocks). At 700M users, block-level dedup typically saves 30-70% storage (users share Excel, PDFs, videos across teams). Reference: Dropbox on block-level dedup.
- The S3-to-own break-even is ~100 PB. Below that, S3's economies of scale beat you. Above that, you can beat S3 by building your own. Dropbox crossed this in 2015-16 and never looked back. Reference: Dropbox Magic Pocket blog.
- Reed-Solomon erasure coding is the durability primitive. Instead of 3× replication (200% storage overhead) → 8+2 erasure coding (25% overhead) with the same 11-nine durability. The math: any 8 of 10 fragments reconstructs the block. Reference: Erasure coding vs replication comparison.
The 3 senior insights before we start Chapter 1
- Owning storage above 100 PB is a genuine competitive advantage. Every serious cloud storage company (Backblaze, Wasabi, Cloudflare R2) has followed Dropbox's playbook. The 2010s decade taught the industry that Own > Rent above the break-even. Reference: Backblaze's own storage story.
- Reed-Solomon 8+2 is the industry standard. Not 3-way replication. Not RAID-6. Reed-Solomon coding gives 25% overhead for 11-nine durability. Facebook uses 10+4 for cold storage; Dropbox uses 8+2. Reference: Facebook f4 warm BLOB storage paper (OSDI 2014).
- Shingled Magnetic Recording (SMR) is 25% cheaper than PMR. SMR writes overlapping tracks (like shingles on a roof) — hence read/rewrite issues on random writes. For write-once-read-many cloud storage, this trade-off is a winning cost lever. Reference: Dropbox SMR blog. Amazon Glacier and Google Cloud Archive use similar techniques.
Chapter map for the journey ahead
- Chapter 1 — Requirements (upload, sync, share, permissions, version history)
- Chapter 2 — Capacity estimation (700M users, exabyte-scale)
- Chapter 3 — API design (upload, sync, share, notification)
- Chapter 4 — Data model (users, files, blocks, shares, versions)
- Chapter 4.5 — Block-level dedup: 4 MB block, SHA-256 hash, dedup ratio math
- Chapter 5 — L4 MVP: S3 + Django + Postgres. Works up to 10K users
- Chapter 6 — L5: Block service + MySQL sharding + Redis hot metadata
- Chapter 6.5 — The sync algorithm: delta sync, rsync-based, conflict resolution
- Chapter 7 — L6: Magic Pocket migration. S3-to-own break-even math
- Chapter 7.5 — Reed-Solomon 8+2 erasure coding: durability math + implementation
- Chapter 8 — L7: SMR drives + own edge + $75M/year savings
- Chapter 9 — Failure modes: block loss, sync conflicts, ransomware attacks
- Chapter 10 — Trade-off matrix (Dropbox vs Google Drive vs iCloud vs OneDrive)
- Chapter 11 — Interview masterclass: 45-min mock, questions to ask
- Chapter 12 — Defense: the 20 hardest interview questions on Dropbox
Ready? Chapter 1 next: what did the CTO actually ask for?
Dropbox at 700M users runs on Magic Pocket — Dropbox's own storage system with 4 MB blocks, SHA-256 dedup, Reed-Solomon 8+2 erasure coding (25% overhead vs 200% for 3× replication), on cheap SMR drives, distributed across regions. The S3-to-own break-even is ~100 PB; below that S3 wins, above that own it. Dropbox saves $75M+/year by owning above the threshold.
- Why did Dropbox migrate off S3 and how much did it save?
- What is the S3-to-own break-even point (~100 PB)?
- What is Reed-Solomon 8+2 erasure coding vs 3× replication?
- Why does Dropbox use 4 MB blocks with SHA-256 dedup?
- Why is Shingled Magnetic Recording (SMR) 25% cheaper than PMR?
Every concept below has its own interactive, animated page in the Learning Tracks section. Read them any time you want to go deeper than the mentor prose above — they're the reusable foundation this chapter is built on.
The algorithm behind block placement in Magic Pocket and MySQL shard placement at L5+.
The sync algorithm at L6 uses cache invalidation via file-notifs — pushed to clients when their files change on the server.
Chapter 1 next: what did the CTO actually ask for? Upload, sync, share, permissions, version history — each has functional and non-functional requirements per level. Get these wrong and you'll design the wrong system.
Components used in this design
Study each component's deep dive to understand it in isolation.
Load Balancer
Distributes incoming traffic across a pool of servers for scale and fault tolerance.
CDN (Content Delivery Network)
A globally distributed cache that serves static and cacheable dynamic content close to the user.
SQL Database
A row-oriented, ACID-compliant relational database — the default for transactional workloads.
Message Queue
Async task queue that decouples producers from consumers and smooths bursts. Each message is processed exactly once (per-message ACK model).
Object Storage (S3, GCS, Azure Blob)
Durable, cheap, flat-namespace storage for blobs — images, videos, backups, logs, and any large binary object.
Search Engine (Elasticsearch, OpenSearch, Meilisearch)
Inverted-index-based full-text and structured search — the right tool for 'find me records matching this query' when SQL LIKE isn't fast enough.
Rate Limiter
Enforces per-caller (per-user, per-IP, per-tenant) request budgets to protect downstream systems from abuse and overload.
Patterns applied in this design
Study each pattern's deep dive for the recurring solution logic.
Cache-aside (Lazy loading)
Reads dominate your workload, and every read hits a slow store (database, disk, network). Latency and load on the primary storage climb until the primary becomes the bottleneck.
Sharding (Horizontal partitioning)
Your dataset or write throughput has outgrown a single node. A single primary DB or single cache node can't hold the data or handle the QPS.
Explore next — related systems
If you enjoyed this problem, these share similar patterns or challenges.