nosql-db

Distributed Database (DynamoDB, Cassandra, Spanner)

Horizontally-scaled database with automatic partitioning and replication — the answer when one node can't hold the data or the traffic.

Why it exists

A single-node database has a ceiling: RAM for the working set, CPU for query planning, network for connections, disk for durability. At extreme scale (petabytes, millions of RPS, global users), no single node is big enough. Distributed databases spread data across dozens or thousands of nodes with automatic sharding, replication, failover, and (in some cases) global consistency. They pay for this with operational complexity and (often) weaker consistency semantics.

How it works

Data is partitioned across nodes by a partition key (usually hashed via consistent hashing). Each partition is replicated (typically 3× within a region, plus cross-region for global systems). Reads and writes go to any replica; quorum protocols (R + W > N) tune consistency vs availability. Some systems (DynamoDB, Cassandra) are leaderless — any node can accept a write and gossip it. Others (Spanner, CockroachDB) are leader-per-partition using Raft for consensus, giving stronger consistency at the cost of more coordination.

Scaling characteristics

Linear horizontal scale: add nodes, get more capacity. DynamoDB serves trillions of requests/day. Cassandra clusters handle millions of writes/sec. Latency: single-digit ms for local reads/writes; tens of ms cross-region. Storage is essentially unlimited (petabytes). Cost scales with capacity — pay per RCU/WCU on DynamoDB, per node on Cassandra.

When to use it

Massive scale where single-node databases can't cope
Global user base with regional presence
Extreme write throughput (>100K writes/sec sustained)
Predictable-shape workloads (simple lookups by partition key)
You need automatic replication + failover without babysitting

When NOT to use it

Small-to-medium scale — Postgres is much simpler
Complex ad-hoc queries — distributed DBs are usually key-value or wide-column, not relational
Transactions across many keys — most distributed DBs limit cross-key transactions
Teams without operational NoSQL experience — the learning curve is real

Failure modes

Hot partition: bad key choice causes one node to be pegged while others idle
Silent tombstone accumulation in Cassandra (deletes that never actually delete) — hurts read latency
Rebalancing during scale-up disrupts throughput temporarily
Eventual consistency confuses application logic — write, read from another node, get old value
Cross-region replication lag causes stale reads during regional failover
Cost overrun from misconfigured provisioned capacity (DynamoDB)

Alternatives

Sharded SQL (Vitess, Citus) — familiar SQL semantics + horizontal scale, more operational care needed
NewSQL (Spanner, CockroachDB, TiDB) — SQL + horizontal scale + global consistency (at latency cost)
Object storage + database index — for very large blobs; S3 + Postgres index is a common combo
Time-series DB (InfluxDB, Timescale) for time-series workloads specifically

Interview questions

Compare DynamoDB and Cassandra. When would you pick each?
Design a partition key for a chat system. What's the risk with a bad choice?
Explain quorum reads and writes. When would you use strong quorum vs local quorum?
How does Spanner achieve external consistency? What's TrueTime?
Your DynamoDB cost tripled last month. Common causes and how to diagnose?
How would you migrate 100 TB from Postgres to Cassandra with zero downtime?

Systems that use this component

See how the real designs on this platform put distributed-db to work — concrete usage context per system.

YouTube

intermediate

Spanner for global metadata

Open system

Netflix

intermediate

CockroachDB + DynamoDB for global user data

Open system

Payment System

intermediate

Spanner for transaction ledger requiring linearizability

Open system

The story of NewSQL

2012, Google: atomic clocks solve the last-mile problem of distributed SQL

For 40 years, the industry accepted an unspoken rule: distributed = eventually consistent. NoSQL leaned in. Then in 2012, Google published the Spanner paper and shattered the assumption. Spanner offers external consistency — the strictest guarantee possible — across data centers on different continents. Real ACID. Real SQL. At planet scale.

The trick was TrueTime: GPS + atomic clocks in every Google data center, exposing a bounded uncertainty interval to the DB layer. Instead of "the current time is X", TrueTime says "the current time is X ± 7ms". Spanner waits out the uncertainty at each commit, then assigns a globally-unique timestamp. Now every transaction has a total order, worldwide. Consistency without the coordination cost most systems couldn't afford.

CockroachDB (2015) proved you could do it without atomic clocks — using hybrid logical clocks + Raft consensus. YugabyteDB (2017) did it too, with wire-compatible Postgres. TiDB (2016) did it MySQL-compatible. Suddenly the NewSQL category was real: relational databases that scale like NoSQL. Every startup that wanted to be Google-scale but couldn't give up SQL had an answer.

The core insight: distributed SQL isn't just about scale — it's about not having to think about scale. You write single-region SQL and the database transparently splits data across regions, replicates via Raft, routes reads to nearest replica, and commits with correctness. The mental model stays "one database." The operational reality is dozens of nodes across continents. This is the abstraction the industry chased for 20 years.

What distributed SQL gets you

Horizontal writes

Every node accepts writes. Add nodes → linear scale.

Zero-downtime failover

Raft leader change in ~1s. No manual promotion.

Multi-region correctness

External consistency. No lost writes on partition.

Standard SQL

Postgres/MySQL wire protocol. Your existing tools work.

The cost: higher single-region write latency (Raft round-trip adds 2-10ms). More complex operational model. Not a fit for everything — but a game-changer when it fits.

Historical timeline

1990s
Distributed transactions attempted
IBM Db2, Oracle RAC try. Two-phase commit gets a bad reputation for slow + fragile.
2003
Paxos becomes practical
Google Chubby (based on Paxos) shows how consensus can be used in production for coordination.
2006
Bigtable+Chubby+GFS trilogy
Google's internal infrastructure proves distributed systems can be reliable if you pick right primitives.
2007
Dynamo paper explains AP path
Amazon proves you can go without consensus. NoSQL rebellion begins.
2011
Google F1 built on Spanner
Google's AdWords replaces MySQL sharded fleet with F1 on Spanner. $30B business runs on distributed SQL.
2012
Spanner paper published
"Spanner: Google's Globally-Distributed Database", OSDI 2012. TrueTime. External consistency. Everything changes.
2014
Raft paper (Ongaro & Ousterhout)
Raft is designed to be understandable Paxos. Sparks the modern consensus library ecosystem (etcd, TiKV, HashiCorp Raft).
2015
CockroachDB starts
Ex-Google Andy Kimball + Peter Mattis. Open-source Spanner clone. Uses hybrid logical clocks instead of atomic clocks.
2016
TiDB (PingCAP)
Chinese startup builds MySQL-compatible distributed SQL. Rocket-ships in Chinese internet. Global by 2020.
2017
YugabyteDB open-sourced
Ex-Facebook Kannan Muthukkaruppan. Postgres-compatible distributed SQL. Two APIs (SQL + Cassandra-CQL).
2017
Google Cloud Spanner GA
Spanner leaves Google-internal. Now anyone with a GCP account can rent distributed SQL by the hour.
2019
CockroachDB 19.2 goes multi-region
Data-domiciling regulations (GDPR) push distributed SQL into mainstream — pin data to a region while getting global reads.
2022
Serverless distributed SQL
Cockroach Serverless launches. Pay-per-request tier. Spanner PostgreSQL interface arrives. Neon serverless PG (single-region but similar UX).
2024
AI + distributed SQL converges
Cockroach, Yugabyte, TiDB add vector support. NewSQL becomes the substrate for global AI apps.

Raft consensus: how distributed SQL commits with correctness

Every write in a distributed SQL DB goes through Raft consensus — a leader-based replication protocol designed to be understandable. Watch a commit:

Auto-advances every 2.4s

1. Client proposes write

Client sends INSERT to any node. That node figures out which range this key belongs to, and forwards to the range leader.

Leader

✓ ack

Follower 1

waiting...

Follower 2

waiting...

Majority quorum: With 3 nodes, 2 must ack. With 5, 3. This is why distributed SQL usually uses odd counts (3, 5, 7).

Failure tolerance: 3 nodes tolerates 1 failure. 5 tolerates 2. Data safe as long as majority alive.

TrueTime: how Spanner uses atomic clocks to skip round-trips

Consensus is expensive: 1 round-trip per commit. Google's trick: use TrueTime — globally-synchronized clocks with bounded uncertainty — to avoid coordinating for read-only transactions.

Traditional distributed DB (no TrueTime)

Every commit round-trip. Read transactions need to talk to leader to get commit timestamp. High latency.

Client:   TX begin
Node:     ← ask leader for timestamp (2ms)
Node:     execute reads
Client:   TX commit
Node:     ← Raft consensus (2ms)
          ← durable commit

Spanner with TrueTime

GPS + atomic clocks per DC. TT.now() returns [earliest, latest]. Wait out the uncertainty at commit; assign a globally-total-order timestamp.

Client:  TX begin
Node:    read at TT.now().latest (no wait)
Client:  TX commit
Node:    ts = TT.now().latest
         wait out uncertainty (~7ms)
         ← ts is now GLOBALLY unique
         ← commit

The result: Spanner offers external consistency (strict serializability) globally, with commit latency ~10ms. Reads can be served from any replica at the "safe timestamp" without talking to leader. This is why Spanner powers Google Ads globally.

CockroachDB alternative: Uses hybrid logical clocks (HLC) — logical counter combined with wall clock. Works without atomic hardware. Trade: bounded uncertainty is looser (~500ms in the worst case), which slightly weakens consistency claims.

Auto-sharding: how the DB splits itself as it grows

Distributed SQL divides your table into ranges (Cockroach), tablets (Spanner), or regions (TiDB). Each range is replicated by a Raft group. When ranges grow beyond a size threshold (typically 64-512MB), they auto-split. When traffic is unbalanced, ranges auto-rebalance across nodes.

Table: users (PK: user_id)

Range 1: user_id 0 → 999999

Leader: Node A · Followers: Node B, Node C · Size: 256MB

Range 2: user_id 1000000 → 1999999

Leader: Node B · Followers: Node C, Node D · Size: 380MB

Range 3: user_id 2000000 → 2999999

Leader: Node C · Followers: Node D, Node E · Size: 512MB

Range 3 hits 512MB — auto-splits into Range 3a + 3b

New ranges auto-assigned to under-loaded nodes.

No manual sharding: No shard-key decisions. DB auto-splits based on size + hotspots.

No downtime: Splits happen in the background. Live traffic uninterrupted.

Hot ranges rebalance: If one range gets 10x the traffic, Cockroach's load-based splitter carves it further to spread load.

Multi-region deployment trade-offs

The biggest win of distributed SQL is going multi-region without losing correctness. But multi-region SQL has real latency costs — physics doesn't care about your architecture:

Single-region (default)

Write: 1-3ms

Read: 1-3ms

Tolerates: Single AZ loss OK

Data loss: None

Use when: Start here. Same as running standard PG in one region — but with auto-sharding + zero-downtime failover.

Multi-AZ (same region)

Write: 2-5ms

Read: 1-3ms local, 2-5ms cross-AZ

Tolerates: Any 1 AZ can die

Data loss: Zero

Use when: Baseline HA. All modern managed distributed SQL runs this by default.

Multi-region (leader in one)

Write: 50-100ms (cross-region Raft)

Read: 1-3ms local (follower reads)

Tolerates: Region loss = 2-5min recovery

Data loss: Zero if survivor has quorum

Use when: Global reads with acceptable write latency. Good for user data pinned to home region.

Multi-region (per-row homing)

Write: Fast local for owner region, slow for foreign region

Read: 1-3ms local

Tolerates: Region loss = affected rows unavailable

Data loss: Zero

Use when: GDPR / data domicile. Each row lives in its user's region. Cockroach REGIONAL BY ROW.

Multi-region active-active

Write: 1-3ms local

Read: 1-3ms local

Tolerates: Any region can die

Data loss: Zero

Use when: Extreme HA. But now every write must resolve conflicts at commit — expensive at scale.

Key insight: distributed SQL doesn't abolish physics. Cross-region round-trips still cost 50-100ms. What it does is give you a configurable place on the CAP triangle — pick your topology per table (or per row!) based on your access pattern.

Product comparison

Product	Origin	Compatibility	Strength	Weakness
Google Cloud Spanner	Google (2012)	Spanner SQL + Postgres wire (2022)	Only DB with atomic-clock guarantees. Powers Google Ads. External consistency global.	GCP-only. Expensive at low scale. Slower cold-start than single-region PG.
CockroachDB	Ex-Google (2015)	Postgres wire	Best OSS distributed SQL. Multi-region row-level homing. Serverless tier.	Higher single-node latency than PG. Complex ops if self-hosted.
YugabyteDB	Ex-Facebook (2017)	Postgres wire + Cassandra CQL	Two APIs. YSQL for relational, YCQL for wide-column. Multi-region OSS.	Community smaller than Cockroach. Some Postgres features lag.
TiDB	PingCAP, China (2016)	MySQL wire	MySQL-compatible + HTAP (row store TiKV + column store TiFlash).	MySQL protocol lock-in. Ops still complex outside their cloud.
FoundationDB	Apple (~2013, GA 2018)	KV + SQL layer	Powers iCloud + Snowflake metadata. Rock-solid layered architecture.	Bring-your-own SQL layer. Steep learning curve. Small user community.
PlanetScale (Vitess)	YouTube (2011)	MySQL wire	MySQL at YouTube scale. Branching DBs like Git. Zero-downtime schema changes.	MySQL-only. Foreign keys unsupported in sharded mode.
SingleStore (Memsql)	2011, rebrand 2020	MySQL wire	HTAP with in-memory rowstore + columnar store. Very fast analytics + OLTP.	Cost. Not truly distributed SQL for global HA — more OLTP+OLAP hybrid.
Apache Ignite	GridGain (2014)	SQL + KV	In-memory grid + SQL. Compute + data collocated.	Complex ops. Not competitive with modern distributed SQL for correctness.
MariaDB Xpand	Clustrix (2019 acq)	MySQL wire	Distributed MySQL via Clustrix engine. Multi-writer.	Small user base. Uncertain roadmap.
Amazon Aurora Limitless	AWS (2023)	Postgres + MySQL wire	Sharded Aurora. Postgres/MySQL compatibility. AWS-managed.	Early product. AWS lock-in. Sharding UX still evolving.

How to choose: GCP + strongest consistency → Spanner. Postgres-compatible OSS → CockroachDB or YugabyteDB. MySQL-compatible → PlanetScale or TiDB. AWS-native at massive scale → Aurora Limitless (once mature). Apple-style layered architecture → FoundationDB.

12 real-world distributed SQL deployments

Google Ads

F1 on Spanner — $30B business

Google's AdWords runs on F1, a distributed SQL frontend on Spanner. Every ad auction, every impression, every conversion — ACID transactions across dozens of DCs. Migration from MySQL sharded fleet was 2 years of engineering. F1 handles ~2M transactions/sec globally.

Doordash

CockroachDB for order state

Doordash uses CockroachDB for order state, delivery tracking, dasher assignment. Multi-region for regional resilience. Handles ~4M orders/day. Migrated from vertically-scaled PG when they couldn't fit anymore.

Netflix

CockroachDB for licensing

Netflix uses CockroachDB for the licensing service that tracks which content is available in which country. Multi-region deployment ensures playback decisions are fast + consistent globally.

Shopify

Vitess at 76M req/min (Black Friday)

Shopify runs MySQL + Vitess for merchant data. Vitess shards, replicates, does cross-shard aggregation. Black Friday 2023 peaked at 76M req/min. Zero-downtime schema changes essential.

Ping An Bank

TiDB at real bank scale

Chinese banking giant Ping An runs core transactions on TiDB. Petabytes of data, real banking correctness (regulated). Public case study proves distributed SQL can meet Tier 1 financial requirements.

Apple iCloud

FoundationDB for metadata

Apple acquired FoundationDB in 2015. Now powers metadata for iCloud, Photos, iTunes purchases. Custom SQL layer on top. Handles trillions of KV operations/day with ACID.

Snowflake

FoundationDB for metadata plane

Snowflake's metadata catalog (which shards which files where) runs on FoundationDB. When Snowflake plans a query, it's reading from FDB. Snowflake's entire business depends on FDB's correctness.

Robinhood

CockroachDB for money movement

Robinhood uses CockroachDB for money-movement (buys, sells, transfers). Multi-region for HA. Every transaction crosses at least 3 Raft rounds, but latency is acceptable for financial workloads.

Bilibili

TiDB replaces MySQL sharded fleet

Chinese video platform Bilibili replaced hundreds of MySQL shards with TiDB. Same MySQL wire protocol so app didn't change. Auto-sharding replaced operational sharding pain. Multi-DC deployment.

Deutsche Börse

YugabyteDB for market data

German stock exchange operator uses YugabyteDB for market data reference tables. Regulated environment. Multi-region for DR. Zero data loss requirement mandates distributed SQL semantics.

Wayfair

Multi-region CockroachDB

Wayfair runs multi-region CockroachDB for user profile + cart. US-East + US-West active-active. Cart data survives even during regional outages. Explained in their 2022 engineering blog.

TikTok / ByteDance

Custom + open-source stack

ByteDance built their own ByteHTAP for TikTok scale, but also uses TiDB extensively. The Chinese internet giants have been the largest distributed SQL adopters — nothing else scales to their user counts.

Key takeaways

1Distributed SQL gives you ACID + horizontal scale + zero-downtime failover. The abstraction the industry chased for 20 years.
2The engine is Raft consensus — majority of replicas must ack before commit. 3 nodes tolerates 1 failure; 5 tolerates 2.
3Google Spanner uses TrueTime (atomic clocks + GPS) for external consistency without extra round-trips. Cockroach uses HLC instead.
4Distributed SQL auto-shards: data splits into ranges, each replicated by a Raft group. No manual shard-key decisions.
5Multi-region isn't free. Cross-region writes cost 50-100ms. Use per-row homing or read-follower for latency-sensitive work.
6Postgres-compatible (Cockroach, Yugabyte) means your existing tools work. MySQL-compatible (TiDB, PlanetScale) too.
7For most workloads, single-region PG is still the right start. Reach for distributed SQL when horizontal writes or multi-region correctness demand it.
8The 2020s frontier is serverless distributed SQL (Cockroach Serverless, Spanner PG mode, Aurora Limitless) — scale-to-zero + pay-per-request.

References & further reading

• Corbett, J. et al. (2012). "Spanner: Google's Globally Distributed Database." OSDI. The foundational paper.
• Bacon, D. et al. (2017). "Spanner: Becoming a SQL System." SIGMOD. How Spanner grew into a real DB.
• Ongaro, D. & Ousterhout, J. (2014). "In Search of an Understandable Consensus Algorithm." The Raft paper.
• Shute, J. et al. (2013). "F1: A Distributed SQL Database That Scales." VLDB. AdWords on Spanner.
• CockroachDB Docs: "Architecture — Distribution Layer." Best free introduction to Cockroach internals.
• Kleppmann, M. (2017). Designing Data-Intensive Applications. Chapter 9 (Consistency + Consensus) is essential.
• Lampson, B. (1996). "How to Build a Highly Available System Using Consensus." Foundational thinking.
• Kimball, A. (2015). "Living Without Atomic Clocks." CockroachDB blog on HLC vs TrueTime.
• YugabyteDB Docs: "Architecture Overview." Detailed Raft + storage layer explanation.
• TiDB Docs: "TiDB Architecture." Best Chinese-origin distributed SQL doc set.