Skip to main content
nosql-db

NoSQL Database

Umbrella for document, wide-column, and key-value stores optimized for horizontal scale over strict schema.

Why it exists

SQL databases scale vertically well but struggle horizontally at massive write throughput. NoSQL databases were built to trade some of SQL's guarantees (strict schema, cross-row transactions, complex JOINs) for horizontal scalability and specific workload optimizations. They come in four flavors: document (MongoDB), wide-column (Cassandra, DynamoDB), key-value (Redis-style but persistent), and graph (Neo4j) — each optimized for a different access pattern.

How it works

Document stores store JSON-like blobs with flexible schemas; queries navigate the document structure. Wide-column stores organize data as row keys → many columns, with each column optionally in a different family (physical disk layout); great for time-series and high-write workloads. Key-value stores are simple hashmaps at planetary scale. Underneath: LSM trees for write efficiency, consistent hashing for shard placement, and quorum reads/writes for tunable consistency.

Scaling characteristics

Wide-column and KV stores scale linearly by adding nodes; write throughput per node ~10K-100K ops/sec depending on payload. Read amplification (LSM compactions) can spike disk IOPS. Consistency is tunable: strong reads cost latency + coordination; eventual reads are cheap. Storage is often much cheaper per GB than SQL because there's no need for a page cache large enough to fit indexes.

When to use it

  • High write throughput (>50K writes/sec sustained)
  • Schemaless or rapidly-evolving data models
  • Time-series, event ingestion, activity feeds
  • Global distribution with eventual consistency being fine
  • Simple lookup patterns (by key, by row-key range) — no ad-hoc JOINs needed

When NOT to use it

  • Transactional workloads that need cross-row ACID (payments, inventory)
  • Rich relational queries (JOINs across many tables)
  • Small datasets where the operational cost of a distributed store isn't worth it
  • Teams with no NoSQL experience — the learning curve on modeling and operations is real

Failure modes

  • Hot partitions: a poorly-chosen partition key concentrates traffic on one node — throughput plateaus even though the cluster looks idle
  • Silent data loss during quorum-write with insufficient replicas configured
  • Read-after-write inconsistency confuses application logic — must design for eventual visibility
  • LSM compaction storms cause read latency spikes; needs headroom in disk IOPS
  • Schema migration is per-document (or per-row) — takes a long time on large datasets

Alternatives

  • SQL DB with sharding (Vitess for MySQL, Citus for Postgres) — SQL semantics with horizontal write scale
  • NewSQL (Spanner, CockroachDB) — SQL + horizontal scale, with the consistency trade-off
  • Object storage (S3) for very large blob payloads — cheaper than any database
  • Search engine (Elasticsearch) for full-text + faceted query patterns

Interview questions

  • Choose a partition key for a chat app that stores 100K messages/sec. What's the risk?
  • Explain quorum reads and writes. How would you tune R and W for a write-heavy vs read-heavy load?
  • Your Cassandra cluster has one node at 90% disk usage while others are at 40%. Diagnosis?
  • How do you do a schema evolution (add a new field) in DynamoDB?
  • What are the trade-offs between DynamoDB provisioned and on-demand capacity?
  • You need strong consistency on a write-then-read from the same client. How do you achieve it in Cassandra?

Systems that use this component

See how the real designs on this platform put nosql-db to work — concrete usage context per system.

Twitter/X timeline

intermediate

Manhattan (Twitter's Cassandra fork) for tweets

Open system

Instagram

intermediate

Cassandra for user timeline

Open system

Netflix

intermediate

Cassandra + DynamoDB for viewing history

Open system

WhatsApp

intermediate

Mnesia (Erlang's built-in) for chat metadata

Open system

Ride Sharing

intermediate

Cassandra for trip history and driver metadata

Open system
The NoSQL rebellion

2006, Google: Bigtable proves you can throw SQL away — if you know what you're doing

For 30 years, SQL was the only serious answer to "where do we store data?" Then Google hit a wall. In 2006, they had a petabyte-scale web index across thousands of machines, and no SQL DB could handle it. Their answer was Bigtable — a distributed, sparse, sorted map with millions of columns per row. It gave up joins, transactions, and SQL. It gained linear horizontal scale.

One year later, Amazon published the Dynamo paper (2007) — the shopping cart problem, solved by consistent hashing + vector clocks + eventual consistency. Facebook engineers read Bigtable + Dynamo, mashed them together, and released Cassandra (2008). 10gen released MongoDB (2009) — schemaless documents. LinkedIn released Voldemort. Twitter released FlockDB. Every big tech company had one.

"SQL is dead" declared the blogs of 2010. "NoSQL solves scale." The rebellion was on. Then the reality check: NoSQL trades away things you might actually need. Joins. Transactions. Consistency. Rich queries. Schema safety. Around 2014, the industry started adding SQL back — DynamoDB added transactions (2018), MongoDB added multi-document transactions (2018), Cassandra added lightweight transactions. Distributed SQL (Spanner, CockroachDB) proved you could have both. The rebellion morphed into a pluralism: pick the right tool for each shape.

The core insight: NoSQL is not one thing — it's four categories (KV, wide-column, document, graph) each optimized for a specific data shape and access pattern. Choose based on how you'll query, not on hype. If your data has strict relationships and you need joins, use SQL. If it's a huge stream of independent facts (user events, IoT), a wide-column NoSQL wins. If it's deeply nested and schema-flexible (product catalogs, user profiles), documents. If it's a network of relationships (fraud rings, social graphs), a graph DB.

The four NoSQL shapes
Key-Value
DynamoDB, Redis. Simplest. "Give me the row for this key."
Wide-column
Cassandra, ScyllaDB, HBase. Rows with millions of columns.
Document
MongoDB, CouchDB. JSON documents. Schema-flexible.
Graph
Neo4j, ArangoDB. Nodes + edges. Traversal-native.
Rule of thumb: If you can't sketch your access pattern in one sentence, you probably want SQL. NoSQL rewards clarity about how you'll read.

Historical timeline

  1. 1979
    First KV store: dbm
    Ken Thompson writes dbm for Unix. First key-value database. Simple hash-on-disk. Direct ancestor of every KV store.
  2. 1990
    Object databases emerge
    ObjectStore, GemStone, and others try to escape relational rigidity. Never quite succeed. Set the ground for future rebellions.
  3. 2004
    Berkeley DB dominates embedded KV
    Sleepycat's Berkeley DB (embedded KV) becomes the go-to for high-perf lookups. Later acquired by Oracle.
  4. 2006
    Google Bigtable paper
    "Bigtable: A Distributed Storage System for Structured Data." Sparse, sorted, distributed map. Powers Search, Maps, Gmail. Wide-column category is born.
  5. 2007
    Amazon Dynamo paper
    "Dynamo: Amazon's Highly Available Key-value Store." Consistent hashing + vector clocks + eventual consistency. The shopping cart problem, solved.
  6. 2008
    Cassandra open-sourced by Facebook
    Avinash Lakshman (co-author of Dynamo) fuses Bigtable + Dynamo at Facebook. Handles messages inbox. Apache project 2009.
  7. 2009
    MongoDB 1.0
    10gen releases MongoDB — JSON documents, schemaless. "Web-scale" developer experience. Explodes in popularity.
  8. 2009
    Redis 1.0
    Salvatore Sanfilippo (antirez) releases Redis. In-memory KV store. Data structures beyond strings. Cache + queue + queue in one.
  9. 2012
    AWS DynamoDB launches
    Amazon productizes Dynamo as a managed service. Predictable performance. Pay per capacity. Serverless NoSQL born.
  10. 2014
    The counter-revolution: distributed SQL
    CockroachDB starts. Spanner exits Google-internal. Distributed SQL proves NoSQL's scale story isn't exclusive to NoSQL.
  11. 2018
    NoSQL gets transactions
    DynamoDB adds transactions. MongoDB adds multi-document transactions. Cassandra has LWT. NoSQL isn't as anti-ACID as it once claimed.
  12. 2020
    ScyllaDB matures
    C++ Cassandra rewrite hits GA. 10x throughput on same hardware. Discord and others migrate massive workloads.
  13. 2023
    MongoDB 7.0 + Vector Search
    MongoDB adds native vector indexes (Atlas). NoSQL becomes AI-native. Neo4j adds graph-based RAG patterns.
  14. 2024
    DuckDB reshapes analytics
    OLAP in-process. Not classically NoSQL but redefines what a "non-relational" DB can look like at the analytics layer.

The four NoSQL data models — click to explore

NoSQL isn't one thing. Each of these four shapes is optimized for a specific data topology. Match the model to your access pattern:

Key-Value

Shape
{
  "user:42": { "name": "Alice", "email": "a@x.com" },
  "user:43": { "name": "Bob", ... },
  "session:xyz": { "userId": 42, "exp": ... }
}
Query: GET user:42
Strengths: O(1) lookup by key. Trivially horizontal. Simplest possible mental model.
Weaknesses: Can only query by key. No relationships. Range/scan support varies.
Use for: Session cache, feature flags, user profiles by ID, config, rate-limit counters.
Products: DynamoDB, Redis, Memcached, etcd, Riak KV, BerkeleyDB, LevelDB, RocksDB

CAP theorem in practice — when the network splits

Brewer's CAP theorem says: in a distributed DB, when the network partitions, you can pick Consistency (reject writes to keep data correct) or Availability (accept writes at the cost of temporary inconsistency). You can't have both. Try it:

Region A (majority)
Node 1, Node 2, Node 3
✓ writes accepted
↔ all nodes talk
Region B (minority)
Node 4, Node 5
✓ writes accepted
Both regions reachable. All writes committed. Consistent + Available. No trade-off exists when the network works.
CP examples: MongoDB (majority reads), HBase, ZooKeeper, etcd, Spanner. Use when correctness > availability. Banking, inventory, primary key uniqueness.
AP examples: DynamoDB (default), Cassandra, Riak, CouchDB. Use when uptime > instant consistency. Shopping carts, social media, activity feeds, IoT ingestion.

Tunable consistency — pick per-query, not per-database

Modern NoSQL (Cassandra, DynamoDB, MongoDB) lets you choose consistency per operation. Same DB, same table, different consistency levels for different queries. Compose freely:

ONE / eventual

Latency: Lowest (~1-5ms)
Write: 1 replica confirms
Read: 1 replica returns
Risk: Read stale data. May miss recent writes.
Use: Analytics, logs, non-critical activity feeds.

QUORUM (majority)

Latency: Medium (~5-20ms)
Write: N/2 + 1 confirms
Read: N/2 + 1 respond
Risk: Reads see last committed write. Balanced.
Use: Most workloads. Sweet spot for Cassandra + DynamoDB.

ALL / strong

Latency: Highest (~50ms+)
Write: All replicas confirm
Read: All replicas respond
Risk: Any node down = write fails. Fragile.
Use: Rare — better to reduce replication factor if you truly need this.

LOCAL_QUORUM (multi-DC)

Latency: Low intra-DC (~5ms)
Write: Majority in local DC only
Read: Majority in local DC only
Risk: Different DCs can diverge. Reconcile async.
Use: Multi-region apps where cross-region latency is unacceptable.

SERIAL (LWT)

Latency: Very high (~50-200ms)
Write: Paxos consensus
Read: Paxos consensus
Risk: Adds contention.
Use: Rare hot-path (unique constraints, compare-and-set). Not for volume.
The 90/10 rule: in a Cassandra fleet, ~90% of writes use QUORUM, ~10% use ONE for analytics that can tolerate lag. Rarely ALL. Almost never SERIAL. This mix optimizes cost + latency.

Partitioning: how NoSQL scales horizontally

Every NoSQL row lives on exactly one partition (with replicas). The partition key + hash function decides which node stores it. Type a key + change shard count:

Try: order:99, tenant:acme, session:xyz
Routed to shard
Shard 0
Hash of user:42 = 1170328996 % 4 = 0
S0
S1
S2
S3
Hash partitioning (DynamoDB, Cassandra): uniform distribution, no hot spots per-key. But no range queries across keys.
Range partitioning (Bigtable, HBase): keys sorted globally. Range scans blazing fast. Hot-key risk if writes cluster on one range.
Consistent hashing (from Dynamo): When you add/remove nodes, only a fraction of keys move — not the whole dataset. This is the trick that makes NoSQL rebalance affordable. See the Consistent Hashing deep dive →

Product comparison

ProductTypeModelStrengthWeakness
DynamoDBKV / docManagedPredictable p99 latency. Serverless. Auto-scale. Global tables.Query patterns must be modeled up-front. Expensive at very high RCU/WCU.
CassandraWide-columnOSSLinear write scale. Multi-DC native. No masters.JVM tuning nightmare. Repair pain. Steep learning curve.
ScyllaDBWide-columnOSS + EnterpriseC++ Cassandra rewrite — 10x throughput. Same API.Younger ecosystem. Some Cassandra features lag.
MongoDBDocumentOSS + AtlasBest DX for document data. Rich query language. Multi-doc transactions.Historic reliability critiques (better now). Sharding UX complex.
HBaseWide-columnOSSHadoop-native. Strong consistency (CP). Massive scale at Yahoo/Facebook.Requires Hadoop + Zookeeper. Ops complexity. Declining momentum.
RedisKV + data structsOSS + ManagedFastest KV. Streams, pub/sub, sorted sets — Swiss Army knife.RAM-only for hot data. Persistence trade-offs. Single-writer per shard.
Riak KVKVOSSOriginal Dynamo-style AP. Rock-solid replication. Was Basho's flagship.Basho went bankrupt. Community small. Not recommended for new projects.
CouchDBDocumentOSSMVCC + multi-master sync. Powers PouchDB (offline-first apps).Older query experience. Less popular than MongoDB.
Neo4jGraphOSS + EnterpriseBest-in-class graph DB. Cypher query language. Rich tooling.Enterprise features gated. Single-writer for OSS. Sharding via Fabric (Enterprise).
ArangoDBMulti-modelOSS + EnterpriseDocuments + graphs + KV in one engine. AQL is powerful.Fewer users than specialized DBs. Not best-in-class at any one shape.
BigtableWide-columnManaged (GCP)Google's original wide-column. Ultra-low-latency at massive scale.GCP-only. No transactions. Complex data modeling.
FirestoreDocumentManaged (GCP)Real-time subscriptions. Offline-first mobile. Perfect for Firebase apps.Query limitations. Cost surprises. Vendor lock-in.

How to choose: AWS + serverless → DynamoDB. Time-series or IoT → Cassandra/ScyllaDB. Nested docs → MongoDB. Real-time mobile → Firestore. Fraud/social graph → Neo4j. Ultra-low latency cache → Redis. Multi-DC without ops → managed (DynamoDB, Atlas, MongoDB Cloud).

12 real-world NoSQL deployments

Amazon

DynamoDB serves the entire retail catalog

Amazon retail runs on DynamoDB for everything from the shopping cart to product catalog to Prime membership. >40M requests/sec at Prime Day peaks. Single-digit-millisecond p99. Each service owns its tables, uses single-table design for related entities. Zero operational overhead — no server admin.

Netflix

Cassandra: 200 clusters, 30PB

Netflix runs 200+ Cassandra clusters serving user profiles, viewing history, personalization. ~30PB total. Multi-region active-active with LOCAL_QUORUM. Their custom Priam tool automates ops. When you press play, at least 5 Cassandra reads happen before the video starts.

Discord

ScyllaDB migration: from PB of Cassandra

Discord messages sit in ScyllaDB after migrating from Cassandra. Reason: 90% cost reduction + 10x throughput on same nodes. Their "How Discord Stores Trillions of Messages" blog is a NoSQL migration classic. Now serves 1T+ messages.

Facebook

HBase for messages, Cassandra's birthplace

Facebook built Cassandra for the messages inbox in 2008, then replaced it with HBase because they wanted stronger consistency. Both invented at Facebook! Today HBase serves messages, insights. Cassandra thrives outside Facebook.

Instagram (Meta)

Cassandra for feed ranking data

Instagram uses Cassandra for user activity, feed candidate stores. Every action (like, follow, view) writes to Cassandra. The ranking service reads from it to build personalized feeds. Handles 10M writes/sec at peak.

Uber

Custom NoSQL (Schemaless) on MySQL

Uber's Schemaless is a NoSQL layer built on MySQL. Provides KV semantics with schema-less values. Handles trip data at massive scale. Combines MySQL's ops maturity with NoSQL's schema flexibility.

Slack

Vitess-backed MySQL + custom NoSQL

Slack runs channels + messages on Vitess (MySQL) but uses custom NoSQL sharding for very hot workloads. Learned from many years of Cassandra pain. Their conclusion: SQL when you can, NoSQL when you must.

Spotify

Cassandra + PostgreSQL hybrid

Spotify stores user library and taste graph in Cassandra, while playlists, users, artists live in PostgreSQL. Cassandra for high-write user-events; PG for entities. Now migrating some to their own Bigtable-based store.

Snapchat

Bigtable for the vanishing-message model

Snapchat runs on Google Bigtable. Every snap = a row keyed by (user_id, timestamp). Trivial TTL for auto-deletion. Petabyte scale, sub-millisecond reads. Native GCP integration means minimal ops.

LinkedIn

Espresso: document NoSQL for member data

LinkedIn built Espresso — a MySQL-based document store — for member profiles, activity, messaging. Custom because MongoDB wasn't reliable at their scale in 2013. Powers ~1B users' profile lookups.

Trulia

Neo4j for real estate graph

Trulia uses Neo4j to model relationships between properties, neighborhoods, schools, agents. Complex "find homes near good schools in walkable areas" queries — traversals that would kill a relational join, run in ms on graph.

eBay

Cassandra + MongoDB for different shapes

eBay runs Cassandra for the seller catalog (high-write, time-partitioned) and MongoDB for user preferences (nested, schema-flexible). One team, two DBs, right tool per shape. A common enterprise pattern.

Key takeaways

  • 1NoSQL isn't one thing — it's four categories (KV, wide-column, document, graph). Match model to access pattern.
  • 2The Bigtable + Dynamo papers (2006-2007) started the movement. Every modern NoSQL is a descendant of one or both.
  • 3CAP theorem in practice: during partition, choose CP (reject on minority) or AP (accept both, reconcile). No fifth option.
  • 4Tunable consistency is your friend — 90% QUORUM, 10% ONE for analytics. Pick per-query, not per-database.
  • 5Partition keys define scale. Model access patterns first, choose partition key to distribute load and enable your queries.
  • 6The 2018 wave of transactions (DynamoDB, MongoDB) mostly closed the correctness gap. NoSQL isn't anti-ACID anymore.
  • 7For most workloads, start with SQL. Reach for NoSQL when you have specific shape/scale needs SQL can't handle.
  • 8Managed NoSQL (DynamoDB, Atlas, Firestore) removes 80% of the operational pain. Self-hosted Cassandra is a full-time job.

References & further reading

  • • Chang, F. et al. (2006). "Bigtable: A Distributed Storage System for Structured Data." OSDI. The paper that launched wide-column.
  • • DeCandia, G. et al. (2007). "Dynamo: Amazon's Highly Available Key-value Store." SOSP. The paper that launched AP NoSQL.
  • • Brewer, E. (2000). "Towards Robust Distributed Systems." The CAP theorem talk. Read alongside Gilbert & Lynch (2002) formal proof.
  • • Vogels, W. (2009). "Eventually Consistent." CACM. Amazon CTO's explanation of eventual consistency in production.
  • • Kleppmann, M. (2017). Designing Data-Intensive Applications. Chapters 5-6 on replication + partitioning are canonical.
  • • Sivasubramanian, S. et al. (2017). "Amazon Aurora" SIGMOD paper — for the counter-narrative.
  • • Lakshman, A. & Malik, P. (2010). "Cassandra: A Decentralized Structured Storage System." The Cassandra paper.
  • • MongoDB blog: "MongoDB's New Aggregation Framework" and the multi-doc transactions announcement.
  • • Discord Engineering: "How Discord Stores Trillions of Messages" — ScyllaDB migration classic.
  • • Netflix Tech Blog: "Cassandra Backups and Restores at Netflix" — operational reality of Cassandra at scale.
  • • Uber Engineering: "Schemaless: Uber Engineering's Trip Datastore." Custom NoSQL on MySQL.