Distributed Lock (ZooKeeper, etcd, Redis Redlock)
Cross-node mutual exclusion — 'only one node can do this at a time' when you can't rely on a single-node lock.
Why it exists
Some things must happen exactly once across a fleet: 'only one scheduler triggers the nightly job', 'only one worker processes this order', 'only one leader accepts writes'. On a single machine this is a mutex. Across N machines, you need a distributed lock — a coordination primitive backed by a consensus system (or clever cache tricks). It's one of the hardest primitives in distributed systems, and one of the easiest to get subtly wrong.
How it works
The lock is a key in a distributed store; acquiring the lock means atomically writing your ID with a TTL if no one else holds it. Consensus-based locks (ZooKeeper, etcd) use Paxos/Raft — a lock is a znode/kv with a lease; the client renews the lease periodically; if the client dies, the lease expires and the lock is available. Redis Redlock uses a majority-vote across N independent Redis nodes with a short TTL; the algorithm has known issues but is fast. Fencing tokens (monotonic numbers issued at lock acquisition) prevent 'zombie' locks from corrupting downstream state.
Scaling characteristics
Locks are inherently sequential — throughput is bounded by lock acquire+release rate. ZooKeeper handles ~10K lock ops/sec per cluster; etcd is similar. Redis Redlock is faster (~50K/sec) but with weaker guarantees. Lock contention (many nodes wanting the same lock) kills throughput; design to minimize contention.
When to use it
- Leader election (one primary node in a cluster)
- Distributed cron (only one scheduler fires each trigger)
- Guaranteed single-execution of critical operations
- Coordination during rolling deploys (one instance at a time)
- Rate limiting shared across instances (with care)
When NOT to use it
- Anywhere you can design without a lock — e.g., partition-and-process (each node owns some shards, no contention)
- Fine-grained locking on hot resources (item, user) — high contention kills you; use optimistic concurrency instead
- When correctness depends on 'lock forever' semantics — locks with TTLs can be released while the holder still thinks they own it
Failure modes
- Split-brain: two nodes both think they hold the lock (e.g., holder's GC pause exceeds TTL). Mitigate with fencing tokens.
- Lock service outage stops all coordination. Design for graceful degradation.
- TTL too short → holder loses lock while still working. Too long → dead holder blocks progress.
- Redis Redlock consistency issues under clock skew are famously debated (Martin Kleppmann vs antirez).
- Watchdog process fails to renew lease → holder loses lock silently.
Alternatives
- Optimistic concurrency (CAS) at the database — often what you actually want
- Partition-and-process (each node owns its shards; no lock needed) — the right answer if you can arrange it
- Kafka consumer groups — implicit leader election for partition assignment
- Cloud-native leader election (Kubernetes lease objects) — clean if you're on K8s
Interview questions
- Explain why a naive Redis lock (SETNX with TTL) can allow two clients to think they hold the lock. Fix?
- What's a fencing token, and how does it prevent zombie locks?
- Compare ZooKeeper and etcd for a distributed lock use case.
- Design a distributed cron system where each of 10 nodes might try to run the job.
- Your lock holder is GC-paused for 30 seconds. The TTL is 20 seconds. What happens, and what's the fix?
- When would you use optimistic concurrency instead of a distributed lock?
Systems that use this component
See how the real designs on this platform put distributed-lock to work — concrete usage context per system.
1978, Lamport: the paper that proved distributed locks are harder than you think
Leslie Lamport's 1978 paper — "Time, Clocks, and the Ordering of Events in a Distributed System" — set the stage. His central insight: in a distributed system, you cannot trust wall clocks. Two processes might disagree about which event happened first, and no amount of NTP sync will fully save you. Yet many operations depend on a strict order — who acquired the lock first, which write wins, whose transaction commits.
The 1985 FLP Impossibility Theorem (Fischer, Lynch, Paterson) went further: no deterministic algorithm can guarantee consensus in an asynchronous system with even one crash failure. This is the bad news. But real systems make progress anyway using either probabilistic algorithms (Paxos, Raft) or synchronizing assumptions (leases with timeouts).
The industrial answer came in stages. Google Chubby (2006) — Paxos-based, coarse-grained leases. Apache ZooKeeper (2008) — open-source Chubby-alike, powered Hadoop. etcd (2013) — Raft-based, simpler API, powers Kubernetes. Meanwhile, developers reaching for "quick" locks used Redis SETNX — and started fighting the correctness pitfalls that Antirez documented in his 2016 Redlock proposal, which Martin Kleppmann then critiqued in one of the most-shared distributed-systems posts of the decade.
The core lesson: if you need a distributed lock, you probably need less lock than you think. First ask: can I redesign to be idempotent? If yes, no lock. If no, use a lock service that gives you fencing tokens (monotonic epochs) — because even the "correct" lock can't save you from a client that pauses for GC longer than the lease. Redis without fencing is unsafe. ZooKeeper and etcd with fencing are safe.
Historical timeline
- 1978Lamport's clocks paper"Time, Clocks, and the Ordering of Events." Foundational. Introduces logical clocks + happened-before.
- 1985FLP impossibility theoremFischer, Lynch, Paterson prove that no deterministic async consensus algorithm can tolerate even 1 crash. The bad news.
- 1988Paxos published (delayed)Lamport's Paxos paper is written, but famously hard to understand. Sits mostly unread for years.
- 1989Leases introducedGoogle's Gray + Cheriton propose leases: locks with expiry. The key primitive that makes distributed locks practical.
- 2001Paxos Made SimpleLamport writes a friendlier version. Industry finally understands + adopts.
- 2006Google Chubby paperMike Burrows publishes "The Chubby Lock Service." Paxos-based lock service inside Google. Powers GFS, Bigtable, MapReduce.
- 2008Apache ZooKeeperYahoo builds an open-source Chubby-alike. Becomes the coordination service for Hadoop, HBase, Kafka.
- 2013etcd released by CoreOSRaft-based, HTTP + JSON API. Simpler than ZooKeeper. Becomes the coordination substrate of Kubernetes.
- 2014Raft publishedOngaro + Ousterhout's "In Search of an Understandable Consensus Algorithm." Paxos' approachable cousin.
- 2016Antirez publishes RedlockRedis-based distributed lock algorithm. Widely adopted for its simplicity.
- 2016Kleppmann's Redlock critique"How to do distributed locking" — Kleppmann shows Redlock breaks under GC pauses + clock drift. Sparks industry debate.
- 2016Antirez's responseAntirez rebuts. The debate becomes a canonical "how to think about distributed correctness" case study.
- 2019Consul KV as lock serviceHashiCorp Consul adds first-class distributed locks via session-based KV. Popular for microservice coordination.
- 2024Postgres advisory locks resurgeModern advice: for many workloads, PG advisory locks are the simplest, safest distributed lock — just use your existing DB.
The classic bug: why a lock without fencing is unsafe
This is Martin Kleppmann's classic scenario — the reason the Redis vs ZooKeeper debate matters. Watch a pause-based race condition, and how fencing tokens fix it:
1. Client A acquires lock
Client A calls acquire("job-42"). Lock service records: A owns job-42 for 30s.
Fencing tokens — the fix for the pause bug
A fencing token is a monotonically increasing number returned by the lock service on every acquire. Every write to protected storage includes the token. Storage refuses writes with a token smaller than the largest it has seen. Even if a paused client returns with a stale lock, its writes are rejected. This is what makes ZooKeeper, etcd, and Chubby safe — and what Redis lacks by default.
Without fencing (Redis SETNX naive)
ok = lock.acquire("job-42")
// ... GC pause 40s here ...
storage.write("result", data)
// storage has no idea we lost the lock
// clobbers whoever won meanwhileVulnerable to any pause > lease TTL. GC, network hiccup, VM suspend — all cause data corruption.
With fencing (ZooKeeper zxid, etcd revision)
token = lock.acquire("job-42") // e.g. 42
// ... GC pause 40s here ...
storage.write("result", data, token=42)
// storage: last-seen token was 43 (B got it)
// REJECTED. Client A learns it lost.Storage is the arbiter — even if the lock service is confused, the token comparison saves correctness.
The Redlock controversy: a distributed-systems classic
In 2016, Antirez (creator of Redis) published Redlock: an algorithm using N=5 Redis nodes with majority quorum for distributed locking. Martin Kleppmann (author of DDIA) published a critique titled "How to do distributed locking" that shook the community. The debate is worth reading in full — but here's the summary:
Antirez's Redlock claim
- • Get lock from majority (3 of 5) Redis nodes
- • Each with the same key + TTL
- • If total elapsed time < TTL → lock held
- • Works even if some Redis nodes crash
Kleppmann's critique
- • Depends on wall-clock timing (bad!) — clock jumps break it
- • GC pause > TTL → wrong client thinks it holds lock
- • No fencing token → storage can't validate ownership
- • Redlock protects "efficiency" not "correctness"
Eight patterns — which lock service to reach for
Not every problem needs a full-blown consensus system. Match the pattern to your risk profile:
Advisory lock (Postgres)
Single-Redis SETNX
Redlock (multi-Redis)
ZooKeeper ephemeral node
etcd lease + revision
Consul session-based KV
DynamoDB conditional writes
Cloud provider lock services
Product comparison
| Product | Model | Strength | Weakness |
|---|---|---|---|
| Apache ZooKeeper | OSS | The industry standard for correctness locks. Battle-tested. Rich primitives (ephemeral, sequential, watches). | JVM ops. Session timeout complexity. Older API. |
| etcd | OSS | Modern Raft-based. Simple HTTP + gRPC. Powers Kubernetes. Great for cloud-native. | Fewer features than ZK (no watches for arbitrary paths pre-v3). |
| Consul | OSS + Enterprise | Service discovery + KV + locks in one. Multi-DC. HashiCorp support. | License changed (2023). Overkill just for locks. |
| Google Chubby | Google internal | The original. Paxos-based. Powers all Google infra. | Not available externally. |
| Redis (single) | OSS + Managed | Ubiquitous. Simple. Fast (sub-ms). | Not safe for correctness locks. Redlock critique. |
| Redlock (multi-Redis) | OSS | Redis-based distributed lock. Better than single-node Redis for efficiency. | Kleppmann-critiqued. Still not safe for correctness. |
| Hazelcast IMap | OSS + Enterprise | In-memory data grid with lock semantics. JVM-friendly. | JVM cluster ops. Split-brain concerns. |
| PostgreSQL advisory locks | OSS | Free. Uses your existing DB. No new infra. | Session-scoped. Not for cross-DB coordination. |
| DynamoDB conditional writes | Managed (AWS) | AWS-managed. Safe with version fencing. Serverless. | AWS-only. Requires app-side lease pattern. |
| Azure Blob leases | Managed (Azure) | Built into Blob storage. 15-60s leases. Auto-fenced. | Azure-only. Blob-scoped only. |
How to choose: Kubernetes-native → etcd. HashiCorp shop → Consul. Kafka/Hadoop legacy → ZooKeeper. Simple app-level → PG advisory locks. AWS-native → DynamoDB conditional. Cheap efficiency lock → Redis (with awareness). Correctness lock → ZK or etcd, always with fencing.
12 real-world distributed lock deployments
etcd for leader election of every controller
Every Kubernetes controller (scheduler, controller-manager, etc.) elects a leader via etcd leases. Only the leader takes actions. On leader failure, another instance takes over in seconds. This is what makes K8s HA.
ZooKeeper for Kafka broker coordination
Kafka relied on ZooKeeper for broker discovery, controller election, and topic metadata for >10 years. Recent versions moved to KRaft (Kafka's own Raft implementation), but ZK is still deployed at massive scale.
ZooKeeper birthplace
ZooKeeper was born at Yahoo! to coordinate Hadoop clusters. Named after "the zoo" of animals (Hadoop, Pig, Hive) it coordinated. Battle-tested on hundreds of thousands of nodes.
ZooKeeper via Curator library
Netflix built Apache Curator — a client framework on top of ZooKeeper — to hide the sharp edges (session handling, retry policies). Used across most of Netflix's Java-based services.
Zookeeper for MySQL primary election
Uber uses ZooKeeper (via Orchestrator) to handle MySQL primary failover. When the primary dies, ZK-based coordination picks a new one. Sub-30s failover.
Chubby for tablet server leadership
Every Bigtable tablet server acquires a Chubby lock to serve its tablets. On lease expiry, the tablet is reassigned. This is what makes Bigtable auto-recover from server crashes.
Consul for HA + backend storage
Vault (secret management) uses Consul for HA — the active instance holds a Consul lock; others stand by. Consul also serves as one of Vault's backend storage options.
Postgres advisory locks for job runners
GitHub Actions uses Postgres advisory locks to coordinate job runner assignment. One job = one runner picking it up. Simple, no new infra. Scales because PG handles millions of QPS.
Idempotency keys instead of locks
Stripe famously avoids distributed locks by using idempotency keys. Every mutation includes a client-generated key; server dedupes at the DB layer. Simpler + safer than any lock service.
Redis + fencing for order lock
Shopify uses Redis-based distributed locks for coordination but layers compare-and-swap on the DB row as fencing. If the Redis lock lease expired, the DB write fails. Belt + suspenders.
Kinesis Client Library uses DynamoDB for lease coordination
KCL (Kinesis Client Library) uses a DynamoDB table for lease coordination — which worker owns which shard. Version fencing on the lease record prevents dual-processing.
Bootstrap lockfile via etcdadm
Even etcd itself uses a distributed-lock approach during cluster bootstrap: one node writes a "discovery URL" race, first wins. This is coordination-of-the-coordinator.
Key takeaways
- 1First rule of distributed locks: don't use one if you don't have to. Idempotent operations don't need coordination.
- 2Wall clocks lie in distributed systems. TTL-based locks without fencing are correctness bugs waiting to happen.
- 3Fencing tokens (monotonic epochs) are the fix. Storage validates: "is my token >= last-seen?"
- 4Redis SETNX + Redlock give you an efficiency lock (avoid duplicate work), not a correctness lock. Understand the difference.
- 5For correctness, use ZooKeeper, etcd, or Consul — consensus-backed with fencing. Or Postgres advisory + row versions for simpler cases.
- 6The Kleppmann vs Antirez debate is the canonical text on how to think about correctness in distributed systems. Read both sides.
- 7Cloud-native shops typically use etcd. Enterprise Java shops use ZooKeeper. HashiCorp shops use Consul.
- 8Better than any distributed lock: use idempotency keys + database compare-and-swap on version columns. Stripe made this famous.
References & further reading
- • Lamport, L. (1978). "Time, Clocks, and the Ordering of Events in a Distributed System." CACM.
- • Fischer, M., Lynch, N., Paterson, M. (1985). "Impossibility of Distributed Consensus with One Faulty Process." The FLP paper.
- • Gray, C. & Cheriton, D. (1989). "Leases: An Efficient Fault-Tolerant Mechanism for Distributed File Cache Consistency." SOSP. Introduces leases.
- • Burrows, M. (2006). "The Chubby Lock Service for Loosely-Coupled Distributed Systems." OSDI. The Chubby paper.
- • Junqueira, F., Reed, B. (2013). ZooKeeper: Distributed Process Coordination. O'Reilly. Best ZooKeeper book.
- • Kleppmann, M. (2016). "How to do distributed locking." Blog post. The Redlock critique.
- • Antirez, S. (2016). "Is Redlock safe?" The reply.
- • Ongaro, D. & Ousterhout, J. (2014). "In Search of an Understandable Consensus Algorithm." Raft.
- • Kleppmann, M. (2017). DDIA, chapter 8 (The Trouble with Distributed Systems), chapter 9 (Consistency + Consensus).
- • etcd Docs: "Distributed locks with etcd." Concrete recipes.