Kafka — from ZooKeeper to KRaft

For its first 10 years, Apache Kafka needed ZooKeeper. Broker registration, controller election, topic metadata, ACLs, quotas, partition assignments — every one of those pieces of cluster state lived in a ZooKeeper ensemble running alongside the Kafka brokers. If you ran Kafka in production, you ran ZooKeeper too. Two clusters, two failure modes, two upgrade paths, twice the on-call pain.

In April 2021, Kafka 2.8 shipped KRaft mode (KIP-500) — Kafka's own embedded Raft implementation that finally let Kafka manage its own metadata without an external consensus service. Kafka 3.3 (Oct 2022) marked KRaft production-ready. Kafka 4.0 (March 2025) made KRaft the only supported mode — ZooKeeper support was removed entirely. A 15-year architectural dependency dissolved.

The change wasn't just about killing ZooKeeper. It was about making Kafka's control plane look like its data plane: an event log that every controller replays, managed by the same familiar Kafka producer/consumer/replication machinery. Once you frame it that way, using Kafka's own battle-tested log to store metadata about Kafka is the obvious move — the puzzle is only why it took a decade.

What ZooKeeper actually did for Kafka

Before KRaft, ZooKeeper handled everything Kafka couldn't do for itself:

• Controller election — one broker was elected "controller" via a ZooKeeper ephemeral znode (/controller). Whoever created it first won. When the controller died, the znode disappeared and everyone raced to recreate it.
• Broker registration — each broker created an ephemeral znode under /brokers/ids/N. Session timeout → znode gone → broker considered dead.
• Topic metadata — /config/topics/T plus /brokers/topics/T/partitions/P. Partition-to-broker assignments, replication factors, ISR lists.
• ACLs and quotas — user permissions, throughput limits, client rate limits.
• Coordination watch notifications — every broker set watches on many znodes; ZooKeeper pushed change notifications so brokers could reload metadata.

This worked. But at large scale it broke in specific ways. Controller failover took tens of seconds in clusters with 200k+ partitions because the new controller had to read all metadata from ZooKeeper serially and re-establish watches. ZooKeeper's own quorum was a separate operational problem — with its own upgrades, security patches, network configuration, and monitoring. And for small deployments, you ran three ZooKeeper nodes purely to manage one broker. That's a lot of accidental complexity.

The KRaft architecture

In KRaft mode, some brokers run as controllers — typically 3 or 5, forming a Raft quorum. One is the active controller; the rest are standbys tailing the log. The metadata itself lives in a special internal topic called __cluster_metadata, which is just a Kafka topic — the same log-structured, replicated thing every other topic is — but managed by KRaft's consensus module rather than a normal broker.

Every metadata change is written as a record in the metadata log: "topic X created," "broker 7 joined," "partition P's ISR is now {5, 8, 12}." The active controller appends the record, replicates it via Raft, and once quorum is reached the record is committed. Every other broker in the cluster replays the metadata log as a passive follower — they don't need to be told individually about changes; they read them off the log like any Kafka consumer.

This is the elegant part: the whole cluster's state is a replicated log, and every broker's view of "current cluster state" is just its replayed position in that log. There's no watch-based invalidation. No cache coherence problem. The log is the source of truth.

Controller election in KRaft

The 3 or 5 controller nodes run standard Raft: heartbeats, election timeouts, terms (Kafka calls them epochs). When the active controller dies, another controller times out, becomes candidate, requests votes, wins, and becomes active. Because it was already tailing the metadata log as a Raft follower, it's already caught up — no cold-start metadata read from ZooKeeper. Failover is typically < 1 second even in massive clusters.

The migration timeline — why it took 6 years

KIP-500 was proposed in 2019. Kafka 4.0 finally removed ZooKeeper support in 2025. That's a long time for a "just replace ZooKeeper" change — because it was never just that. It required Kafka to grow a correct Raft implementation, a migration path from thousands of existing production ZooKeeper deployments, and a bridge mode where clusters ran with both systems simultaneously during upgrade.

Confluent's enterprise customers reported that clusters that used to take 30-60 seconds to recover from a controller failure (200k+ partition environments) now recover in < 1 second. Small deployments no longer need to run 3 extra ZooKeeper nodes for a single Kafka broker. Operators get one system to patch and monitor instead of two.

Partition leader election — the other consensus in Kafka

There's a common confusion worth clearing up: Kafka has TWO layers of leadership.

• Controller election — which broker manages cluster metadata. This is what KRaft (or ZooKeeper) does. There is one active controller per cluster.
• Partition leader election — for each of the thousands of partitions, which broker is the leader replica that serves reads and writes. This is NOT a Raft election — it's a simple assignment written by the active controller into the metadata log. The controller picks a leader from the partition's ISR (In-Sync Replica set) and everyone reads that assignment.

So Raft happens once per cluster (for controllers), not once per partition. This is fundamentally different from CockroachDB or TiKV, which run MultiRaft — one Raft group per range. Kafka's partition replication uses ISR (In-Sync Replicas) semantics — related to Raft but simpler, since the controller does the deciding.

Where KRaft runs in production

The mental model — Kafka KRaft in one paragraph

Key takeaways

• KRaft eliminates the ZooKeeper dependency. One system to run, patch, and monitor instead of two.
• Kafka has two layers of leadership: KRaft picks the cluster controller; the controller writes partition leader assignments into the metadata log. Only the first uses Raft.
• Cluster metadata is stored in __cluster_metadata — a Kafka topic with Raft-driven replication. Every broker tails it. The log is the source of truth.
• Controller failover: < 1 second in KRaft vs 15-60 seconds in ZooKeeper mode for large clusters — because standby controllers are already caught up.
• Kafka 2.8 (2021) shipped KRaft as preview. Kafka 3.3 (Oct 2022) declared it production-ready. Kafka 4.0 (March 2025) removed ZooKeeper entirely.
• KRaft is not MultiRaft — it runs one Raft group per cluster (for the controllers), not one per partition. Partition replication uses ISR semantics instead.

References

• KIP-500 (2019) — Replace ZooKeeper with a Self-Managed Metadata Quorum.
• KIP-595 (2020) — A Raft Protocol for the Metadata Quorum.
• KIP-631 (2020) — The Quorum-based Kafka Controller.
• KIP-833 (2022) — Mark KRaft as Production-Ready.
• Kafka 4.0 release notes (2025) — ZooKeeper support removed.
• Colin McCabe — "Apache Kafka Made Simple" (Confluent blog, 2020) — the KIP-500 vision essay.
• Ongaro & Ousterhout (2014) — "In Search of an Understandable Consensus Algorithm." The Raft paper KRaft is based on.