stream

Kafka

A distributed, partitioned, replicated commit log for event streaming, high-throughput ingest, and decoupled services.

Why it exists

When you want to decouple producers from consumers, absorb bursts, replay history, and fan out the same event to N different downstream systems — traditional message queues (RabbitMQ, SQS) run out of gas. Kafka reframes the problem as an ordered, durable, replayable log. It's the backbone of most modern event-driven architectures.

How it works

Kafka stores messages in topics, which are split into partitions. Each partition is an append-only log on disk (fast sequential writes). Producers append to a partition (chosen by key hash for ordering per key); consumers read from an offset they track themselves. Replication is via leader-follower per partition. Consumers organize into groups; each partition is consumed by exactly one consumer in the group, providing parallelism. Retention is time- or size-based, not consumption-based — messages stay until they age out.

Scaling characteristics

A single Kafka broker handles ~100 MB/s per partition on commodity SSD, scaling to ~1 GB/s per broker with tuning. Partitions are the unit of scale: 100 partitions × 4 brokers gives ~400 MB/s throughput. Latency is ~5-10ms end-to-end for producers with acks=all. Storage is cheap — Kafka is often used for months of history.

When to use it

Event streaming for pub-sub with replay
High-throughput ingest (100K+ events/sec) that decouples from processing
Change Data Capture (Debezium → Kafka) for downstream sync
Building an event-sourced system where the log is source of truth
Fan-out to multiple independent consumers (analytics, search, notifications)
Backpressure absorption (producers keep publishing even when consumers slow down)

When NOT to use it

You need per-message ACK (like RabbitMQ) — Kafka is offset-based, not per-message
Sub-millisecond delivery latency — Kafka's floor is single-digit ms
You have very few events (<10K/day) — a queue is much simpler operationally
You need strict message ordering across the whole topic — Kafka orders per-partition only

Failure modes

Consumer lag grows unbounded → hot topic backs up. Mitigate with backpressure + scaling consumers.
Partition rebalance on consumer join/leave causes brief processing pause. Mitigate with cooperative rebalancing.
Broker down → partition replicas re-elect leader (~30s). Messages may need re-send from producer.
Bad message poisons a consumer → all consumers on that partition stall. Mitigate with a dead-letter topic.
Producer retries + acks=all can cause duplicates. Mitigate with idempotent producer (enable.idempotence).
Under-replicated partitions during a broker outage risk data loss if another broker fails.

Alternatives

RabbitMQ / ActiveMQ — traditional queues, per-message ACK, no replay
SQS — managed, simple, no ordering guarantees on standard queues
AWS Kinesis — Kafka-like managed offering
Redpanda / Pulsar — Kafka-compatible or Kafka-like with different trade-offs
Pub/Sub (Google) — managed pub-sub, at-least-once, no partition ordering

Interview questions

How does Kafka guarantee ordering, and when is that guarantee violated?
What's a consumer group, and what happens when a consumer joins?
How do you deliver exactly-once with Kafka? (Hint: it's tricky.)
Your consumer lag is 10 minutes and growing. Walk me through your debugging.
How would you use Kafka for CDC from a Postgres primary?
Producer with acks=all + idempotence — what does that guarantee, and what does it cost?
What if you need cross-partition ordering (e.g., a global sequence number)?

DEEP DIVE

Industry-grade explanation

Understand

Mental model

Kafka is a **distributed commit log with time-based retention**. Not a queue. Not a database. A log. Think of it as the write-ahead log of your entire company — every event ever, in append-only order, replayable by anyone at any time. Consumers don't 'remove' messages; they read at an offset and choose when to advance. The log is the source of truth; every downstream service (search index, cache, ML feature store, analytics warehouse) is a derived materialized view of that log. Once you internalize this — 'the log first, materialized views second' — event-driven architecture becomes obvious. This is Jay Kreps's 2013 insight, formalized in *The Log* essay.

Understand

Why it exists

Pre-Kafka (early 2010s LinkedIn), point-to-point ETL between systems was a combinatorial mess: N producers × M consumers = N×M integrations, each with its own back-pressure, error handling, and schema. LinkedIn's data engineering team was drowning in one-off pipelines that broke constantly. Jay Kreps and team asked: what if everything published to ONE durable, ordered, replayable log, and every consumer read from that log at its own pace? Suddenly N + M integrations. Every new downstream is just 'subscribe to the log.' The insight was 30 years old (WAL for databases) applied at company-integration scale. Kafka open-sourced in 2011 and now underpins Netflix, Uber, Airbnb, most of Fortune 500 event infrastructure.

Internals

Under the hood

**Per-partition log:** Each partition is a directory of segment files on the broker's disk. Segments are append-only files (default 1GB each). Producers append; consumers read via file offset (byte position). Because writes are sequential appends and reads are sequential scans, Kafka gets ~100MB/s per disk on commodity SSD without breaking a sweat. Modern brokers use `sendfile()` for zero-copy — the OS pipes bytes from disk to socket without going through user space. **Replication:** Every partition has 1 leader + N followers (typically N=2, replication factor 3). Producers write only to leader. Followers pull from leader (like a slave replica). ISR (In-Sync Replicas) is the set of followers currently caught up. With acks=all, producer waits for all ISR to ACK before considering the write durable. **Consumer groups + offset management:** Each partition is assigned to exactly one consumer per group (parallelism = partition count). Consumers commit offsets back to Kafka itself (in a special __consumer_offsets topic — dogfooding). Rebalance happens on join/leave; cooperative sticky rebalance (KIP-429) minimizes disruption. **Idempotent + transactional producers:** With enable.idempotence=true, producer sends a monotonic sequence number per partition; broker deduplicates on retry. With transactional producer (transactional.id set), producer + consumer can participate in a Kafka transaction spanning multiple topics — the foundation of the 'exactly-once' story (which is really 'effectively-once' when side effects are external). **Log compaction:** For key-based topics (like user-profile-updates), Kafka can compact — keep only the latest value per key. Old versions garbage-collected. Enables using Kafka as a materialized key-value store.

Data structures & algorithms

Append-only log segment files (immutable, memory-mapped)
Offset index (sparse, maps byte offset → message offset)
Time index (sparse, maps timestamp → offset)
ISR set (in-memory, per-partition)
Consumer offset topic (__consumer_offsets, log-compacted)
ZooKeeper metadata (pre-KRaft: cluster membership, partition assignment) or KRaft quorum controller (post-2.8)

At scale

Performance characteristics

latency

Producer end-to-end: 5-10ms with acks=all + linger.ms=0. Sub-ms achievable with acks=1 + fire-and-forget but risks data loss. Consumer lag is application-dependent; well-tuned systems maintain sub-second lag at 100K msg/sec.

throughput

Per broker: 500 MB/s writes, 1 GB/s reads on commodity SSD with 12+ partitions per broker. Cluster-wide: linear scaling. LinkedIn runs clusters processing 7 trillion msgs/day.

memory

Broker: 6-8 GB heap for JVM + rest for OS page cache (critical! Kafka relies on page cache for read performance). Client: minimal — records batched in memory (batch.size default 16KB).

cpu

Producer/consumer: compression (Snappy, LZ4, Zstd) dominates CPU. Broker: page cache misses and replication are the main CPU costs.

network

Replication multiplies producer throughput by RF (usually 3x). 100 MB/s writes with RF=3 = 300 MB/s cross-broker network. Plan for it.

storage

Cheap and predictable. 100 MB/s × 86400 sec/day × 3 replicas × 7 days = 180 TB for a week of retention. Compaction can massively reduce for key-based topics.

At scale

Scaling model

Scale by adding partitions (to a limit ~200K per cluster) and brokers. Partition count = max parallelism for any single consumer group. Increase partition count only up: adding partitions is easy, decreasing requires a full rebuild. Rule of thumb: target ≥1 partition per consumer, ≤4000 partitions per broker. For multi-tenant clusters, use quotas (producer_byte_rate, consumer_byte_rate) to prevent noisy neighbors.

At scale

Bottlenecks

**Disk write bandwidth** — sequential writes cap at ~100-200 MB/s per SSD. Add disks or partitions.
**Network** — replication multiplies producer throughput; a 10 Gbps NIC saturates at ~350 MB/s producer with RF=3.
**Consumer processing rate** — if consumers can't keep up, lag grows; the log is elastic but consumer is not.
**Partition count vs broker memory** — too many partitions = too much file-descriptor + metadata pressure.
**Rebalance frequency** — frequent consumer joins/leaves cause cascading rebalances; cooperative sticky mitigates.

Recovery

Automatic when network heals. Manual intervention only for prolonged partitions.

Architect's decision

Consistency, durability, availability

Consistency

**Per-partition total order** — all messages in a single partition are strictly ordered by offset. **Cross-partition:** no ordering guarantee. **Read-your-writes:** producers see their own writes on the same partition immediately after ACK. **Delivery semantics:** at-most-once (fire-and-forget), at-least-once (default), effectively-once (idempotent producer + transactional writes). Exactly-once end-to-end requires idempotent consumer or transactional consumer-write pattern.

Durability

With acks=all + min.insync.replicas=2 + RF=3, a message is considered committed only when written to 2+ replicas. Survives single broker loss. Survives disk failure on 1 replica. Does NOT survive simultaneous loss of 2 replicas (rare — deploy across AZs to make correlated failure unlikely). Backups: mirror to a second Kafka cluster (MirrorMaker 2) in a different region for DR.

Availability

During broker outage: partition-level failover in 10-30s (leader election). Cluster remains available for other partitions. Producers see brief errors on the affected partitions and retry (idempotent producer makes this safe). Consumer groups pause briefly during rebalance. Target ≥99.9% for well-run clusters; Netflix reports 99.99%+.

Cross-cutting

Security concerns

**AuthN:** SASL/SCRAM (username/password) or mTLS (client certs) or SASL/OAUTHBEARER (integrated with OIDC).
**AuthZ:** ACLs per topic per principal (read, write, describe). Manage via kafka-acls CLI or Confluent Control Center.
**Encryption in transit:** TLS 1.2+ mandatory between clients and brokers, and between brokers themselves.
**Encryption at rest:** Filesystem-level or cloud-provider block encryption (LUKS, EBS encryption). Kafka has no native at-rest encryption.
**Tenant isolation:** Use quotas (per-user byte rate limits) to prevent noisy neighbors. Multi-tenant clusters need clear ACLs.
**PII / GDPR:** Kafka's append-only + retention model conflicts with 'right to be forgotten.' Solutions: log compaction (tombstone-then-compact), external anonymization pipeline, or short retention.

Production

Observability checklist

kafka.consumer.lag (per group, per partition) — the single most important metric
UnderReplicatedPartitions — indicates broker or network trouble
IsrShrinksPerSec / IsrExpandsPerSec — replication health
RequestQueueTimeMs (p99) — broker under load
MessagesInPerSec / BytesInPerSec / BytesOutPerSec — throughput
PartitionsCount + LeaderCount per broker — balance across cluster
Consumer commit rate + offset commit failures — application health
Producer send rate + retry rate — producer health
Distributed tracing: propagate trace headers via message headers (OTel Kafka propagator)

Architect's decision

Cost model

Drivers

**Broker compute** — usually i3.xlarge or m5.2xlarge on AWS (~$150-300/mo per broker)
**Storage** — replication factor × retention × throughput. 500 MB/s × 7 days × RF=3 = ~800 TB. Instance-storage SSDs are cheapest.
**Cross-AZ network transfer** — replication across AZs costs $0.01/GB (AWS). At 500 MB/s × 3 replicas × 24hr = ~130 TB/day = ~$1300/day just in network transfer.
**ZooKeeper (pre-KRaft)** — 3-5 small nodes, $50-100/mo
**Managed offerings** (MSK, Confluent Cloud) — 2-5x the DIY cost but zero ops. Break-even point ~10-20 brokers.

Formulas

storage_TB = throughput_MB_per_sec × 86400 × retention_days × RF ÷ 1024²
monthly_cost ≈ broker_count × broker_cost + storage_TB × $0.10/GB/mo + cross_AZ_GB × $0.01

Optimizations

Compression: enable Snappy/LZ4/Zstd on producer — 3-5x storage reduction
Log compaction on key-based topics — retention independent of message volume
Tiered storage (Kafka 3.6+) — hot data on SSD, cold data in S3 — huge savings for long retention
Reserved Instances / Savings Plans for broker fleet — 40%+ off on-demand
Confluent's KRaft (removes ZooKeeper) simplifies ops

Architect's decision

Alternatives compared

RabbitMQ

Pros

Per-message ACK, flexible routing (topic, direct, fanout, headers exchanges), lower operational complexity, well-suited for small-to-medium queues

Cons

No replay, no partitioning, throughput ceiling ~50K msg/sec, memory-based (can OOM on backup)

Verdict

Better than Kafka for task queues, RPC-style messaging, small-scale event flows. Worse for event streaming, replay, or high-throughput.

AWS SQS

Pros

Fully managed, infinite scale, cheap, zero ops. Standard queues nearly infinite throughput.

Cons

No ordering (standard) or 300 msg/sec (FIFO), no replay, no fan-out (need SNS→SQS pattern), max 14-day retention

Verdict

Great for simple task queues on AWS. Not a Kafka replacement — no replay, no event streaming.

AWS Kinesis Data Streams

Pros

Kafka-like log semantics, fully managed, IAM integration, tight AWS ecosystem

Cons

Shard-based scaling (pre-provision), 5 reads/sec per shard, replay limited to 7 days (up to 365 for extra cost), higher $/GB than Kafka

Verdict

Kafka-lite for AWS-native shops. If you need Kafka's ecosystem (Streams, Connect, Schema Registry), stay on Kafka.

Redpanda

Pros

Kafka-wire-compatible, no JVM (C++), no ZooKeeper (built-in Raft), 10x lower tail latency in benchmarks

Cons

Younger ecosystem, less battle-tested at extreme scale, commercial licensing

Verdict

Excellent for latency-sensitive workloads. Kafka still wins on ecosystem breadth (Connect, KStreams, ksqlDB).

Apache Pulsar

Pros

Separates compute (brokers) from storage (BookKeeper) — independent scaling, multi-tenancy first-class, geo-replication built in

Cons

More components to operate (broker + BookKeeper + ZooKeeper), smaller community, harder to hire for

Verdict

Superior architecture on paper for multi-tenant clouds. Kafka wins on adoption, tooling, community.

Google Pub/Sub

Pros

Fully managed, global, at-least-once, unlimited scale, tight integration with GCP data services

Cons

No ordering by default (ordered mode limited), no offset-based replay (only seek by timestamp), lock-in to GCP

Verdict

Great for pure fan-out on GCP. Not a full Kafka replacement.

Architect's decision

Decision framework

Pick Kafka when you have **all** of: (1) high throughput (>10K msg/sec sustained), (2) need for replay (backfilling downstream, recovering from bugs), (3) fan-out to multiple consumers, (4) multi-week retention, (5) team capacity to operate a distributed system OR budget for a managed offering. If ANY of these fail: RabbitMQ (per-message ACK, small scale), SQS (simple, AWS-managed, no replay), or Redis Streams (single-node, simple).

Production

Production usage (real companies)

Source ↗

Original creators of Kafka. Now processes 7+ trillion messages/day across 100+ clusters. Every internal service publishes to Kafka; all downstream analytics, feature store, search, and derived views subscribe. The entire company is glued together by Kafka.

Netflix

Source ↗

Keystone pipeline uses Kafka as the event backbone for all telemetry (user actions, viewing, playback errors). Scales to ~5 million events/sec. Kafka feeds into Flink (real-time analytics), S3 (data lake), Elasticsearch (search).

Uber

Source ↗

Kafka backs dispatch (driver-rider matching events), surge pricing (real-time demand signals), and analytics. Uber's uReplicator (open-source) handles cross-region Kafka replication at extreme scale.

Airbnb

Source ↗

Uses Kafka for CDC — Debezium reads MySQL binlogs, publishes to Kafka, downstream services (search indexer, recommendation engine, financial reporting) subscribe. Enables new consumers without touching the source DB.

Confluent

Source ↗

Founded by Kafka creators to commercialize Kafka. Their managed Confluent Cloud runs Kafka at $10B+ combined customer throughput. Their contribution: KRaft (no ZooKeeper), Tiered Storage, Confluent Schema Registry.

Interview

Common mistakes

**Under-partitioning topics** — starting with 1-3 partitions and hitting single-consumer bottleneck. Rule: start with 20-50 partitions even for medium topics.
**Ignoring acks setting** — default acks=1 doesn't wait for replicas. In interviews, always explicitly discuss acks + min.insync.replicas.
**Chasing exactly-once** — claim exactly-once end-to-end without specifying: 'Only exactly-once within Kafka + Kafka-Streams. Side effects to external systems (email, DB writes) require idempotent consumer.'
**Missing DLQ handling** — poison messages will stall your entire pipeline. Every consumer needs DLQ logic.
**Committing offset before processing** — corrupts at-least-once guarantee. Always process, then commit.
**Not planning for consumer lag alerting** — lag is your #1 operational signal.
**Choosing Kafka for RPC** — Kafka isn't a request/response system. Use gRPC or REST.

Staff level

Staff considerations

At Staff level, Kafka becomes a **platform decision, not a component decision**. Questions to own: How do we govern schema evolution across 500 microservices? (Confluent Schema Registry + compatibility policies.) How do we prevent one team's runaway consumer from starving another? (Cluster quotas + per-tenant topic prefixes.) How do we handle cross-region replication for DR? (MirrorMaker 2, active-passive vs active-active.) How do we manage cost as topics proliferate? (Tiered storage, aggressive compaction, retention governance.) How do we handle PII in an append-only log? (Log compaction with tombstones + external anonymization + short retention on PII topics.) Staff engineers write the Kafka Golden Path doc that says 'here's how our company uses Kafka' and enforce it via CI checks.

Principal level

Principal considerations

At Principal level, Kafka is a **strategic bet**, not a technology. Questions: Do we invest in Kafka mastery (hire, train, build platform) or adopt managed (Confluent Cloud, AWS MSK) and focus engineering elsewhere? What's our event-driven architecture strategy — hub-and-spoke (one big cluster) or federated (per-domain clusters bridged)? How do we align org boundaries with Kafka topics (Conway's Law — teams own topics they publish to)? What's our exit strategy if we ever want to move off Kafka? (Answer: painful, so pick carefully.) At LinkedIn or Uber scale, Kafka IS the company nervous system — a decision to change it is a 3-5 year migration.

Interview

Interview ladder

beginner

What is Kafka and when would you use it over a traditional message queue?
Explain topics, partitions, and consumer groups.
What does 'at-least-once delivery' mean, and how does Kafka provide it?
How do you scale a Kafka consumer application?

senior

How does Kafka guarantee ordering, and what are the exceptions?
Walk through what happens when a broker dies during a write.
What is a consumer group rebalance, and how do you minimize its impact?
How would you set up Kafka for change data capture from a Postgres primary?
Compare Kafka's replication model with a database like Postgres's streaming replication.

staff

Design an event backbone for a company with 200 microservices using Kafka. How do you handle schema governance?
Your Kafka cluster has 500 topics and 10K partitions. How do you plan capacity?
How would you migrate from RabbitMQ to Kafka without downtime for 100+ teams?
Cross-region disaster recovery for Kafka — walk through your topology, RPO, RTO, and cost.
You need to delete a user's data across all Kafka topics for GDPR. How?
Kafka is 40% of your infra bill. How do you cut it in half without losing capability?

principal

Should we adopt Kafka company-wide, or use Pulsar/Redpanda/Cloud-native alternatives? Frame the 5-year TCO.
How does Kafka's architecture align (or misalign) with our org structure? What migrations does Conway's Law suggest?
We're at 10 trillion messages/day and growing 3x/year. What's the ceiling we're hitting first, and how do we architect past it?
Design a governance model for Kafka schemas across 50 teams that balances velocity vs. stability.
Multi-region active-active Kafka — is it worth the complexity? What's the failure mode nobody thinks about?

Architecture Decision Records

ADR 001

Architecture Decision Record

Adopt Kafka as the company-wide event backbone

Context

50+ microservices at LinkedIn (~2010) had grown a spider web of point-to-point ETL pipelines. Every new consumer required source teams to add integration code. Pipelines broke silently. Backpressure was inconsistent. Batch jobs ran nightly with lag. Real-time signal was impossible.

Constraints

Must scale to trillions of messages/day
Must support fan-out from 1 producer to N (unlimited) consumers
Must retain history for at least 7 days for replay/backfill
Must survive single-broker failure with no data loss
Must be self-hostable (2010: no managed cloud broker existed at required scale)

Options considered

Extend existing point-to-point (JMS-based)

Pros

Familiar
Already deployed
No new tech to operate

Cons

Doesn't scale
N×M integration cost keeps growing
No replay
Backpressure inconsistent

Cost: Low upfront, but 10 engineers × 6 months × 2 years = $12M+ in pipeline maintenance

RabbitMQ or ActiveMQ at scale

Pros

Mature
Per-message ACK
Flexible routing

Cons

Memory-based (queues can OOM)
No replay (consumption removes message)
Throughput ceiling ~50K/sec
Multi-cluster complexity

Cost: Medium — 3-5 clusters, ongoing maintenance

Build custom log-based broker (Kafka)

Chosen

Pros

Purpose-built for scale (100K+ msg/sec/broker)
Replay from log for weeks
Fan-out is free — 1 producer, N consumers, no producer change
Sequential IO on commodity SSD = cheap throughput
Ordered per-partition

Cons

New technology (2010: unproven)
Requires operating ZooKeeper + brokers
No per-message ACK model — mental model shift for developers
Ordering only per-partition (need to pick partition keys carefully)

Cost: Medium upfront (build team) + medium ongoing (SRE team), but decades of value

Chosen solution

Build custom log-based broker (Kafka)

Why

Only option that solves N×M integration explosion at the required scale. Log-based model enables the 'materialize downstream from the log' architecture, which decouples producer velocity from consumer count. Sequential-IO design is orders of magnitude cheaper per throughput than random-IO alternatives. LinkedIn had the engineering talent to build and operate.

Rejected alternatives (with reasons)

Extending JMS: doesn't scale, would grow the pipeline mess linearly
RabbitMQ/ActiveMQ: no replay, throughput ceiling, and multi-cluster ops complexity

Trade-offs accepted

Operating a new distributed system (needed a dedicated infra team)
Per-message ACK model went away — developers had to rethink 'has this been processed?' via offsets
Ordering only per-partition — required careful partition-key design (user_id, order_id) in every topic

Consequences

Enabled real-time analytics, feature store, event sourcing, and CDC — all built on the log
Kafka became a strategic asset: LinkedIn open-sourced it (2011), spawned Confluent (2014), now used at every FAANG-scale company
Locked LinkedIn into log-based thinking — hard to imagine reverting to point-to-point ever

When would we reverse this decision?

Kafka can't scale to the next order of magnitude (unlikely — LinkedIn already runs at 7T msg/day)
A new tech with 10x lower operational overhead + better semantics emerges (Pulsar, Redpanda, Warpstream all vying — none decisive yet)
Managed cloud offering becomes 10x cheaper than self-hosted at scale (Confluent, MSK, Warpstream trending this way)

Knowledge graph

Learn these first

TCP + basic distributed systems (partial failure, replication)
Understand pub/sub vs queue semantics
Consistency models — eventually consistent replication
Basic messaging concepts (producer, consumer, ACK)

Where this appears in the curriculum

View event stream for analytics and recommendations

Open system

DEEP DIVE — INTERACTIVE

Interactive walkthrough with animated visuals

In 2011, three engineers at LinkedIn — Jay Kreps, Neha Narkhede, and Jun Rao — had a problem. LinkedIn had dozens of services that all needed to share data with each other. Every pair of services had built its own custom pipeline: user activity → recommendation engine, user activity → search index, user activity → analytics warehouse. Result: N² integrations, brittle, hard to change. They wanted a single pipeline all services could read from.

They built Kafka — a distributed append-only log that any producer can write to and any consumer can read from, replayable. Not just a message queue — a durable, replicable, replayable log. Today, Kafka runs at nearly every major internet company: Netflix, Uber, Airbnb, LinkedIn, Twitter, PayPal, Bloomberg, and hundreds more. It processes trillions of events per day globally. And it's the foundation for event-driven architecture, stream processing (Kafka Streams, Flink), CDC (Debezium), and data-warehouse ingestion.

Historical framing

2011 — LinkedIn open-sources Kafka (Jay Kreps, Neha Narkhede, Jun Rao).
2013 — Confluent founded to commercialize Kafka.
2015 — KIP-98 ships: transactional producer + exactly-once semantics.
2016 — Kafka Streams library — building stream processors as apps.
2020 — KIP-500 accepted — remove ZooKeeper dependency.
2021 — KRaft (Kafka Raft) ships as preview; own consensus + metadata.
2024 — KRaft becomes default; ZooKeeper deprecated for new clusters.

Interactive: producer → topic → consumer

The core Kafka data flow. Producer writes to a topic (an append-only log). Consumer reads from a topic, at its own pace. The magic: many consumers can read the same topic independently.

Producer → topic → consumer — step 1 of 6

Producer prepares message M1 with key='user:alice'.

Topic partitioning — how Kafka scales

A single log on one machine has a throughput ceiling — millions of messages per second is not enough for LinkedIn / Netflix scale. Kafka's solution: partition a topic into multiple logs, each on a different broker. A producer picks a partition per message (usually by key hash), consumers process partitions in parallel.

Interactive: partition assignment by key hash

Key "alice" always goes to partition 0. Ordering within partition is preserved. Try different keys — they scatter across partitions but every same-key message goes to the same one.

Ordering guarantee: within a partition, order is preserved. Across partitions, no ordering. This is why the "partition key" choice matters — every message with the same key goes to the same partition and stays in order. Change the key, break the ordering.

Replication — ISR (In-Sync Replicas)

Each partition has 1 leader + N followers (typically 3-way replication). The leader accepts all writes; followers replicate. A write is considered committed when it's replicated to min-ISR replicas (default 2 of 3).

Interactive: click replicas to change their state

ISR count: 3 / 3 · ✓ Writes commit (min-ISR=2 satisfied). Even 1 broker failure is tolerated.

Consumer group rebalance — the scale-out story

A consumer group is a set of consumer processes that together consume a topic. Each partition is assigned to exactly one consumer in the group. Add a consumer → rebalance happens → some partitions move to the new consumer. Kafka handles this automatically.

Interactive: add/remove consumers, watch partitions rebalance

2 consumers in the group, sharing 6 partitions.

→ C0

→ C1

→ C0

→ C1

→ C0

→ C1

Each partition is assigned to exactly one consumer. Add a consumer → some partitions reassign to it. Remove a consumer → its partitions redistribute. Kafka handles this automatically via the group coordinator broker.

Exactly-once semantics — the hard problem

By default, Kafka gives at-least-once delivery — messages may be duplicated on retries. For financial or idempotency-sensitive workloads, KIP-98 (2015) introduced exactly-once semantics (EOS): transactional producer + idempotent producer + read-committed consumer.

Idempotent producer

Producer assigns each message a producer-ID + sequence-number. Broker dedupes on retry.

Transactional producer

Group multiple writes across topics in a single transaction. All-or-nothing commit via 2PC.

Read-committed consumer

Consumer skips messages from uncommitted or aborted transactions. Sees consistent view.

All 3 together = exactly-once semantics (EOS). Shipped in KIP-98 (2015). Essential for financial workloads. Cost: ~10-30% latency overhead. Use only when needed.

Retention policies

Kafka retains messages, but for how long? Two policies:

Time-based retention

Default: retention.ms=604800000 (7 days). After 7 days, older messages are deleted regardless of consumption.

Use for: event streams, activity data, audit logs.

Log compaction

cleanup.policy=compact. Kafka keeps only the latest value per key. Old versions removed during compaction.

Use for: changelogs, materialized views, cache-refresh topics.

Product comparison

Product	Hosting	JVM?	Exactly-once	Best for
Apache Kafka (self-hosted)	You run it	Yes	Yes	Full control, custom tuning
Confluent Cloud	Managed SaaS	Yes	Yes	Managed Kafka + schema registry + connectors
AWS MSK	Managed on AWS	Yes	Yes	AWS-native Kafka
AWS MSK Serverless	Managed serverless	Hidden	Yes	Auto-scale, spike-heavy workloads
Redpanda	Self-hosted or Cloud	No (C++)	Yes	10x lower P99 latency
Apache Pulsar	Self-hosted	Yes	Yes (via txn)	Multi-tenancy, geo-replication
Amazon Kinesis	AWS proprietary	N/A	Yes (per-shard)	AWS-native, simpler than Kafka
Google Pub/Sub	GCP managed	N/A	At-least-once by default	GCP-native, no partitions to manage

Applied in real systems

LinkedIn — the origin story

LinkedIn built Kafka in 2011 to unify their internal pipelines. Today they run some of the largest Kafka deployments (1 trillion+ messages/day). Their Espresso database CDC, activity data, monitoring all flow through Kafka.

Netflix Keystone

Netflix Keystone — 500B events/day

Netflix's Keystone platform routes 500 billion events per day through Kafka to downstream analytics + Flink stream processors + S3. Backbone of their data infrastructure. Presented at multiple conferences.

Uber uReplicator

Uber — cross-region Kafka replication

Uber runs Kafka in every region + operates uReplicator (open-source) to replicate topics cross-region. Trip events, driver locations, payment events — all Kafka. Uber runs 100+ Kafka clusters globally.

Confluent Cloud

Confluent Cloud — Kafka as a service

Confluent (founded 2013 by the LinkedIn Kafka team) offers managed Kafka on AWS, GCP, Azure. Multi-tenant, serverless, elastic. Popular with startups avoiding Kafka operational overhead.

AWS MSK

AWS MSK — managed Kafka

Amazon Managed Streaming for Kafka. Runs the OSS Kafka in AWS with automated patching + cluster management. Serverless option since 2022. Standard choice for AWS shops.

Redpanda

Redpanda — C++ Kafka rewrite

Redpanda (2019+) is a Kafka-API-compatible broker written in C++ using thread-per-core Seastar framework. 10x lower P99 latency. No JVM, no ZooKeeper, single binary. Used by Alpaca, Hivemapper.

Debezium CDC

Debezium — CDC from databases to Kafka

Debezium is a Kafka Connect plugin that streams row-level changes from Postgres, MySQL, MongoDB, SQL Server into Kafka topics in real time. Foundation of modern CDC pipelines.

Apache Flink

Flink + Kafka — stream processing

Flink reads from Kafka topics, processes stateful streams (windows, joins, aggregations), writes back. Uber, Alibaba, Netflix all use this pattern for real-time analytics and fraud detection.

Slack

Slack Job Queue on Kafka

Slack's background job system (post to channels, notifications, integrations) runs on Kafka. Their engineering blog details the migration from Redis-based Sidekiq to Kafka for better durability + replay.

Bloomberg

Bloomberg market data

Bloomberg publishes market data through Kafka for internal consumers. Multi-tenant, ultra-low-latency (single-digit ms). Extensively tuned deployment with hundreds of brokers.

Key takeaways

Kafka is a durable, replayable, partitioned log — not just a message queue. Messages persist (days/weeks). Many consumers can read independently.
Partitions scale throughput. Within a partition, order is preserved. Across partitions, no ordering. The partition key choice matters.
ISR (In-Sync Replicas) with min-ISR gives you durability. RF=3 + min-ISR=2 means the write survives up to 1 replica failure.
Consumer groups parallelize processing across partitions. Rebalance happens automatically when consumers join/leave.
Exactly-once semantics (KIP-98, 2015): transactional producer + idempotent producer + read-committed consumer. Essential for money-adjacent workloads.
Retention is by time or size. Log compaction keeps only the latest value per key — useful for changelog-shaped data.
KRaft (KIP-500) removes the ZooKeeper dependency. Kafka now runs standalone using its own Raft-based consensus.
Kafka is the backbone of every modern data-intensive company. Learning it is a Staff-Engineer skill.

References

Kreps, Narkhede, Rao (2011) — Kafka: A Distributed Messaging System for Log Processing. NetDB Workshop.
Kreps (2013) — "The Log: What every software engineer should know about real-time data's unifying abstraction." LinkedIn Engineering blog.
KIP-98 (2017) — Exactly Once Delivery and Transactional Messaging.
KIP-500 (2020) — Replace ZooKeeper with a Self-Managed Metadata Quorum.
Narkhede, Shapira, Palino (2017) — Kafka: The Definitive Guide. O'Reilly.
Kafka official docs — kafka.apache.org/documentation.