Kafka-like Streaming Platform

Same startup, same engineer #4. Thirteenth Monday.

Your CTO drops by. "Every system we've built now depends on Kafka. Slack fans out messages via Kafka. Instagram feeds signals via Kafka. Netflix's Keystone pipeline moves 15 billion events per day. Payment system writes ledger events. Time to understand how Kafka ACTUALLY works so we know when to use it, when to use SQS, and when to build something else. Ship a design in 12 weeks."

She pauses. "For context — LinkedIn (where Kafka was invented in 2010) now runs trillions of events per day across their cluster. Uber runs 4 trillion messages/day through Kafka. Discord processes 5 trillion messages/day. Netflix's Keystone is 15B events/day. Kafka is the invisible backbone of every large distributed system. If you don't understand partitioning, replication, consumer groups, exactly-once semantics, and tiered storage, you'll design systems that break in production."

Here's the paradigm shift:

Distributed Cache was in-memory, ephemeral, latency-optimized. Data can be lost.

Kafka is a distributed log — a fundamentally different abstraction:
- Persistent by design — every message is on disk with configurable retention (hours to years).
- Ordered per partition — offsets give exact position within a partition; consumers replay from any offset.
- Consumer-controlled — the broker doesn't push; consumers pull at their own pace. This is why Kafka scales to trillions of msgs/day where RabbitMQ tops out at millions.
- Immutable append-only — a message once written is never changed. This maps directly to event sourcing + CQRS + Saga.

LinkedIn's answer was to invent the distributed commit log abstraction (Jay Kreps' original 2013 blog: "The Log"). If you interview at LinkedIn, Uber, Netflix, or Confluent — this blog is required reading. Kafka is now Apache open-source, but the design decisions come from LinkedIn's 2010-era needs.

The real 2024 numbers

• LinkedIn: 7+ trillion messages/day through Kafka clusters (LinkedIn Engineering: Kafka at scale)
• Uber: 4+ trillion messages/day, 15+ PB/day ingested (Uber uReplicator + Kafka blog)
• Netflix Keystone: 15B events/day, powered by Kafka + Flink (Netflix Tech Blog)
• Confluent Cloud: exabyte-scale multi-tenant
• KIP-405 Tiered Storage: shipped in Kafka 3.6+ (2023-2024), moves cold data to S3 for 10x cost reduction
• Kafka default partition count: 3-100 per topic — power of 2 recommended (16, 32, 64) for future rebalancing
• Replication factor 3 is the industry default (survives 1 broker failure with 2 alive minimum)

Interview soundbite: "Kafka is a distributed commit log with 4 design decisions: (1) topic partitioning for horizontal scale, (2) leader-follower replication for durability, (3) consumer offsets for replay, (4) log-structured storage for sequential I/O. LinkedIn invented it in 2010 for their 7T-msg/day scale. Uber runs 4T. Netflix Keystone runs 15B events/day. If you don't distinguish partitions from replicas + producer batching from broker batching, you're at L4."

The whole journey at a glance

Every 10× in throughput surfaces a different bottleneck:

text
    ═══════════ KAFKA-LIKE STREAMING ACROSS 4 SCALES ═══════════

    L4 (10K msg/s)     L5 (1M msg/s)      L6 (100M msg/s)          L7 (7T msgs/day LinkedIn)
    SQS or 1 Kafka     Kafka cluster      + Tiered storage         Multi-region + geo-fanout
    12 weeks · $500/mo 6 months · $20K/mo 18 months · $500K/mo     ongoing · $50M+/yr

    ┌────────┐         ┌────────┐         ┌── Producer clients ─┐  ┌── Global producers ──┐
    │Producer│         │Producer│         │  batching + compres │  │  1000s of services    │
    │clients │         │clients │         │  ack=all safety     │  │  billion+ events/hr   │
    └───┬────┘         └───┬────┘         └─┬──┬──┬──┬─────────┘  └──┬──┬──┬──┬──────────┘
        │                  │                │  │  │  │              │  │  │  │
     ┌──▼───┐           ┌──▼──┐          ┌──▼──▼──▼──▼─────┐    ┌───▼──▼──▼──▼──────────┐
     │AWS   │           │Kafka│          │  Kafka brokers  │    │ Regional Kafka         │
     │SQS or│           │3    │          │  20-100 nodes   │    │ + MirrorMaker 2 for    │
     │1 Kafk│           │brokr│          │  Multi-AZ       │    │ geo-fanout             │
     │broker│           │Multi│          └───┬──────┬─────┘     └──┬──┬──┬──┬──────────┘
     └──┬───┘           │-AZ  │              │      │              │  │  │  │
        │               └──┬──┘           ┌──▼──────▼───┐        ┌──▼──▼──▼──▼──────────┐
        │                  │              │ RF=3 sync    │        │ Kafka clusters       │
        │              ┌───▼──┐           │ min.insync=2 │        │ 100s of brokers      │
        │              │Kafka │           │ ack=all      │        │ per region           │
        │              │topics│           └───┬───────────┘       │ + KIP-405 Tiered     │
        │              │+ part│           ┌───▼──────────┐        │ Storage to S3        │
        │              │itions│           │ Tiered stor   │       └──┬──┬──┬──┬──────────┘
        │              └───┬──┘           │ hot=disk      │           │  │  │  │
        │                  │              │ warm=SSD      │        ┌──▼──▼──▼──▼──────────┐
        │              ┌───▼─┐            │ cold=S3       │        │ Consumers            │
     ┌──▼───┐          │Cons │            │ KIP-405       │        │ (Flink, Spark,       │
     │Cons  │          │umer │            └───┬──────────┘        │  in-house apps)      │
     │umer  │          │grps │            ┌───▼──────────┐        └──┬──┬──┬──┬──────────┘
     │(1-2  │          └─────┘            │ Consumer     │            │  │  │  │
     │apps) │                             │ groups       │         ┌──▼──▼──▼──▼──────────┐
     └──────┘                             │ exactly-once │         │ Flink jobs (stateful │
                                          │ semantics    │         │ streaming, exactly-  │
                                          └──────────────┘         │ once)                │
                                                                    └──┬──────────────────┘
                                                                       │
                                                                    ┌──▼──────────────────┐
                                                                    │ Downstream storage:  │
                                                                    │ Iceberg / HDFS /     │
                                                                    │ ClickHouse / Redshift│
                                                                    └─────────────────────┘

    Bottleneck        Bottleneck          Bottleneck                Bottleneck
    Throughput cap:   Broker disk fills   Retention grows           Multi-region latency +
    ~10K msg/s per    faster than         unboundedly. Tiered       geo-fanout needs
    broker on          consumers can       storage moves cold        MirrorMaker 2. Cost
    ordinary hw.      catch up.           to S3 (10x cheaper).      = the enterprise deal.

   Chapter 5      Chapters 6+6.5      Chapter 7+7.5              Chapter 8
   walks          walks through       walks through KIP-405       walks through 7T
   through        Kafka partition +   Tiered Storage 2024 +       msgs/day LinkedIn,
   L4 MVP         RF=3 min-insync=2   consumer group semantics    MirrorMaker 2, and
                  + producer batching                              geo-fanout economics

  Key insight: Kafka is a DISTRIBUTED LOG. Producer batching + broker sequential I/O
  + consumer pull model = trillion-msg/day throughput. RF=3 + min.insync.replicas=2
  + acks=all = ZERO message loss on 1 broker failure. Consumer groups + offsets = replay
  + exactly-once semantics. If you name 4 primitives (partitions, replication, consumer
  offsets, log-structured storage) you're L6+. If you cite KIP-405 Tiered Storage
  you signal L7 awareness of 2024's biggest Kafka change.

The same 4 tiers as clean architecture diagrams

L4 · 10K msg/s · SQS or 1 Kafka broker · $500/mo · 12 weeks:

L5 · 1M msg/s · Kafka cluster · $20K/mo · 6 months:

L6 · 100M msg/s · Cluster + KIP-405 Tiered Storage · $500K/mo · 18 months:

L7 · 7T msgs/day LinkedIn · Multi-region + MirrorMaker 2 · $50M+/yr:

Why every 10× breaks the architecture

1. Partitioning enters early. L4 = 1 partition (topic order). L5+ = 10-1000 partitions per topic. Each partition = one thread of parallelism, one consumer per partition per consumer group. Reference: Kafka partitions docs.

1. Replication safety is the L5 signal. acks=all + replication.factor=3 + min.insync.replicas=2 = ZERO message loss on 1 broker failure. Anything less can silently lose data on failover. Reference: Kafka Confluent durability docs.

1. KIP-405 Tiered Storage is the 2024 L7 lever. Kafka 3.6+ ships built-in tiered storage: hot data on broker disks, cold data on S3. 10x cost reduction for retention > 7 days. Reference: KIP-405 design doc. Do NOT say "Kafka can't handle long retention" — it can, since 2024.

The 3 senior insights before we start Chapter 1

1. Kafka is a distributed log, NOT a message queue. Every candidate says "we use Kafka for messaging." Fewer explain that it's a REPLAYABLE, PERSISTENT, ORDERED-PER-PARTITION commit log. This is why Kafka works for event sourcing, CQRS, Saga, changelogs, and Kafka Streams state stores. RabbitMQ can't do these because messages are consumed and gone. Reference: Jay Kreps' "The Log" blog 2013.

1. Exactly-once semantics is the L6 probe. enable.idempotence=true + transactional.id + Kafka Transactions API + Flink checkpoints = exactly-once end-to-end. Naive Kafka is at-least-once. If you can't distinguish these, you're at L5. Reference: Kafka EOS blog by Jason Gustafson.

1. MirrorMaker 2 replaced MirrorMaker 1 for cross-region. LinkedIn's MirrorMaker 1 had 20+ known issues; MM2 (based on Kafka Connect) is the modern replacement. Naming MM2 signals awareness of multi-region deployment reality. Reference: Kafka MirrorMaker 2 docs.

Chapter map for the journey ahead

• Chapter 1 — Requirements (produce, consume, retention, ordering, delivery guarantees)
• Chapter 2 — Capacity estimation (7T msgs/day LinkedIn, 4T Uber reference)
• Chapter 3 — API design (produce, consume, seek, commit offset, admin)
• Chapter 4 — Data model (topics, partitions, offsets, segments)
• Chapter 4.5 — Log-structured storage: sequential I/O + zero-copy
• Chapter 5 — L4 MVP: SQS or 1 Kafka broker. Works to 10K msg/s
• Chapter 6 — L5: Kafka Cluster + RF=3 + consumer groups
• Chapter 6.5 — Consumer group rebalancing: eager vs cooperative sticky
• Chapter 7 — L6: Exactly-once semantics + transactions + KIP-405 Tiered Storage
• Chapter 7.5 — Deep-dive: idempotent producers + Kafka Transactions API
• Chapter 8 — L7: Multi-region + MirrorMaker 2 + geo-fanout economics
• Chapter 9 — Failure modes: broker crash, ISR shrink, consumer lag, tiered storage backpressure
• Chapter 10 — Trade-off matrix (Kafka vs Pulsar vs RabbitMQ vs SQS vs Kinesis vs Pub/Sub)
• Chapter 11 — Interview masterclass: 45-min mock, questions to ask
• Chapter 12 — Defense: the 20 hardest interview questions on streaming

Ready? Chapter 1 next: what did the CTO actually ask for?