Skip to main content
stream

Stream Processor (Flink, Kafka Streams, Spark Structured Streaming)

Continuous computation over event streams — windowed aggregations, joins, ML features, and event-driven derived views.

Why it exists

Once you have a Kafka pipeline, you often need to compute things over that stream: 'count events per user per 5-minute window', 'join clicks with impressions', 'detect fraud in real time'. A batch job runs every N hours; a stream processor runs continuously with sub-second latency. This is what turns event streams into actionable data products.

How it works

The processor reads events from one or more sources (Kafka topics), applies operators (map, filter, group-by, window, join, aggregate), and writes results to sinks (Kafka topics, databases, dashboards). State (like running counts) is stored in a local key-value store (RocksDB embedded); checkpointed to durable storage so recovery is exactly-once. Windows can be tumbling (fixed intervals), sliding (overlapping), or session (activity-based). Watermarks handle late-arriving events.

Scaling characteristics

Throughput scales with parallelism (partition count). Flink handles millions of events/sec per cluster; Kafka Streams runs embedded in your JVM app. State size can be tens of GB per operator instance. Latency is typically ~100ms end-to-end for simple pipelines; complex joins can push to seconds.

When to use it

  • Real-time analytics dashboards (live user counts, top-N)
  • Feature engineering for online ML (fraud detection features per user)
  • Continuous ETL — deriving denormalized views from event streams
  • Change data capture pipelines (Postgres → Debezium → Flink → search index)
  • Complex event processing (patterns like 'user login followed by password change within 60s')

When NOT to use it

  • Simple event forwarding — Kafka Connect is usually cheaper
  • Ad-hoc historical analytics — batch (Spark, Snowflake) is more efficient for one-off queries
  • Sub-10ms latency requirements — even Flink adds tens of milliseconds
  • Small event volumes (<1K/sec) — a simple consumer app is simpler operationally

Failure modes

  • Checkpoint failures on slow storage cause processing to stall — monitor checkpoint duration
  • State grows unbounded (missing TTL on aggregations) — RocksDB gets huge and compaction dominates CPU
  • Watermark stalls when one partition has slow events — computation waits for the laggard
  • Backpressure from a slow sink cascades upstream — always monitor source lag
  • Rebalance during a scale-up moves state across nodes — brief pause in processing

Alternatives

  • Kafka Streams — embedded in your JVM app; simplest deployment; state lives with your app
  • Spark Structured Streaming — great if you already have Spark; higher latency (~seconds)
  • Materialize / RisingWave — SQL-first streaming DBs; easier ergonomics for SQL teams
  • Custom Kafka consumer with in-app aggregation — good for simple cases; complex windowing is painful
  • Batch job every N minutes — pragmatic when 'real-time' can mean 5 minutes

Interview questions

  • Explain the difference between tumbling and sliding windows. Give an example use case for each.
  • Your stream processor's state is 500 GB and growing. How do you diagnose and fix?
  • How do watermarks work, and what happens if events arrive out of order?
  • Compare exactly-once semantics in Kafka Streams vs Flink.
  • Design a fraud-detection feature: 'flag any card that had a login in one country and a purchase in another within 5 minutes.' What operators?
  • Your stream processor is doing 100ms latency; PM wants 10ms. What levers do you have?

Systems that use this component

See how the real designs on this platform put stream-processor to work — concrete usage context per system.

Netflix

intermediate

Flink for real-time recommendation feature computation

Open system

Twitter/X timeline

intermediate

Heron (Twitter's replacement for Storm) for real-time trending topics

Open system

Ride Sharing

intermediate

Flink for surge pricing signals + ETA prediction

Open system

Instagram

intermediate

Feed ranking real-time updates

Open system
The story of continuous computation

2011, LinkedIn: batch is dead, long live the stream

For 40 years, data processing meant batch. Nightly ETL, Monday-morning reports, dashboards refreshed at 2AM. Then in the late 2000s, businesses started needing real-time answers. Amazon wanted to recommend the next product before the customer checked out. Uber wanted to price a ride before the rider got in. Fraud detection needed to happen in the same second as the transaction. Batch was too slow.

Jay Kreps at LinkedIn saw the pattern. In 2011 he co-created Kafka — a durable log that could serve as the "central nervous system" for data. But Kafka just stored events; you needed something to process them. Enter stream processing. Apache Storm (2011) came from Twitter. Apache Samza (2013) came from LinkedIn. Apache Flink (2014) came from a German research project, TU Berlin's Stratosphere. And most importantly, in his 2015 essay "Questioning the Lambda Architecture," Kreps argued that the whole distinction between "batch" and "stream" was artificial: batch is just stream with a large window.

The industry converged on that view. Kafka Streams (2016) offered a library for stream processing in your own service. Databricks Delta Lake (2019) unified batch + streaming on the same tables. Apache Beam (2016, Google) tried to give one API for both. Today, when engineers say "stream processing," they mean building applications that transform, aggregate, and enrich continuous data — with millisecond-to-second latency, exactly-once semantics, and state that survives failures.

The core insight: stream processing changes your mental model. Instead of "run a query against a table," you think "subscribe to events + maintain a state." Instead of "did this fact match?" you think "within the last 5 minutes, did the pattern occur?" This shift lets you build fraud detection, anomaly alerts, real-time metrics, ETL — all as continuous, always-on applications rather than periodic jobs.

The stream-processing stack
Input: log / queue
Kafka, Kinesis, PubSub — the immutable event source.
Processing engine
Flink, Kafka Streams, Spark, Beam — transform + aggregate events.
State store
RocksDB, in-memory — hold aggregations, join tables.
Sink: DB / log / alert
Cassandra, DynamoDB, another Kafka topic, PagerDuty.
Golden rule: if latency matters and data flows continuously, stream. If you just need "yesterday's report," batch is fine.

Historical timeline

  1. 1970s
    CEP and active DBs
    Complex event processing appears in academia. Snoop, TREAT, others. Ideas ahead of hardware.
  2. 1997
    Aurora + Borealis at Brown/MIT
    Academic streaming DBs. Introduce many concepts (windows, punctuation) later adopted by industrial systems.
  3. 2003
    Google MapReduce paper
    Batch processing becomes tractable at web scale. Ironically, sets the stage for the stream revolt.
  4. 2011
    Apache Kafka open-sourced
    LinkedIn releases Kafka. The log becomes a first-class primitive. Enables the whole stream-processing ecosystem.
  5. 2011
    Apache Storm at Twitter
    Nathan Marz builds Storm for real-time analytics at Twitter. First widely-used industrial stream processor.
  6. 2013
    Apache Samza at LinkedIn
    Kafka-native processor. Stateful. Fault-tolerant via changelog to Kafka.
  7. 2014
    Apache Flink from Stratosphere
    Berlin research project graduates to Apache. Native streaming with event-time + watermarks.
  8. 2014
    Spark Streaming (micro-batch)
    Spark adds streaming as tiny batches. Easier for existing Spark users.
  9. 2015
    Kreps: "Questioning the Lambda Architecture"
    Essay reframes stream vs batch as false dichotomy. Sparks "kappa architecture" discussion.
  10. 2016
    Kafka Streams library
    Streaming as a library in your app, not a separate cluster. Sub-second latency, exactly-once with Kafka transactions.
  11. 2016
    Apache Beam
    Google open-sources the Dataflow API as Apache Beam. Unified batch + stream.
  12. 2018
    Flink 1.5 exactly-once
    Flink matures exactly-once semantics for entire pipelines. Production-grade correctness.
  13. 2019
    Databricks Delta Lake
    Batch + streaming on same tables. Blurs the boundary further. Emerging "lakehouse" pattern.
  14. 2021
    Redpanda + Materialize
    New wave: Kafka-compatible in Rust/C++ (Redpanda), streaming SQL DB (Materialize). Modernizes the stack.
  15. 2024
    AI + streaming converges
    LLM inference on streams — Confluent Cloud AI, RisingWave, Bytewax. Real-time embeddings + RAG. New category emerging.

Batch vs Stream — the false dichotomy that shaped an industry

For years engineers debated batch vs streaming as if they were opposites. They're not. Jay Kreps: "Batch is just stream with a very large window."

Batch processing

Bounded input. Runs to completion. High throughput. High latency.

Every night at 2AM:
1. Read yesterday&apos;s events
2. Aggregate by user
3. Write to results table
4. Dashboard updated
Latency: hours-days
Throughput: TB/hour
Correctness: deterministic
Products: Spark, Hadoop MapReduce, Snowflake, BigQuery batch.

Stream processing

Unbounded input. Runs forever. Low latency. Continuous updates.

Continuously:
1. Subscribe to events topic
2. Aggregate per-user 5-min windows
3. Update user_activity table
4. Dashboard live-updates
Latency: ms-seconds
Throughput: lower per-record but continuous
Correctness: event-time + watermarks required
Products: Flink, Kafka Streams, Spark Structured Streaming, Beam.
Kappa architecture (Kreps 2014): just use streaming for everything. Replay from the log for historical processing. No separate batch pipeline. Cleaner, less code, one code path. Now the default for greenfield systems.

Windowing — the core abstraction of stream processing

Events flow forever. You want aggregates like "transactions in the last 5 minutes." The mechanism is windowing: chop the stream into bounded chunks. Four window types cover 99% of needs:

Tumbling windows

How: Non-overlapping fixed-size windows. Every event belongs to exactly one window.
5-min tumbling: [10:00-10:05], [10:05-10:10], [10:10-10:15]...
Use when: Simple periodic aggregations: "transactions per 5 minutes."
Event time vs processing time: events can arrive out of order (mobile clients batch uploads). Do you window by when the event happened (event time) or when it arrived (processing time)? Event-time + watermarks is the correct-but-harder choice. Flink and Beam support it natively; Kafka Streams approximates it.

Delivery semantics — the same problem, again

Stream processors face the same three delivery guarantees as message queues, plus one extra challenge: they must maintain state alongside the messages. Exactly-once is meaningful only when state changes are atomic with output emissions.

At-most-once

Process each event 0 or 1 times. On crash, lose in-flight work.
Rare. Only for tolerable-lossy work (metrics, logging).

At-least-once (default in most streamers)

On crash, replay from last committed offset. May reprocess some events.
Works if aggregations are idempotent. Common default in Storm, older Kafka Streams.

Exactly-once (Flink 1.5+, Kafka Streams with transactions)

State changes + output writes + offset commits happen inside a distributed transaction. Either all commit or none.
Needed for financial + strict-count aggregates. Adds ~10-20% overhead.
Flink's trick: asynchronous distributed snapshots (Chandy-Lamport algorithm). Barrier markers flow through the pipeline; every operator snapshots state when it sees the barrier. On failure, replay from last consistent snapshot. Elegant.

State: the thing that makes stream processing hard

Stateless stream processing (map, filter) is trivial. State makes it hard: a running count, a windowed aggregation, a join table. State must survive crashes without expensive restarts.

State store: RocksDB (Kafka Streams, Flink)

Embedded LSM-tree DB per operator instance. Handles GB-scale state per task. Local disk + memtable + SSTs.

State backup: changelog to Kafka

Every state mutation logged to a compacted Kafka topic. On failure, rebuild state by replaying the changelog. Zero-loss guarantee.

Checkpointing (Flink)

Periodic snapshots to distributed storage (S3, HDFS). Async. Restore from any recent checkpoint on failure.

Watermarks (Flink, Beam)

Metadata that flows in-band: "no more events with timestamp < T." Enables event-time window closing + late-event handling.

Recovery time is a KPI. If state is 100GB and your app crashes, does it take 30s or 30 minutes to be back? Small local state + fast Kafka replay = fast recovery. Giant snapshots to S3 = slow. Design state size deliberately.

Product comparison

ProductModelStrengthWeakness
Apache FlinkOSSNative streaming. Event-time. Exactly-once. Rich state APIs. Highest performance.JVM ops. Steep learning curve. Complex tuning.
Kafka StreamsOSS (library)No cluster to run — just a library in your Java/Kotlin service. Kafka-native.Kafka-only. Java-only. Limited to 1 topic partitioning strategy.
Apache Spark Structured StreamingOSSFamiliar Spark API. Batch + streaming unified. Big ecosystem.Micro-batch adds latency (100ms+ min). Event-time is bolted-on.
Apache StormOSSOriginal industrial streaming platform. Real-time (ms latency).Weak state APIs. Losing users to Flink.
Apache SamzaOSSKafka-native. Simple mental model. LinkedIn scale.Small community. Less-active development.
Apache BeamOSS (SDK)Unified batch + stream API. Portable across runners (Flink, Spark, Dataflow).Abstraction leaks. Slower than native runners.
Google Cloud DataflowManagedGoogle's Beam-native managed streaming. Autoscaling. Zero ops.GCP-only. Expensive at scale.
Amazon Kinesis Data AnalyticsManagedAWS-managed Flink (KDA). Autoscaling. Integrates with Kinesis Streams.AWS-only. Lags open-source Flink features.
MaterializeManaged + OSSStreaming SQL as a DB. Materialized views incrementally maintained on streams.Newer product. Cost at scale. SQL only.
RisingWaveOSS + CloudStreaming SQL DB. Cloud-native, Rust-based, Postgres-compatible.Younger community. Fewer connectors than Flink.
BytewaxOSS + CloudPython-native stream processing. AI-friendly.Newer. Fewer features than Flink. Python-only.
Databricks Delta Live TablesManagedDeclarative pipelines. Batch + streaming on Delta Lake. UI-first.Databricks-only. Expensive.

How to choose: Sub-second exactly-once + complex state → Flink. Simple app + Kafka already → Kafka Streams. Batch team + streaming ambition → Spark Structured Streaming. Managed AWS → Kinesis Data Analytics. SQL first → Materialize or RisingWave. Python + AI → Bytewax.

12 real-world stream processing deployments

Uber

Flink for ETA + surge pricing

Uber uses Apache Flink for real-time trip pricing, ETA calculations, driver dispatch scoring. Processes ~5M events/sec. State stores driver location, rider queue, historical patterns.

Alibaba

Flink at 4B events/sec (Singles Day)

Alibaba runs the world's largest Flink deployment for real-time analytics during Singles Day (Nov 11). Peaks: 4 billion events/second. Powers live dashboards, fraud detection, recommendations.

Netflix

Keystone streaming (Flink)

Netflix's Keystone pipeline uses Flink to enrich, aggregate, and route ~1 trillion events/day. Powers Video analytics, personalization signals, A/B test evaluation.

LinkedIn

Samza for real-time newsfeed

LinkedIn built Samza for real-time processing of the professional graph. Every profile update, connection request, feed interaction flows through Samza pipelines.

Cloudflare

Custom stream analytics with Kafka Streams

Cloudflare uses Kafka Streams for real-time DNS analytics, DDoS pattern detection, and bot classification. Sub-second decisions at the edge.

Robinhood

Faust (Python) for trade events

Robinhood used Faust (Python Kafka Streams port) for real-time trade event processing. Custom aggregations, fraud pattern detection. Now migrating to Flink.

Twitter

Heron replaced Storm

Twitter built Heron in 2015 as a Storm replacement. Better ops story, same programming model. Powers real-time analytics of tweets, trending topics.

Spotify

Beam on Dataflow for playlist ranking

Spotify uses Apache Beam on Google Dataflow for real-time playlist recommendations, listening analytics. Same code runs as batch for backfills.

PayPal

Kafka Streams for fraud

PayPal runs real-time fraud detection with Kafka Streams. Every transaction is scored against user history + patterns in milliseconds. Blocks fraud before it commits.

Shopify

Kafka Streams + Materialize

Shopify combines Kafka Streams for transformations and Materialize for streaming SQL analytics. Real-time merchant dashboards.

DoorDash

Flink for delivery timing

DoorDash uses Flink to predict delivery time in real-time — considering restaurant prep time, driver load, traffic. Updates continuously as conditions change.

Snowflake

Snowpipe streaming into warehouse

Snowflake's Snowpipe Streaming continuously ingests events into the warehouse. Blurs the streaming/batch line. Widely adopted for near-real-time BI.

Key takeaways

  • 1Stream processing is about continuous computation on unbounded data. Every stream framework is trying to solve: state + windowing + exactly-once.
  • 2Kreps was right: batch is just streaming with a huge window. Kappa architecture unifies both under the log.
  • 3Windowing comes in 4 flavors: tumbling, hopping, session, global. Pick based on your query semantics.
  • 4Event time vs processing time matters for correctness. Event time + watermarks is the right way when order matters.
  • 5Exactly-once requires atomic state + output + offset commits. Flink and Kafka Streams (with transactions) do this.
  • 6State store (RocksDB embedded, changelog to Kafka, checkpoints to S3) is the machinery. State size = recovery time.
  • 7For most workloads: Flink for complex state + correctness, Kafka Streams for simple app-scoped work, Spark Structured Streaming if already on Spark.
  • 8SQL-first (Materialize, RisingWave, Snowpipe Streaming) is the future for analytics teams. AI-native (Bytewax, RAG streams) is emerging.

References & further reading

  • • Kreps, J. (2014). "Questioning the Lambda Architecture." O'Reilly Radar. The essay that reframed the field.
  • • Kreps, J. (2014). "I Heart Logs." O'Reilly. The log as the primitive of distributed data.
  • • Akidau, T. et al. (2015). "The Dataflow Model." VLDB. Google's foundational paper. Basis for Beam.
  • • Carbone, P. et al. (2017). "State Management in Apache Flink." VLDB. How Flink handles state.
  • • Marz, N. (2015). Big Data. Manning. Introduces Lambda architecture (critiqued but foundational).
  • • Kleppmann, M. (2017). DDIA, Chapter 11 (Stream Processing). Essential.
  • • Akidau, T., Chernyak, S., Lax, R. (2018). Streaming Systems. O'Reilly. The definitive book on streaming.
  • • Flink Documentation: "Streaming Concepts" and "Fault Tolerance." Best free resource.
  • • Kafka Streams Docs: "Streams Concepts." Excellent conceptual walkthrough.
  • • Chandy, K. M. & Lamport, L. (1985). "Distributed Snapshots: Determining Global States." ACM TOCS. The algorithm behind Flink checkpoints.