foundations
Batch vs streaming processing
8 min read
Fully authored
Two paradigms for computing over data, and where each wins.
Batch runs over BOUNDED data at scheduled times; streaming runs over UNBOUNDED data continuously. The choice sets your latency floor, cost, and mental model.
| Dim | Batch | Streaming |
|---|---|---|
| Latency | Hours-days | Milliseconds |
| Complexity | Simple | State mgmt, watermarks, event time |
| Cost per record | ~$0.001 | ~$0.01 |
| Backfill | Easy — rerun | Hard — replay from Kafka |
| Failure recovery | Idempotent rerun | Checkpointing + exactly-once |
| Best for | Reports, ML training, warehousing | Fraud detection, alerts, real-time dashboards |
Kappa architecture — the modern answer
Instead of running BOTH batch and streaming (Lambda architecture), run ONLY streaming with Kafka replay for backfill. Simpler codebase, one language, one framework. Uber, LinkedIn, and Netflix all migrated from Lambda to Kappa in the 2018-2020 window.
Practice what you just read
Every foundation concept has a companion quiz to close the loop.