Backpressure

Problem

Producers publish faster than consumers can process. Queues grow unbounded, memory fills up, and everything crashes.

Context

Any producer-consumer relationship where the two sides can drift in throughput.

Solution

Consumers signal upstream when they're overwhelmed (queue depth threshold, memory pressure). Producers slow down (block writes, return 429, buffer to disk, or drop low-priority messages). Reactive Streams codifies this with 'demand-based' consumption — consumers pull only what they can handle.

Trade-offs

• Producers must handle the 'slow down' signal — either block, drop, or apply their own backpressure upstream
• Slower producers = slower end-to-end throughput (that's the point)
• Backpressure signals can be racy — by the time producer sees the signal, queue has grown more

Failure modes

• Producer ignores backpressure and buffers locally → memory blowup
• Backpressure signal is delayed → queue grows before signal reaches producer
• Cascading backpressure blocks the whole pipeline

When to use

• Streaming pipelines (Flink, Kafka Streams)
• Async processing where consumers can be slow
• Any bounded resource downstream that can be overwhelmed

When NOT to use

• Synchronous request/response — the client waiting is natural backpressure
• Fire-and-forget with drop-OK semantics