Retry with exponential backoff + jitter

Problem

Transient failures (network hiccup, brief overload) succeed on retry. Retrying immediately makes overload worse (retry storm).

Context

Any client calling a flaky dependency.

Solution

On failure, wait an exponentially-growing amount (100ms, 200ms, 400ms, 800ms...) plus random jitter (± 50%). Cap the max backoff (30s). Cap total retries (5). Also: don't retry non-retryable errors (400, 403).

Trade-offs

• Long-tail latency: successful requests are fast, retries make failures slow
• Retry storm risk if backoff/jitter is wrong
• Idempotency required — retries can double-charge or double-post if not

Failure modes

• No jitter → all clients retry at the same moment (thundering herd)
• No cap → retries forever (blocks user)
• Retrying non-idempotent operations causes duplicates

When to use

• Any RPC or HTTP call to a downstream service
• Message queue processing
• Database transactions on retryable errors (deadlock, timeout)

When NOT to use

• 4xx client errors (they'll keep failing)
• Non-idempotent operations without idempotency keys
• Real-time paths where latency matters more than success