Circuit breaker

Problem

A downstream service is unhealthy. Your service keeps calling it, waiting the full timeout on every call, exhausting your thread pool, and cascading the failure back to your callers.

Context

You have a synchronous dependency (database, third-party API, another microservice) that occasionally fails or degrades. Without protection, its failures propagate through your service and become YOUR failures.

Solution

Wrap calls to the dependency in a state machine with three states: CLOSED (calls flow through normally, we count failures), OPEN (calls fail fast without hitting the dependency; a timer runs), HALF-OPEN (after the timer, we let one probe call through). If the probe succeeds, close the circuit; if it fails, re-open. This lets the dependency recover without being flooded, and keeps your service responsive.

Trade-offs

• When OPEN, real requests fail fast — you're trading some correctness for a lot of availability
• Tuning the failure threshold and reset timer is finicky; wrong values cause thrashing
• Adds latency on every call (state machine check + occasional probe)
• Requires a fallback story: what do you return when the circuit is open?

Failure modes

• Threshold too sensitive → circuit opens on transient blips; too slack → doesn't protect
• Reset timer too short → probes flood the recovering dependency (thundering herd)
• No fallback → circuit opens, but callers still see errors — you just moved the failure
• Shared circuit state across many instances requires a coordinator (Redis, in-memory sync) or accepts local decisions

When to use

• Every synchronous call to an external dependency
• You have a meaningful fallback (cached result, default response, degraded feature)
• The dependency is known to sometimes degrade (which is all of them)

When NOT to use

• Async messaging — the queue is the buffer; a circuit breaker is redundant
• In-process function calls that never fail transiently