Circuit breaker and bulkhead: stop one failure from spreading
When one dependency fails, it can drag everything down with it. Circuit breakers and bulkheads contain the damage.
Failures spread. A slow database makes your API threads wait, the threads run out, and now even healthy endpoints stop answering. Two patterns stop this.
Circuit breaker
Like an electrical breaker, it opens when things go wrong so the fault does not burn the house.
Three states:
- Closed: calls go through normally. The breaker counts failures.
- Open: after too many failures (for example 50% errors in the last 10 seconds), calls fail immediately without touching the broken service. This protects it and gives it time to recover, and it saves your own threads.
- Half-open: after a waiting period, a few trial calls are allowed. If they succeed, close the breaker. If not, open it again.
Tune the failure threshold, the minimum number of calls before judging and the wait time.
What to do when the circuit is open
Return something useful, a fallback:
- A cached or default value (for example "recommendations unavailable", show popular items).
- A reduced feature, such as hiding the section.
- A clear error that the caller can handle quickly.
Bulkhead
Ships have sealed compartments so a hole floods only one. A bulkhead isolates resources so one dependency cannot use them all:
- A separate thread or connection pool per dependency. If the payment service hangs, it exhausts only the payment pool, while search and login still work.
- Separate pools or queues for critical vs. low-priority traffic.
- Per-tenant or per-client limits, so one heavy user does not starve others.
Load shedding
When the system is overloaded, accept less and answer fast rather than accept everything and time out. Drop low-priority requests first, return 503 with Retry-After, and protect core paths such as login and checkout.
Graceful degradation
Design features as optional layers. A news page can load without comments, recommendations or view counts. Decide in advance which parts are essential.
Test it
Practise failure: kill a dependency in staging, add artificial latency and check that breakers open, fallbacks appear and recovery works. Untested resilience code usually fails when needed.
Takeaway: isolate dependencies with bulkheads, stop calling failing ones with circuit breakers, shed load early, and always have a fallback.
System Design series, Part 5 (failure and security): 2 of 6. Previous: Timeouts, retries and backoff: how not to cause a retry storm.
