Timeouts, retries and backoff: how not to cause a retry storm

Every network call can be slow or fail. How you handle that decides whether a small problem stays small.

Always set a timeout

A call without a timeout can hang forever and hold a thread, a connection and memory. Set a timeout on every outgoing call: connect, read and total. Choose it from real latency numbers: a little above the p99, not 60 seconds "to be safe".

Retry only when it is safe

A retry helps when the failure is temporary: a timeout, a dropped connection, a 503. It does not help with a 400 or a 404. And you may retry only operations that are idempotent (see the idempotency part), or a retry can charge a customer twice.

Exponential backoff with jitter

If a service is struggling and every client retries at once, the retries become the attack. Wait longer after each failure, and add randomness:

wait = random(0, min(cap, base * 2^attempt))

• Exponential: 100 ms, 200 ms, 400 ms, 800 ms and so on, up to a cap.
• Jitter: a random part so thousands of clients do not retry at the same moment ("thundering herd").
• A maximum number of attempts (often 2 to 4).

The multiplication problem

Suppose service A calls B, B calls C, and each layer retries 3 times. One failing C can receive 3 x 3 = 9 calls for each user request, and the load grows with every layer. This is a retry storm, and it can turn a slow dependency into a full outage.

Defences:

• Retry in one layer only, usually the one closest to the user.
• A retry budget: allow retries only up to a percentage (for example 10%) of normal traffic.
• Pass a deadline down the call chain so lower layers stop working on requests that the caller already gave up on.

Other useful ideas

• Fail fast: if a dependency is clearly down, stop calling it (see the circuit breaker part).
• Hedged requests: for rare slow calls, send a second request after a short wait and take the first answer. Use only for safe reads.
• Log every retry so you can see them in monitoring.

Takeaway: timeout everything, retry only temporary and idempotent failures, back off with jitter, and cap total retries across layers.

---

System Design series, Part 5 (failure and security): 1 of 6.