Design a distributed cache
When one cache server is not enough, you need many. We design partitioning, replication, eviction and failure behaviour.
Goal: a fast, shared cache holding more data and handling more requests than one machine can, with graceful behaviour when nodes fail.
Requirements
- Very low latency (sub-millisecond to a few ms).
- Scale to many nodes; add or remove nodes with little disruption.
- Keep hot data in memory with sensible eviction.
- Survive node failure without taking down the database.
Partitioning
Split keys across nodes. Plain hash(key) mod N reshuffles almost everything when N changes, so use consistent hashing (with virtual nodes) or fixed hash slots (as Redis Cluster does with 16,384 slots mapped to nodes).
Where routing happens
- Smart client: the client library knows the layout and goes straight to the right node.
- Proxy: clients talk to a proxy that routes. Simpler clients, one more hop.
- Server redirect: nodes tell the client to ask another node (Redis Cluster style).
Replication and availability
Keep a replica for each partition. If the primary dies, promote the replica. For a pure cache you can also accept data loss: the database holds the truth, and missing keys are simply refilled on the next read. The danger is a sudden wave of misses hitting the database.
Eviction
Memory is limited: LRU, LFU or random eviction plus TTL. Keep a memory limit and sensible policy so the cache never grows without bound.
Consistency with the database
Usually cache-aside: read from cache, on a miss read the database and fill the cache. On update, write the database then delete the cache key. Deleting is safer than updating because it avoids overwriting with stale values in races.
Problems to design for
- Hot key: one very popular key overloads its node. Replicate it across nodes, add a small local in-process cache in front, or split it into several keys.
- Cache stampede: many requests rebuild an expired key at once. Use a lock so one request refills it, jitter on TTLs, and early refresh.
- Cache penetration: requests for non-existent keys always miss. Cache a short "not found" marker or use a Bloom filter.
- Cache avalanche: many keys expire together. Add random jitter to TTLs.
- Node loss: with consistent hashing only a fraction of keys are lost, but protect the database with rate limits and request coalescing.
Operations
Monitor hit rate, memory use, evictions, latency and network. A falling hit rate is an early warning.
Takeaway: partition with consistent hashing or hash slots, replicate for availability, evict with LRU and TTL, and design explicitly for hot keys and stampedes.
System Design series, Part 4 (designing the infrastructure): 2 of 4. Previous: Design a distributed unique ID generator (Snowflake).
