Distributed locks: how to build one and why it is hard
A distributed lock keeps two servers from doing the same job at once. Learn the Redis recipe, its dangers, and when to avoid locks entirely.
When several servers share a resource, you sometimes need mutual exclusion: only one may act at a time, for example one worker running a payout, or one process refreshing a cache entry. A distributed lock provides that across machines.
The basic Redis recipe
To acquire: SET lock:payout <unique-id> NX PX 30000
NX: set only if the key does not exist, so only one caller wins.PX 30000: expire after 30 seconds, so a crashed holder cannot block everyone forever.<unique-id>: a random value that identifies the holder.
To release: delete the key only if its value is still your id, done atomically in a small Lua script. Otherwise you might delete a lock that someone else acquired after yours expired.
Why it is harder than it looks
- Expiry vs long work. If your job runs longer than the lock time, the lock expires while you still think you hold it, and a second worker enters. Either renew the lock while working or keep the critical section short.
- Pauses. A process can freeze (garbage collection, a slow disk, a VM pause), wake up after its lock expired, and write anyway.
- Failover. If the Redis primary dies before replicating the lock to a replica, a new primary may not know about it.
- Clock and network assumptions. Distributed time is unreliable.
Redis proposed Redlock (locking on several independent Redis nodes). Distributed-systems researchers have argued it is unsafe when correctness truly depends on the lock, and its author disagreed. The practical reading: use Redis locks for efficiency (avoiding duplicate work), not for correctness.
A safer design: fencing tokens
Each lock grant gets an increasing number. The protected resource (database, storage) rejects writes carrying an older number than the latest it has seen, so a stale holder is harmless. This needs a store that checks the token.
Alternatives to a lock
- A unique constraint or conditional update in the database (
UPDATE ... WHERE version = ?). - Idempotency keys so a repeated action is harmless.
- A queue where one consumer handles each key.
- Locks from etcd or ZooKeeper, which rely on consensus and give stronger guarantees.
Takeaway: use locks sparingly, always set an expiry and a unique owner value, and for anything that must be correct, back it with fencing tokens or database constraints.
System Design series, Part 2 (building blocks): topic 10 of 14. Previous: Leader election: how distributed systems pick one boss.
