Leader election: how distributed systems pick one boss
Many systems need exactly one node in charge. Learn how elections work, why majorities matter and how to stop two leaders at once.
Some jobs must be done by one node at a time: accepting writes to a primary database, running a scheduled job, assigning partitions. If two nodes both think they are in charge, you get duplicated work or corrupted data (split brain). Leader election chooses one leader and replaces it when it fails.
The core idea
Nodes need to agree on who leads, even when messages are delayed or nodes crash. The standard answer is a majority (quorum). A candidate becomes leader only if more than half of the nodes vote for it. Two majorities always overlap, so two leaders cannot both be elected for the same period. This is why clusters use an odd number of nodes, like 3 or 5.
Raft in short
Raft is a widely used consensus algorithm (etcd and Consul use it).
- Nodes are followers. The leader sends regular heartbeats.
- If a follower hears no heartbeat for a random timeout, it becomes a candidate, increases its term (an election number) and asks for votes.
- Each node votes once per term. A candidate with a majority becomes leader.
- The random timeout makes it unlikely that two candidates split the vote repeatedly.
A node that sees a higher term steps down. Old leaders are therefore pushed out.
Using an external coordinator
Instead of implementing consensus, apps often use etcd, ZooKeeper or Consul. Each candidate tries to create a key or lock with a lease (a time limit). The one that succeeds is leader and must keep renewing the lease. If it dies, the lease expires and others compete.
The dangerous case: a stale leader
A leader can pause (for example a long garbage collection), lose its lease, and then wake up still thinking it leads. Protect against it with a fencing token: a number that increases with each leadership term. Storage rejects any write with an older token than the newest it has seen.
Practical advice
- Prefer a managed or proven system over writing your own.
- Keep lease and timeout values larger than normal network jitter.
- Make leader-only work idempotent, so a rare overlap does no harm.
Takeaway: elect leaders with a majority vote, track terms, and use fencing tokens so a stale leader cannot do damage.
System Design series, Part 2 (building blocks): topic 9 of 14. Previous: Bloom filter: checking membership with tiny memory.
