Rate limiting: token bucket, sliding window and how to build it
Rate limiting protects your service from abuse, bugs and traffic spikes. Learn the four main algorithms and how to run them across many servers.
A rate limiter caps how many requests a client can make in a period. It stops abuse and brute-force login attempts, protects shared resources and keeps one noisy client from hurting everyone else.
What to limit by
User ID, API key, IP address, or a mix. IP alone is weak: many people share one IP, and attackers rotate IPs.
The main algorithms
- Fixed window: count requests per minute and reset at the minute mark. Simple, but a burst across the boundary can allow nearly double the limit.
- Sliding window: count requests in the last 60 seconds from now. Smoother and more accurate, but needs more memory.
- Token bucket: a bucket refills with tokens at a steady rate; each request takes one. Allows short bursts up to the bucket size while keeping a steady average. Very popular.
- Leaky bucket: requests enter a queue and leave at a fixed rate. Gives a smooth output, but bursts wait or are dropped.
Running it on many servers
Counters kept in each server's memory do not add up. Use a shared fast store such as Redis with atomic operations (INCR with expiry, or a Lua script) so the check and update happen together.
What the client should see
Return HTTP 429 Too Many Requests with a Retry-After header, and headers showing the limit and what remains. Clients can then back off properly.
Where to put it
At the edge or API gateway, so abusive traffic is dropped early. Add stricter limits on sensitive actions like login and password reset.
Decisions to make
Fail open or closed if Redis is down? Different limits per plan? Allow bursts? Write these down.
Takeaway: token bucket in Redis, keyed by user or API key, returning 429 with Retry-After, is a solid default.
Part 11 of 15 in the System Design series. Previous: Consistent hashing: how to add servers without moving everything.
