Latency vs throughput: the two numbers every system design starts with
Latency and throughput are not the same, and improving one can hurt the other. Learn p50, p95 and p99, and why the average hides real problems.
Latency is how long one request takes, from sent to answered. Throughput is how many requests the system completes per unit of time, such as requests per second.
A road is a fair picture: latency is how long one car takes to drive it, throughput is how many cars pass per minute. Adding lanes raises throughput but does not make one car arrive sooner.
They pull against each other
Batching is the classic example. Grouping 100 writes into one database call raises throughput, but the first write now waits for the other 99, so its latency goes up.
Use percentiles, not averages
An average hides pain. If 99 requests take 50 ms and one takes 5 seconds, the average looks fine while one user in a hundred is miserable.
- p50 (median): half of requests are faster than this.
- p95 / p99: 5% or 1% of requests are slower than this.
Look at p99 for user experience. This is called tail latency, and it matters more in systems that call many services, because the slowest call decides the total.
A useful rule
Little's Law: concurrent requests = arrival rate x average time per request. If you get 200 requests per second and each takes 0.5 s, about 100 are in flight at once. This tells you how many workers or connections you need.
How to improve each
- Lower latency: cache, move data closer to users (CDN), cut network hops, add indexes, do less work per request.
- Raise throughput: add servers, use async processing and queues, batch work, use connection pooling.
Takeaway: state both targets up front, for example "p99 under 300 ms at 2,000 requests per second", and measure percentiles, not averages.
Part 2 of 15 in the System Design series. Previous: Scalability explained: vertical vs horizontal scaling.
