Apache Kafka explained: topics, partitions and consumer groups
Kafka stores streams of events in a distributed log that many readers can replay. Here are the core ideas and the trade-offs.
Apache Kafka is a distributed event log. Producers append events, Kafka stores them durably, and consumers read them at their own pace. Unlike a classic queue, a message is not deleted when read. It stays for a set retention time, so many consumers can read it and you can replay history.
Core ideas
- Topic: a named stream, such as
orders. - Partition: a topic is split into partitions. Each partition is an ordered, append-only log. Partitions are how Kafka scales.
- Offset: the position of a message in a partition. Consumers remember where they stopped.
- Broker: a Kafka server. A cluster has several.
Ordering and keys
Order is guaranteed only within a partition. A producer picks the partition by a key: all messages with the same key (say user_id) go to the same partition and stay in order.
Consumer groups
Consumers in a consumer group share the work: each partition is read by exactly one consumer in the group. More consumers means more parallelism, up to the number of partitions. A different group gets its own full copy of the stream. This is how one event can feed billing, analytics and search separately.
Durability
Each partition has a leader and replicas on other brokers. With a replication factor of 3, one broker can die without losing data. Producers choose how many acknowledgements they wait for, which trades speed for safety. Recent Kafka versions manage cluster metadata themselves (KRaft) instead of depending on ZooKeeper.
When to use it
- High-volume event streams, logs and metrics.
- Many independent consumers of the same data.
- Event sourcing and replay.
- Data pipelines between systems.
When not to
For a simple background job queue with low volume, a task queue like RabbitMQ or SQS is easier to run. Kafka is powerful but has real operating cost.
Takeaway: think of Kafka as a durable, replayable log. Pick keys carefully, size partitions for your parallelism, and use it where its strengths matter.
System Design series, Part 2 (building blocks): topic 2 of 14. Previous: Redis internals: why it is fast and how to use it safely.
