Services calling services synchronously is fine until the third one is down. Brokers are how modern systems hand off work across failures, across time, and across teams. The infrastructure layer powering Uber, LinkedIn, Stripe, and most of the modern internet's spine.
A user clicks "place order." Your order service writes to the database, calls the payment service, calls the inventory service, calls the email service, calls the analytics service, calls the recommendation service. Six synchronous network calls happen before the user sees a confirmation page. Each call has its own latency, its own failure rate, its own deploy cadence, its own team. Any one of them being slow or down makes the entire chain slow or down. This is the architecture that worked for the first ten engineers and breaks for the next hundred.
~300ms floor, plus tail. Your user waits.One service's bad day = everyone's bad day.Six consumers = six lines of fragile code in your hot path.Data lost forever.The shorthand for what a broker provides is decoupling along three axes: spatial (producer and consumer don't need each other's address), temporal (they don't need to be up at the same time), and logical (one producer's event can drive many independent consumers). Add to that durability (the broker is the system of record for in-flight events), and you have the four properties that made event-driven architecture eat the world from 2011 onward.
The rest of this module covers the broker landscape (§02 — queues vs logs, the two paradigms), Kafka's anatomy in detail (§03), a live simulator that walks through partitioning and consumer-group rebalancing (§04), and the delivery semantics that decide what kinds of correctness you actually get (§05).
Every message broker is some variation on one of two designs: a queue (where consuming a message removes it) or a log (where consuming reads from a position; the message stays). Both exist for good reasons; they fit different workloads. Picking the wrong one is one of the most common architecture mistakes in mid-stage startups.
Messages live in a queue. Workers pull one at a time, process it, and ack. On ack, the message is deleted. If a worker dies mid-processing, the message returns to the queue for another worker. Multiple workers compete for messages — load is balanced naturally. Built for task distribution: each job done exactly once by one worker.
Messages are appended to an immutable log. Each consumer maintains its own offset — a position in the log it's reading from. Reads don't remove anything. Many consumers can read the same stream at their own pace, going backward to replay or forward to catch up. Built for event streams: a published event drives many independent reactions.
The mental shorthand: queues are for tasks; logs are for events. Tasks have one rightful executor (process this payment, send this email, encode this video) — you want exactly one worker to claim it. Events are facts that may have many independent consequences (user signed up — kick off welcome email, create profile, log to analytics, train recommender). Pick a queue when there's work to be done; pick a log when there are facts to be broadcast.
The reason logs took over the modern event-driven world (Kafka shipping at LinkedIn in 2011, then becoming the lingua franca everywhere) is that logs subsume queues. You can implement queue semantics on top of a log (single consumer group, ack via offset commit) — but you can't implement log replay on top of a queue. The log is the more powerful primitive. For new systems with diverse downstream consumers, defaulting to a log is almost always the right call. NATS, Kinesis, and Redis Streams all followed Kafka's lead. RabbitMQ added log-like streams in 2021. The industry voted.
Since logs won, and Kafka is the canonical log broker, every engineer working on event-driven systems needs to understand its anatomy. Five concepts. The whole architecture pivots on them. The same vocabulary applies to Pulsar, Kinesis, Redpanda, NATS streams — Kafka set the schema.
A named stream of events of a single logical type. Producers publish to a topic; consumers subscribe to a topic. A typical Kafka cluster has hundreds or thousands of topics: user-events, order-placed, page-views, payment-completed, etc.
A topic is split into N partitions — each is an independent ordered log. Messages with the same key always land in the same partition (hash(key) % N). Partitions are the unit of parallelism: each partition can be consumed by a different worker concurrently.
A monotonically increasing integer identifying each message's position within its partition. Consumers track their progress as offsets — "I've processed up to offset 1,247." Offsets are how Kafka does replay: reset your offset to 0 and re-read everything.
A set of consumers cooperating to consume a topic. Each partition is assigned to exactly one consumer in the group. Add a consumer → partitions get rebalanced. Each group has its own offset state, so multiple groups can consume the same topic independently — analytics, billing, and search can all read the same stream at their own pace.
A Kafka cluster is multiple broker servers, with partitions replicated across them. One broker is leader for each partition; others are followers. Replication factor 3 is standard — survives 2 broker failures. Producers and consumers connect to the leader; followers exist for durability.
The two most important consequences to internalize: (1) ordering is only guaranteed within a partition, not across them. If you need strict global order, you need one partition (and you give up parallelism). If you need order per user, use the user ID as the message key — all that user's events land in the same partition and get processed in order. (2) Consumer groups give you both load distribution and broadcast for free. Multiple consumers in one group share the work; multiple groups each read the entire stream. This is the schema that lets Kafka power a dozen downstream systems off one event stream.
Below: a Kafka topic with 4 partitions, a consumer group, and a stepping scenario that demonstrates partitioning by key, consumer assignment, and the rebalance dance that happens when consumers join or leave the group. Eight frames; by the end, the mental model is concrete.
user-events with 4 partitions, one consumer group "analytics" starting with a single consumer. Watch what happens as the group grows and shrinks.Every broker promises to deliver messages. The fine print is how reliably — and the three options have very different cost and correctness implications. The choice has to match the work the consumer is doing: logging vs counting vs charging credit cards demand different guarantees. Pick the wrong one and either lose data or duplicate operations. Both are bad in different ways.
Producer sends without waiting for ack; consumer commits offset before processing. If anything fails, the message is gone. Cheapest, lowest latency. Lossy by design.
Producer retries until broker acks; consumer commits offset only after processing. On crash mid-processing, the message is re-delivered. Duplicates are possible — the consumer must be idempotent.
Requires transactional commits across broker, consumer, and any downstream effects. The hardest guarantee. Kafka's exactly-once semantics (since 0.11) work for Kafka-to-Kafka pipelines but get tricky when external systems are involved.
The honest answer most teams converge on: at-least-once delivery + idempotent consumers. The infrastructure side is simpler (no distributed transaction across systems), and the application side just has to deduplicate — usually by attaching a unique event ID and checking "have I processed this before?" in the consumer. This pattern is so common it has a name: the "outbox pattern", where producers write events to an outbox table within their database transaction, then a separate process publishes from the outbox to the broker. Effectively gives you exactly-once at the boundary, with at-least-once underneath.
One critical pitfall: don't confuse "exactly-once delivery" with "exactly-once processing". Kafka can guarantee the message reaches the consumer once — but if your consumer crashes after processing but before acking, the message gets redelivered. To get true exactly-once effects, you need idempotency in the handler. Marketing copy often blurs this distinction. The papers and the production reality don't. M.31 will dig in.
The terms every Kafka tutorial, every event-driven architecture review, and every distributed log paper assume you already know.
hash(key) % N. The unit of parallelism — one consumer per partition per group.acks=all). The Kafka equivalent of the Raft quorum from M.24.Test the broker intuition. Click an answer; explanation drops in instantly.
Perfect. Brokers, Kafka anatomy, delivery semantics — all internalized. Next: pub/sub patterns and topologies.
The infrastructure layer that makes everything from here onward possible. These ideas return in every event-driven module.
Producers don't know consumers. Consumers don't know producers. The broker is the contract. Spatial, temporal, and logical decoupling — the three axes that make systems composable.
Logs subsume queues. Logs let you replay, fan out to many independent consumers, and add new use cases without touching producers. The default for new systems.
The pragmatic default. Exactly-once is technically possible but expensive and brittle across boundaries. Build idempotency into your consumers; sleep at night.