The idea: who decides when a message moves?
A message broker sits between producers and consumers. It can hand messages over in two ways. With push, the broker sends each message to the consumer as soon as it can, and the consumer says "done" with an ack. With pull, the consumer asks for the next records when it is ready, and remembers how far it got with an offset. RabbitMQ is the best-known push broker; Kafka is the best-known pull broker. The page sends the same 10 messages, one per tick, through both and counts what happens.
Push: RabbitMQ and prefetch_count
A RabbitMQ consumer subscribes with basic.consume. From then on the broker sends (basic.deliver) the next ready message whenever the consumer has fewer unacked messages than its prefetch_count (basic.qos). Delivered messages wait in the client's buffer until the handler takes them. When the handler is done, the client sends basic.ack and the broker deletes the message. If the consumer dies before it acks, the broker puts its unacked messages back in the queue and delivers them again.
The prefetch window is the only brake. The page's window of 3 fits its 3-tick loop exactly (deliver, handle, ack), so a fast consumer is never idle. With prefetch_count = 0 there is no limit at all: the broker sends everything it has (Demo: slow consumer, no prefetch limit).
Pull: Kafka offsets and lag
A Kafka producer appends each record to a partition log, where it gets the next offset. The log keeps records until retention removes them (retention.ms, 7 days by default), whether anyone has read them or not. A consumer calls poll(), which sends Fetch from its current position and gets at most max.poll.records records (500 by default; 3 here). When the batch is handled it commits its offset. If nothing new is in the log, the broker holds the fetch until data arrives (long polling with fetch.min.bytes and fetch.max.wait.ms).
Lag is the log end offset minus the committed offset: how far the consumer is behind. A growing lag is a number on a dashboard, not a crash. The consumer never holds more than one batch.
A slow consumer: where does the backlog go?
When messages arrive faster than the consumer can handle them, the extra messages have to wait somewhere. With push and no prefetch limit they all go to the consumer. In Demo: slow consumer, no prefetch limit the buffer reaches 6 messages at t = 10, past the client's limit of 5, and the consumer is OOM-killed after handling only 2. The broker requeues the 7 unacked messages, including m2, which was half done and will run a second time. A restarted consumer would get the same flood again.
With prefetch_count = 3 (Demo: slow consumer, prefetch 3) the backlog stays in the broker's queue and the consumer never holds more than 2 waiting messages. With pull the backlog stays in the log as lag, and the consumer never holds more than one batch of 3. Both are safe, and push even finishes a little sooner (t = 33 vs t = 39), because a pull consumer waits one round trip for each new batch.
Replay and fan-out
A Kafka consumer can seek back to offset 0 and read the whole log again, for example after fixing a bug in its handler or to fill a new database (Demo: replay from offset 0: 10 records read again). A RabbitMQ queue cannot do that: an acked message is gone, so the producer would have to send it again. (RabbitMQ streams, added in 3.9, are an append-only log with offsets for exactly this case.)
For fan-out, RabbitMQ binds one queue per service to a fanout exchange and copies every message into each queue. Kafka writes each record once, and each consumer group keeps its own offset in the same log. In Demo: fan-out to two services both deliver 20 messages, but push writes 20 copies and pull writes 10. The slower analytics service simply has its own lag.
When to pick which
| Push (RabbitMQ queue) | Pull (Kafka log) | |
|---|---|---|
| Latency when the consumer keeps up | Lower: 2.0 ticks on average here | Higher: 5.6 ticks (waits for the next fetch) |
| Slow consumer | Needs a prefetch limit, or the consumer is flooded | Safe: the backlog is lag in the log |
| After a message is handled | Deleted on ack | Kept until retention |
| Replay | No (producer must resend) | Yes: seek to an offset |
| Fan-out to N services | N queues, N copies | One log, N offsets |
| Consumer ↔ broker messages (fast demo) | 20: one deliver and one ack per message | 13: batches of up to 3 |
| Good fit | Task queues, per-message routing, retries and dead-lettering | Event streams, many readers, reprocessing, high throughput |
See also How Kafka Moves a Message for the full producer-to-consumer path, Backpressure for the same "fast producer, slow consumer" problem inside one process, and Observer vs Publish–Subscribe for where brokers come from.
What the page leaves out
One partition, one queue per service and one consumer per group: no partitions spread across a group, no rebalancing, no competing consumers on one queue. No message is lost or reordered and nobody crashes except the flooded consumer. It leaves out publisher confirms and producer acks, durable vs transient queues, dead-letter exchanges, basic.nack and requeue loops, auto-ack mode, quorum queues and RabbitMQ streams, Kafka's retention and compaction, auto-commit timing, exactly-once transactions, and the fact that Kafka fetches by bytes as well as record count. Ticks are a teaching clock, not real latencies.