The idea: keep the latest value of every key
A normal Kafka topic deletes old data by age or size (cleanup.policy=delete). Some topics are not a stream of events but a changelog: every record says "the current value of key K is now V". For those, the old values are useless once a newer one exists, but the newest value of every key must stay forever. cleanup.policy=compact does exactly that: a background thread, the log cleaner, rewrites the partition's old segments and keeps only the latest record per key.
The page runs one partition of a topic users: the key is a user (ann, bob, cal, dee, eve, fay) and the value is that user's email. A cell shows the key and which email it is: e3 means ann3@mail.io, ann's third address. The numbers are scaled so that everything fits: a segment holds 4 records, one tick of the clock is about 5 seconds, and the time settings are given in ticks.
Clean and dirty parts, and the active segment
A partition is stored as a series of segment files, each named after its base offset (the first offset in it). Only the last one, the active segment, is written to; when it reaches segment.bytes it is closed and a new one starts (it rolls). The cleaner never touches the active segment, which is why the newest records always stay as they were written (Demo: compact a user→email changelog leaves a duplicate of ann in it).
The cleaner remembers how far it got, in a file called cleaner-offset-checkpoint. Everything before the checkpoint is the clean part (the Kafka docs call it the log tail): it has been compacted and has at most one record per key. Everything after it is the dirty part (the head): records written since the last pass, duplicates included. On the canvas the clean segments are green, the dirty ones yellow, the active one blue, and the blue bar is the checkpoint.
When does the cleaner run? The dirty ratio
Recopying a log costs disk reads and writes, so the cleaner does not run after every write. Each time it wakes up (every log.cleaner.backoff.ms, 15 s by default) it picks the log with the highest dirty ratio:
dirty ratio = dirty bytes / (clean bytes + dirty bytes)
and cleans it only if that ratio is at least min.cleanable.dirty.ratio (0.5 by default). With 0.5, at most half of the log can be duplicates waiting to be removed, and each pass removes at least as much as it has to copy. Demo: dirty ratio below 0.5 first compacts the log to 6 clean records, then adds one segment of 4 dirty records: 4 / 10 = 0.4, so the cleaner sleeps again. One more segment makes it 8 / 14 = 0.57, and the next wake-up cleans. The page counts records instead of bytes.
One pass: the offset map, then the recopy
- Build the offset map. The cleaner reads the dirty part from the checkpoint up to the first uncleanable offset (the active segment's base, or an earlier segment that is still too new, see below) and fills a hash table
key → latest offset. The real map stores a 16-byte hash of each key, inlog.cleaner.dedupe.buffer.size(128 MB shared by the cleaner threads); if it fills up, the pass stops early and the rest waits for the next pass. - Recopy. It then reads every segment from the start of the log up to the first uncleanable offset, the clean ones too, and writes a new
.cleanedfile. A record is copied only if the map has no newer offset for its key. Small segments are merged so that the log does not end up as many tiny files. - Swap. The cleaned files replace the old ones (
.cleaned→.swap→.log), and the checkpoint moves to the first uncleanable offset.
Reads and writes go on during the pass: producers keep appending to the active segment and consumers keep fetching, because the old files stay readable until the swap.
Offsets keep their gaps
A record keeps its offset for its whole life; the cleaner never renumbers. So a compacted log has holes, and a consumer reading it sees offsets that increase but jump: Demo: a late consumer rebuilds state reads offsets 3, 5, 6, 7, 8, 9, 10. A consumer's position is always valid: if it points at a removed offset, the fetch simply returns the next record that exists. Code that assumes offset + 1 is the next record breaks on a compacted topic.
The important promise is this: a consumer that reads a compacted partition from the beginning ends with the latest value of every key, the same table it would get by reading every write ever made. That is what lets a new service instance, a cache, or a Kafka Streams state store rebuild its state from the topic.
Tombstones: deleting a key
Compaction keeps the latest record of a key forever, so deleting a key needs a record of its own: a tombstone, a record with the key and a null value. When the cleaner sees it, all older records of that key are dropped like any other duplicate. The tombstone itself must stay for a while, or a consumer that was offline would never learn that the key was deleted. It is kept for delete.retention.ms (24 hours by default) after the pass that first cleaned it (KIP-534), and a later pass removes it (Demo: a tombstone deletes a key). A consumer that is more than delete.retention.ms behind can miss deletes.
min.compaction.lag.ms: let recent records be
By default a closed segment can be compacted right away. min.compaction.lag.ms sets a minimum age: a segment that still holds a record younger than that is uncleanable, so the cleaner stops at its base offset. Consumers then get a guaranteed window in which they see every update, not only the latest. In Demo: min.compaction.lag.ms the lag is 5 ticks: the first pass cleans the old segment but leaves the newer one, duplicates and all; three ticks later it is old enough and the next pass cleans it. (max.compaction.lag.ms works the other way: it forces a log to be cleaned after that time even when the dirty ratio is low.)
compact,delete: retention too
With cleanup.policy=compact,delete both rules apply. The log is compacted, and in addition a segment whose newest record is older than retention.ms is deleted as a whole file, whatever keys it holds. A key whose only remaining record sits in that segment is gone. This suits data that is only useful for a while, such as windowed aggregates in Kafka Streams. Demo: compact,delete (retention too) shows dee, eve and fay disappearing with an old segment while the other keys survive in newer ones.
Where compaction is used
| Topic | Key → value |
|---|---|
__consumer_offsets | (group, topic, partition) → committed offset; only the last commit matters |
| Kafka Streams changelog topics | state store key → value, so a store can be rebuilt on another machine |
Kafka Connect config, offset, status topics | connector → configuration, source position, status |
| CDC / entity topics | primary key → latest row (a delete in the database becomes a tombstone) |
__transaction_state, KRaft's __cluster_metadata snapshots | id → latest state |
See also How Kafka Moves a Message (partitions, replication, consumer groups), Kafka Consumer Groups: Rebalancing (which commits offsets into the compacted __consumer_offsets topic) and the other Kafka internals pages: log segments and retention, replication, KRaft and zero-copy reads. An LSM tree merges sorted files and keeps the newest version of each key in much the same way: LSM Tree vs B-Tree. A compacted topic is often the target of Change Data Capture.
Compaction keeps only the latest record per key, which is why a compacted topic is not an event store; see Event Sourcing and CQRS.
What the page leaves out
Bytes (the page counts records), the hashed keys and the size limit of the offset map, several cleaner threads (log.cleaner.threads) choosing among many logs, I/O throttling (log.cleaner.io.max.bytes.per.second), how Kafka groups segments by their size before recopying them (the page merges the kept records), segment rolling by time (segment.ms), max.compaction.lag.ms, the high watermark and transaction markers (the cleaner never goes past the last stable offset), compressed batches, replicas (each broker compacts its own copy independently) and the retention check running on its own thread.