Conversation from Confluent Community Slack:
Jonathan Olsson
Hi! After some experimentation and discussions in online forums I think I have concluded that since compaction only runs on non-active segments and since a newly created segment only becomes active when at least one record has been written to it, there is no way to guarantee that tombstone:d records gets compacted for partitions that no longer have any write-traffic. Is this a correct assessment? 
Mitch Henderson
max.compaction.lag introduced KIP-354: Add a Maximum Log Compaction Lag - Apache Kafka - Apache Software Foundation works on active segments
or you can force a segment roll with segment.ms. Be careful with this as disk IO is expensive and having too many segments can cause an over use of file descriptors
another small note to those that read this later, None of this is done in a deterministic time. It will happen SOMETIME after the segment.ms or max.compaction.lag has been hit
Jonathan Olsson 3 days ago
Does max.compaction.lag behave differently from segment.ms? Becuase segment.ms does not seem to be enough to enforce compaction, only that a new segment gets created. The new segment doesn’t appear to become active until a record has been written to it.
Jonathan Olsson 3 days ago
Gah, I missed the works on active segments line.
Jonathan Olsson 3 days ago
Ok, so max.compaction.lag will force a new segment to become active depending on how old the first record in the segment (regardless of produced records)?
Mitch Henderson 3 days ago
Yes.
Mitch Henderson 3 days ago
the behavior is described in the linked kip
Jonathan Olsson 3 days ago
Indeed. However, It’s not 100% clear from the KIP if the force roll caused by max.compaction.lag is materially different than the one caused by segment.ms. I will give this a go. Thanks!
Mitch Henderson 3 days ago
They’re the same behavior
Jonathan Olsson 3 days ago
Sorry to be a bother, but as I have alluded to a bit, this does not solve the problem I am referring to. max.compaction.lag.ms does indeed also force a log roll, however, the new segment is still inactive until a record is written to it and so won’t actually compact the old segment.
Consider this topic:
kafka-topics.sh --zookeeper zookeeper:2181 \
--create --topic bar \
--partitions 1 \
--replication-factor 1 \
--config cleanup.policy=compact,delete \
--config max.compaction.lag.ms=10000 \
--config min.cleanable.dirty.ratio=0.000001 \
--config retention.bytes=-1 \
--config retention.ms=-1
with the current content:
$ kcat -b kafka:9092 -t bar -C -K: -o beginning -e
1:Hello
2:World
When adding a tombstone:
$ echo "1:" | kcat -b kafka:9092 -t bar -P -K: -Z
This is what you would see, regardless how long you waited:
kcat -b kafka:9092 -t bar -C -K: -o beginning -e
1:Hello
2:World
1:
It is not until you publish a new record to the new segment that the 1 gets compacted:
$ echo "3:doit" | kcat -b kafka:9092 -t bar -P -K: -Z
# wait a while
$ kcat -b kafka:9092 -t bar -C -K: -o beginning -e
2:World
1:
3:doit
Mitch Henderson
Jonathan is correct, the segment roll doesn’t make the next segment ACTIVE, you’ll need to have some process writing to the partition.