Low volume compact topic not deleting old messages

Hi,

We’re having some difficulties when it comes to topics with low volume of messages being sent to them and compaction.

Example:

the topic has the following settings:

cleanup.policy compact
delete.retention.ms 60000
max.compaction.lag.ms 604800000
min.cleanable.dirty.ratio 0.5
min.compaction.lag.ms 0
segment.ms 14400000

the topic only has 49k messages and there are only a handfull of messages being produced daily.

we are trying to delete a message from the 10th of May by using tombstone messages but for some reason nothing is happening.

Can anyone help identify what may be wrong?

Difficult to say in general, however, there is many contribution factors.

Partitions are split into segments, and the newest (so-called active) segment in which new data lands is not subject to compaction. Only if new data is written, the active segment would get closed eventually (and a new active segment gets created), making the data available for compaction (there is couple of different configs like segment.size and segment.ms controlling this segment “rolling” process).

IIRC, segment.ms is only checked on-write, ie if you write a tombstone “now” into the active segment, the segment would still not roll even if it’s getting older than segment.ms because only the next write would trigger this check. To you might need to write another message (in you case 4hour later) to trigger the roll.

Also, the log cleaner thread only runs if the dirty-ration is reached, so with low throughput topic, this would build up only slowly. Reducing max.compaction.lag.ms could help to enforce a compaction run base on time, even if dirty ration is not reached yet.