We run Apache Kafka and we have several topics there and because of the amount of data we keep them only for 4 hours:
log.retention.hours=4
log.retention.check.interval.ms=300000
And it works as expected and thanks to the load the data does not stay that long after 4 hours so that is fine. But now for second time one topic just stopped doing the retention, the other five (and some bigger) are being deleted fine as can be seen on disk and in log, but this one specific just stopped doing any retention deletion on that particular topic. The only thing that I can see in log is that a lot of deletions were happening at the same time but still feels strange to cause just stop in the retention. Any ideas are welcome as last time we had to stop Kafka, manually delete all the files and start it again and let it figure out what was lost and how to continue and it means we are losing lot of logs going through Kafka normally. And here is snippet from the log with just that topic syslog being deleted for the last time with the lines before and after as well and just grepping this topic to make it shorter as it was also running some other deletes for other topics as well:
[2022-05-12 15:30:24,596] INFO [Log partition=syslog-7, dir=/data/kafka/data] Incrementing log start offset to 7342768449 (kafka.log.Log)
[2022-05-12 15:32:49,507] INFO [Log partition=syslog-0, dir=/data/kafka/data] Found deletable segments with base offsets [117533146039] due to retention time 14400000ms breach (kafka.log.Log)
[2022-05-12 15:32:49,507] INFO [Log partition=syslog-0, dir=/data/kafka/data] Scheduling log segment [baseOffset 117533146039, size 610913752] for deletion. (kafka.log.Log)
[2022-05-12 15:32:49,508] INFO [Log partition=syslog-0, dir=/data/kafka/data] Incrementing log start offset to 117535494639 (kafka.log.Log)
[2022-05-12 15:32:49,509] INFO [Log partition=syslog-9, dir=/data/kafka/data] Found deletable segments with base offsets [7339264694] due to retention time 14400000ms breach (kafka.log.Log)
[2022-05-12 15:32:49,509] INFO [Log partition=syslog-9, dir=/data/kafka/data] Scheduling log segment [baseOffset 7339264694, size 913148471] for deletion. (kafka.log.Log)
[2022-05-12 15:32:49,509] INFO [Log partition=syslog-9, dir=/data/kafka/data] Incrementing log start offset to 7342767511 (kafka.log.Log)
[2022-05-12 15:32:49,510] INFO [Log partition=syslog-2, dir=/data/kafka/data] Found deletable segments with base offsets [117535253776] due to retention time 14400000ms breach (kafka.log.Log)
[2022-05-12 15:32:49,510] INFO [Log partition=syslog-2, dir=/data/kafka/data] Scheduling log segment [baseOffset 117535253776, size 58699640] for deletion. (kafka.log.Log)
[2022-05-12 15:32:49,510] INFO [Log partition=syslog-2, dir=/data/kafka/data] Incrementing log start offset to 117535480847 (kafka.log.Log)
[2022-05-12 15:32:49,510] INFO [Log partition=syslog-3, dir=/data/kafka/data] Found deletable segments with base offsets [7462911255] due to retention time 14400000ms breach (kafka.log.Log)
[2022-05-12 15:32:49,510] INFO [Log partition=syslog-3, dir=/data/kafka/data] Scheduling log segment [baseOffset 7462911255, size 671165604] for deletion. (kafka.log.Log)
[2022-05-12 15:32:49,510] INFO [Log partition=syslog-3, dir=/data/kafka/data] Incrementing log start offset to 7465489528 (kafka.log.Log)
[2022-05-12 15:32:49,512] INFO [Log partition=syslog-11, dir=/data/kafka/data] Found deletable segments with base offsets [7341328180] due to retention time 14400000ms breach (kafka.log.Log)
[2022-05-12 15:32:49,512] INFO [Log partition=syslog-11, dir=/data/kafka/data] Scheduling log segment [baseOffset 7341328180, size 374034996] for deletion. (kafka.log.Log)
[2022-05-12 15:32:49,512] INFO [Log partition=syslog-11, dir=/data/kafka/data] Incrementing log start offset to 7342767027 (kafka.log.Log)
[2022-05-12 15:32:49,513] INFO [Log partition=syslog-12, dir=/data/kafka/data] Found deletable segments with base offsets [7339255632] due to retention time 14400000ms breach (kafka.log.Log)
[2022-05-12 15:32:49,513] INFO [Log partition=syslog-12, dir=/data/kafka/data] Scheduling log segment [baseOffset 7339255632, size 915537834] for deletion. (kafka.log.Log)
[2022-05-12 15:32:49,513] INFO [Log partition=syslog-12, dir=/data/kafka/data] Incrementing log start offset to 7342767514 (kafka.log.Log)
[2022-05-12 15:32:49,514] INFO [Log partition=syslog-5, dir=/data/kafka/data] Found deletable segments with base offsets [7341362593] due to retention time 14400000ms breach (kafka.log.Log)
[2022-05-12 15:32:49,514] INFO [Log partition=syslog-5, dir=/data/kafka/data] Scheduling log segment [baseOffset 7341362593, size 365599697] for deletion. (kafka.log.Log)
[2022-05-12 15:32:49,514] INFO [Log partition=syslog-5, dir=/data/kafka/data] Incrementing log start offset to 7342769250 (kafka.log.Log)
[2022-05-12 15:32:49,515] INFO [Log partition=syslog-13, dir=/data/kafka/data] Found deletable segments with base offsets [7340340358] due to retention time 14400000ms breach (kafka.log.Log)
[2022-05-12 15:32:49,515] INFO [Log partition=syslog-13, dir=/data/kafka/data] Scheduling log segment [baseOffset 7340340358, size 630202638] for deletion. (kafka.log.Log)
[2022-05-12 15:32:49,516] INFO [Log partition=syslog-6, dir=/data/kafka/data] Found deletable segments with base offsets [7339301315] due to retention time 14400000ms breach (kafka.log.Log)
[2022-05-12 15:32:49,516] INFO [Log partition=syslog-6, dir=/data/kafka/data] Scheduling log segment [baseOffset 7339301315, size 903744649] for deletion. (kafka.log.Log)
[2022-05-12 15:32:49,516] INFO [Log partition=syslog-6, dir=/data/kafka/data] Incrementing log start offset to 7342768603 (kafka.log.Log)
[2022-05-12 15:32:49,517] INFO [Log partition=syslog-7, dir=/data/kafka/data] Found deletable segments with base offsets [7339606433] due to retention time 14400000ms breach (kafka.log.Log)
[2022-05-12 15:32:49,517] INFO [Log partition=syslog-7, dir=/data/kafka/data] Scheduling log segment [baseOffset 7339606433, size 822575376] for deletion. (kafka.log.Log)
[2022-05-12 15:33:49,508] INFO [Log partition=syslog-0, dir=/data/kafka/data] Deleting segment 117533146039 (kafka.log.Log)
[2022-05-12 15:33:49,509] INFO [Log partition=syslog-9, dir=/data/kafka/data] Deleting segment 7339264694 (kafka.log.Log)
[2022-05-12 15:33:49,510] INFO [Log partition=syslog-2, dir=/data/kafka/data] Deleting segment 117535253776 (kafka.log.Log)
[2022-05-12 15:33:49,511] INFO [Log partition=syslog-3, dir=/data/kafka/data] Deleting segment 7462911255 (kafka.log.Log)
[2022-05-12 15:33:49,512] INFO [Log partition=syslog-11, dir=/data/kafka/data] Deleting segment 7341328180 (kafka.log.Log)
[2022-05-12 15:33:49,512] INFO Deleted log /data/kafka/data/syslog-2/00000000117535253776.log.deleted. (kafka.log.LogSegment)
[2022-05-12 15:33:49,512] INFO Deleted offset index /data/kafka/data/syslog-2/00000000117535253776.index.deleted. (kafka.log.LogSegment)
[2022-05-12 15:33:49,512] INFO Deleted time index /data/kafka/data/syslog-2/00000000117535253776.timeindex.deleted. (kafka.log.LogSegment)
[2022-05-12 15:33:49,513] INFO [Log partition=syslog-12, dir=/data/kafka/data] Deleting segment 7339255632 (kafka.log.Log)
[2022-05-12 15:33:49,514] INFO [Log partition=syslog-5, dir=/data/kafka/data] Deleting segment 7341362593 (kafka.log.Log)
[2022-05-12 15:33:49,520] INFO [Log partition=syslog-13, dir=/data/kafka/data] Deleting segment 7340340358 (kafka.log.Log)
[2022-05-12 15:33:49,526] INFO Deleted log /data/kafka/data/syslog-11/00000000007341328180.log.deleted. (kafka.log.LogSegment)
[2022-05-12 15:33:49,526] INFO Deleted offset index /data/kafka/data/syslog-11/00000000007341328180.index.deleted. (kafka.log.LogSegment)
[2022-05-12 15:33:49,526] INFO Deleted time index /data/kafka/data/syslog-11/00000000007341328180.timeindex.deleted. (kafka.log.LogSegment)
[2022-05-12 15:33:49,528] INFO Deleted log /data/kafka/data/syslog-5/00000000007341362593.log.deleted. (kafka.log.LogSegment)
[2022-05-12 15:33:49,528] INFO Deleted offset index /data/kafka/data/syslog-5/00000000007341362593.index.deleted. (kafka.log.LogSegment)
[2022-05-12 15:33:49,528] INFO Deleted time index /data/kafka/data/syslog-5/00000000007341362593.timeindex.deleted. (kafka.log.LogSegment)
[2022-05-12 15:33:49,528] INFO [Log partition=syslog-6, dir=/data/kafka/data] Deleting segment 7339301315 (kafka.log.Log)
[2022-05-12 15:33:49,542] INFO Deleted log /data/kafka/data/syslog-13/00000000007340340358.log.deleted. (kafka.log.LogSegment)
[2022-05-12 15:33:49,542] INFO Deleted offset index /data/kafka/data/syslog-13/00000000007340340358.index.deleted. (kafka.log.LogSegment)
[2022-05-12 15:33:49,542] INFO Deleted time index /data/kafka/data/syslog-13/00000000007340340358.timeindex.deleted. (kafka.log.LogSegment)
[2022-05-12 15:33:49,578] INFO [Log partition=syslog-7, dir=/data/kafka/data] Deleting segment 7339606433 (kafka.log.Log)
[2022-05-12 15:33:49,606] INFO Deleted log /data/kafka/data/syslog-7/00000000007339606433.log.deleted. (kafka.log.LogSegment)
[2022-05-12 15:33:49,606] INFO Deleted offset index /data/kafka/data/syslog-7/00000000007339606433.index.deleted. (kafka.log.LogSegment)
[2022-05-12 15:33:49,606] INFO Deleted time index /data/kafka/data/syslog-7/00000000007339606433.timeindex.deleted. (kafka.log.LogSegment)
[2022-05-12 15:33:49,807] INFO Deleted log /data/kafka/data/syslog-0/00000000117533146039.log.deleted. (kafka.log.LogSegment)
[2022-05-12 15:33:49,808] INFO Deleted offset index /data/kafka/data/syslog-0/00000000117533146039.index.deleted. (kafka.log.LogSegment)
[2022-05-12 15:33:49,808] INFO Deleted time index /data/kafka/data/syslog-0/00000000117533146039.timeindex.deleted. (kafka.log.LogSegment)
[2022-05-12 15:33:49,841] INFO Deleted log /data/kafka/data/syslog-3/00000000007462911255.log.deleted. (kafka.log.LogSegment)
[2022-05-12 15:33:49,841] INFO Deleted offset index /data/kafka/data/syslog-3/00000000007462911255.index.deleted. (kafka.log.LogSegment)
[2022-05-12 15:33:49,842] INFO Deleted time index /data/kafka/data/syslog-3/00000000007462911255.timeindex.deleted. (kafka.log.LogSegment)
[2022-05-12 15:33:49,935] INFO Deleted log /data/kafka/data/syslog-12/00000000007339255632.log.deleted. (kafka.log.LogSegment)
[2022-05-12 15:33:49,935] INFO Deleted offset index /data/kafka/data/syslog-12/00000000007339255632.index.deleted. (kafka.log.LogSegment)
[2022-05-12 15:33:49,936] INFO Deleted time index /data/kafka/data/syslog-12/00000000007339255632.timeindex.deleted. (kafka.log.LogSegment)
[2022-05-12 15:33:49,950] INFO Deleted log /data/kafka/data/syslog-9/00000000007339264694.log.deleted. (kafka.log.LogSegment)
[2022-05-12 15:33:49,950] INFO Deleted offset index /data/kafka/data/syslog-9/00000000007339264694.index.deleted. (kafka.log.LogSegment)
[2022-05-12 15:33:49,950] INFO Deleted time index /data/kafka/data/syslog-9/00000000007339264694.timeindex.deleted. (kafka.log.LogSegment)
[2022-05-12 15:33:49,958] INFO Deleted log /data/kafka/data/syslog-6/00000000007339301315.log.deleted. (kafka.log.LogSegment)
[2022-05-12 15:33:49,958] INFO Deleted offset index /data/kafka/data/syslog-6/00000000007339301315.index.deleted. (kafka.log.LogSegment)
[2022-05-12 15:33:49,958] INFO Deleted time index /data/kafka/data/syslog-6/00000000007339301315.timeindex.deleted. (kafka.log.LogSegment)
[2022-05-12 17:45:21,218] INFO [ProducerStateManager partition=syslog-2] Writing producer snapshot at offset 117543798666 (kafka.log.ProducerStateManager)
[2022-05-12 17:45:21,218] INFO [Log partition=syslog-2, dir=/data/kafka/data] Rolled new log segment at offset 117543798666 in 2 ms. (kafka.log.Log)