I have set the property retention.ms to 36000000 which is 10 hours. The topic has active messages, i.e., messages are being produced and consumed. I realized that I don’t need the data for 10 hours now. I need it only for 2 hours.
What would happen if I change the property retention.ms to 7200000 (2 hours)? What will happen to the existing messages?
Welcome @ashwinbhaskar
As you may already know… topics are broken up into partitions, and partitions are broken up into segments. Segments are the actual data files on the file system that hold the topic data. Records are only removed when the log segment is deleted, and a log segment can only be deleted when it’s closed. There are other configurations that determine the size of segments (see configurations starting with segment). So, the retention.ms configuration defines the minimum time that a topic log segment will be retained but other configuration values will determine when a log segment is closed. If you change the retention.ms configuration with data in the topic all the same rules will apply the next time the log cleaning threads evaluate what segments can be cleaned, so reducing the value could result in segments being cleaned, or possibly not depending on the current segment, segment size, etc…
Viewing the retention.ms value as a “Service Level Agreement” for a topic is useful, meaning, this is a minimum time your consumer has to consume the message before it could be removed by the Kafka brokers.
very nice explanation. Thanks!
This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.