Kafka logs are not deleted

Ops
1

Hello.
I want to ask what can be done so that the old logs are automatically deleted?
Log storage path:
/tmp/confluent.861681/kafka/logs

Unfortunately, the logs accumulate and are not deleted by themselves.

[root@inf-kfans-tt01 logs]# ls -lh
total 202M
-rw-rw-r--. 1 kafka kafka  5.8K Feb 17 09:53 controller.log
-rw-rw-r--. 1 kafka kafka  6.4K Feb 16 07:59 controller.log.2022-02-16-07
-rw-rw-r--. 1 kafka kafka  6.4K Feb 16 08:59 controller.log.2022-02-16-08
-rw-rw-r--. 1 kafka kafka  6.4K Feb 16 20:55 controller.log.2022-02-16-20
-rw-rw-r--. 1 kafka kafka  1.6K Feb 16 21:10 controller.log.2022-02-16-21
-rw-rw-r--. 1 kafka kafka   65K Feb 17 05:58 controller.log.2022-02-17-05
-rw-rw-r--. 1 kafka kafka  102K Feb 17 06:55 controller.log.2022-02-17-06
-rw-rw-r--. 1 kafka kafka  104K Feb 17 07:58 controller.log.2022-02-17-07
-rw-rw-r--. 1 kafka kafka  6.4K Feb 17 08:58 controller.log.2022-02-17-08
-rw-rw-r--. 1 kafka kafka     0 Feb 11 10:32 kafka-authorizer.log
-rw-rw-r--. 1 kafka kafka     0 Feb 11 10:32 kafka-request.log
-rw-rw-r--. 1 kafka kafka 1003K Feb 17 09:55 kafkaServer-gc.log
-rw-rw-r--. 1 kafka kafka   12M Feb 16 11:01 kafkaServer-gc.log.0
-rw-rw-r--. 1 kafka kafka   19K Feb 16 11:02 kafkaServer-gc.log.1
-rw-rw-r--. 1 kafka kafka  370K Feb 16 11:37 kafkaServer-gc.log.2
-rw-rw-r--. 1 kafka kafka   19K Feb 16 11:39 kafkaServer-gc.log.3
-rw-rw-r--. 1 kafka kafka  5.4M Feb 16 21:15 kafkaServer-gc.log.4
-rw-rw-r--. 1 kafka kafka  141K Feb 17 06:05 kafkaServer-gc.log.5
-rw-rw-r--. 1 kafka kafka   19K Feb 17 06:06 kafkaServer-gc.log.6
-rw-rw-r--. 1 kafka kafka  468K Feb 17 07:20 kafkaServer-gc.log.7
-rw-rw-r--. 1 kafka kafka   17K Feb 17 07:21 kafkaServer-gc.log.8
-rw-rw-r--. 1 kafka kafka  1.1K Feb 17 07:32 log-cleaner.log
-rw-rw-r--. 1 kafka kafka  1.1K Feb 14 17:34 log-cleaner.log.2022-02-14-17
-rw-rw-r--. 1 kafka kafka  1.1K Feb 15 11:24 log-cleaner.log.2022-02-15-11
-rw-rw-r--. 1 kafka kafka  6.6K Feb 15 13:55 log-cleaner.log.2022-02-15-13
-rw-rw-r--. 1 kafka kafka  6.6K Feb 15 15:07 log-cleaner.log.2022-02-15-15
-rw-rw-r--. 1 kafka kafka  2.2K Feb 16 11:40 log-cleaner.log.2022-02-16-11
-rw-rw-r--. 1 kafka kafka   172 Feb 17 05:53 log-cleaner.log.2022-02-17-05
-rw-rw-r--. 1 kafka kafka  1.1K Feb 17 06:30 log-cleaner.log.2022-02-17-06
-rw-rw-r--. 1 kafka kafka   58K Feb 17 09:55 metadata-service.log
-rw-rw-r--. 1 kafka kafka   62K Feb 15 10:59 metadata-service.log.2022-02-15-10
-rw-rw-r--. 1 kafka kafka   62K Feb 16 20:59 metadata-service.log.2022-02-16-20
-rw-rw-r--. 1 kafka kafka   16K Feb 16 21:15 metadata-service.log.2022-02-16-21
-rw-rw-r--. 1 kafka kafka  2.5K Feb 17 05:53 metadata-service.log.2022-02-17-05
-rw-rw-r--. 1 kafka kafka   27K Feb 17 06:59 metadata-service.log.2022-02-17-06
-rw-rw-r--. 1 kafka kafka   42K Feb 17 07:59 metadata-service.log.2022-02-17-07
-rw-rw-r--. 1 kafka kafka   61K Feb 17 08:59 metadata-service.log.2022-02-17-08
-rw-rw-r--. 1 kafka kafka  102K Feb 17 09:54 server.log
-rw-rw-r--. 1 kafka kafka   71M Feb 16 21:15 server.log.2022-02-16-21
-rw-rw-r--. 1 kafka kafka   16M Feb 17 05:59 server.log.2022-02-17-05
-rw-rw-r--. 1 kafka kafka   95M Feb 17 07:59 server.log.2022-02-17-07
-rw-rw-r--. 1 kafka kafka  179K Feb 17 08:57 server.log.2022-02-17-08
-rw-rw-r--. 1 kafka kafka   95K Feb 17 07:32 state-change.log
-rw-rw-r--. 1 kafka kafka   94K Feb 14 15:07 state-change.log.2022-02-14-15
-rw-rw-r--. 1 kafka kafka   94K Feb 14 17:34 state-change.log.2022-02-14-17
-rw-rw-r--. 1 kafka kafka   95K Feb 15 11:24 state-change.log.2022-02-15-11
-rw-rw-r--. 1 kafka kafka  189K Feb 16 11:40 state-change.log.2022-02-16-11
-rw-rw-r--. 1 kafka kafka   95K Feb 17 05:53 state-change.log.2022-02-17-05
-rw-rw-r--. 1 kafka kafka   95K Feb 17 06:30 state-change.log.2022-02-17-06

Here are the settings server.properties

log.dirs=/tmp/kafka-logs
log.cleaner.enable=true
log.retention.hours=2
log.cleanup.policy=compact,delete
segment.bytes=104857600
retention.ms=300000
log.segment.bytes=104857600
log.retention.check.interval.ms=300000

What did I forget to add?

2

Hi @Administor

seems your mixing two things up.
The logfiles your showing are log4j logfiles including errors, warnings, infos and so on.

The parameter control the retention of kafka topics/messages.

best,
michael

3

Super common question. You’re using the default log4j.properties or not configuring one at all.

Inside of the kafka or confluent distribution tarballs and RPMs there are example log4j.properties files that show how to setup size and time based log rotation. You can point kafka brokers at them with -Dlog4j.configuration=file:/path/to/log4j.properties as a JVM argument.

4

Thanks, Mitchell H. Indeed, log4j.properties should have been written in the configuration.

log4j.rootLogger=INFO, stdout, kafkaAppender

log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=[%d] %p %m (%c)%n

log4j.appender.kafkaAppender=org.apache.log4j.RollingFileAppender
log4j.appender.kafkaAppender.File=${kafka.logs.dir}/server.log
log4j.appender.kafkaAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.kafkaAppender.layout.ConversionPattern=[%d] %p %m (%c)%n
log4j.appender.kafkaAppender.MaxFileSize=500MB
log4j.appender.kafkaAppender.MaxBackupIndex=5

log4j.appender.stateChangeAppender=org.apache.log4j.RollingFileAppender
log4j.appender.stateChangeAppender.File=${kafka.logs.dir}/state-change.log
log4j.appender.stateChangeAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.stateChangeAppender.layout.ConversionPattern=[%d] %p %m (%c)%n
log4j.appender.stateChangeAppender.MaxFileSize=500MB
log4j.appender.stateChangeAppender.MaxBackupIndex=5

log4j.appender.requestAppender=org.apache.log4j.RollingFileAppender
log4j.appender.requestAppender.File=${kafka.logs.dir}/kafka-request.log
log4j.appender.requestAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.requestAppender.layout.ConversionPattern=[%d] %p %m (%c)%n
log4j.appender.requestAppender.MaxFileSize=500MB
log4j.appender.requestAppender.MaxBackupIndex=5

log4j.appender.cleanerAppender=org.apache.log4j.RollingFileAppender
log4j.appender.cleanerAppender.File=${kafka.logs.dir}/log-cleaner.log
log4j.appender.cleanerAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.cleanerAppender.layout.ConversionPattern=[%d] %p %m (%c)%n
log4j.appender.cleanerAppender.MaxFileSize=500MB
log4j.appender.cleanerAppender.MaxBackupIndex=5

log4j.appender.controllerAppender=org.apache.log4j.RollingFileAppender
log4j.appender.controllerAppender.File=${kafka.logs.dir}/controller.log
log4j.appender.controllerAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.controllerAppender.layout.ConversionPattern=[%d] %p %m (%c)%n
log4j.appender.controllerAppender.MaxFileSize=500MB
log4j.appender.controllerAppender.MaxBackupIndex=5

log4j.appender.authorizerAppender=org.apache.log4j.RollingFileAppender
log4j.appender.authorizerAppender.File=${kafka.logs.dir}/kafka-authorizer.log
log4j.appender.authorizerAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.authorizerAppender.layout.ConversionPattern=[%d] %p %m (%c)%n
log4j.appender.authorizerAppender.MaxFileSize=500MB
log4j.appender.authorizerAppender.MaxBackupIndex=5