Does confluent kafka (cp-kafka) support Log4j RollingFileAppender with Time/Size Based Triggering Policy & Rollover Strategy? Doesn't seem to work

Ops Containers
1

I am using the Confluent Community Docker Image for Apache Kafka & trying to have the Kafka logs rolled over daily with a Time & Size Based Triggering Policy & Rollover Strategy.

As mentioned in the docs, I have extended the image & set it up with a Log4j configuration via properties.

However i see that only basic configuration of org.apache.log4j.RollingFileAppender without any daily rolling or policies appears to work (the commented out config settings in the property file below).

The examples on the docs also appears to use only the very old Log4j1.x org.apache.log4j.DailyRollingFileAppender which as we know (and as per 1.x docs) should not even be used. I am not even sure if Kafka packaged apache-log4j-extras along with log4j1.x earlier.

Below are the Dockerfile & log4j properties i am using. Any of you guys had any luck with it? Am i doing something incorrectly in the below?

Dockerfile:

FROM confluentinc/cp-kafka:7.0.2
COPY log4j.properties.template /etc/confluent/docker/log4j.properties.template

log4j properties:

log4j.rootLogger={{ env["KAFKA_LOG4J_ROOT_LOGLEVEL"] | default('INFO') }}, stdout, rollingFile

log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=[%d] %p %m (%c)%n

#log4j.appender.rollingFile=org.apache.log4j.RollingFileAppender
#log4j.appender.rollingFile.File=/var/log/kafka/kafka-server.log
#log4j.appender.rollingFile.filePattern=/var/log/kafka/kafka-server-%d{yyyy-MM-dd}-%i.log
#log4j.appender.rollingFile.layout=org.apache.log4j.PatternLayout
#log4j.appender.rollingFile.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %m%n
#log4j.appender.rollingFile.MaxFileSize=5KB
#log4j.appender.rollingFile.MaxBackupIndex=15
#log4j.appender.rollingFile.append=true

log4j.appender.rolling.type=RollingFile
log4j.appender.rolling.name=rollingFile
# Below is the default..just making explicit.
log4j.appender.rolling.append=true
log4j.appender.rolling.fileName=/var/log/kafka/kafka-server1.log
log4j.appender.rolling.filePattern=/var/log/kafka/kafka-server1-%d{yyyy-MM-dd}-%i.log
log4j.appender.rolling.layout.type=PatternLayout
log4j.appender.rolling.layout.pattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %m%n
log4j.appender.rolling.policies.type=Policies
log4j.appender.rolling.policies.time.type=TimeBasedTriggeringPolicy
# Below is the default..just making explicit.
log4j.appender.rolling.policies.time.interval=1
log4j.appender.rolling.policies.size.type=SizeBasedTriggeringPolicy
log4j.appender.rolling.policies.size.size=20MB
log4j.appender.rolling.strategy.type=DefaultRolloverStrategy
# Delete all files older than 15 days
log4j.appender.rolling.strategy.delete.type=Delete
log4j.appender.rolling.strategy.delete.basePath=/var/log/kafka/
log4j.appender.rolling.strategy.delete.ifLastModified.type=IfLastModified
log4j.appender.rolling.strategy.delete.ifLastModified.age=15d


{% set loggers = {
  'kafka': 'INFO',
  'kafka.network.RequestChannel
```: 'WARN',
  'kafka.producer.async.DefaultEventHandler': 'DEBUG',
  'kafka.request.logger': 'WARN',
  'kafka.controller': 'TRACE',
  'kafka.log.LogCleaner': 'INFO',
  'state.change.logger': 'TRACE',
  'kafka.authorizer.logger': 'WARN'
  } -%}


{% if env['KAFKA_LOG4J_LOGGERS'] %}
{% set loggers = parse_log4j_loggers(env['KAFKA_LOG4J_LOGGERS'], loggers) %}
{% endif %}

{% for logger,loglevel in loggers.items() %}
log4j.logger.{{logger}}={{loglevel}}
{% endfor %}

The official link from Confluent that still points to the usage of very old DailyRollingFileAppender. Perhaps Confluent still recommends using this i guess even though Log4j recommends against using it…

Thanks

2

Ideally, you’d have your container always log to stdout. Use other tools like Filebeat or Fluentd to capture Docker logs and roll into files.