Problem with writing Kafka Connect `log4j` logs to a Kafka Topic

Ops
16

Hi @mmuehlbeyer!

So, I finally had a chance to check your proposal, and with some help of it, I found out the actual reason of the issue.

First of all, about these libraries you are mentioning - for the reason unknown to me, I don’t have the mentioned libraries in my $CONFLUENT_BASE/share/java/confluent-common for my installation.

I also checked it for the most current version of Confluent Platform Community and Enterprise (because the current is 6.1.1 and I am now still on 6.1.0) - and there no such libraries there as well under confluent-common directory.

However, in both my installation and the most current one, I have these libraries in $CONFLUENT_BASE/share/java/ksqldb directory.

I don’t know if it was intended to be like this, but this is the reality we have here. I don’t know though how all these libraries are loaded from $CONFLUENT_BASE/share/java by the Confluent Platform processes, probably @rmoff could clarify on this a little more.

Nevertheless, I was quite sure that unless I use the io.confluent.common.logging.log4j.StructuredJsonLayout as the log message layout, I shouldn’t need these libraries, and that is why I looked for other differences in yours and mine configs.

So, I noticed, that you are using your kafka_appender only for particular loggers, and you didn’t add it as a root logger.

And when I adjusted my config the same way, so that only Kafka Connect logger logs are written to my Kafka Topic, it all started to work!

Everything else about the config stayed the same - the topic name starting with _ symbol and having dash - symbol in the name.

The bottomline is, that you probably cannot use KafkaLog4jAppender for the root logger.

My final and working connect-log4j.properties file looks like this now:

log4j.rootLogger=INFO, stdout, connectAppender

log4j.logger.org.apache.kafka.connect=INFO, connectKafkaAppender

log4j.logger.org.apache.zookeeper=ERROR
log4j.logger.org.reflections=ERROR

# The `%X{connector.context}` parameter in the layout includes connector-specific and task-specific information
# in the log message, where appropriate. This makes it easier to identify those log messages that apply to a
# specific connector. Simply add this parameter to the log layout configuration below to include the contextual information.
#
#connect.log.pattern=[%d] %p %m (%c:%L)%n
connect.log.pattern=[%d] %p %X{connector.context}%m (%c:%L)%n

# Send the logs to the console.
#
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=${connect.log.pattern}

# Send the logs to a file, rolling the file at midnight local time. For example, the `File` option specifies the
# location of the log files (e.g. ${kafka.logs.dir}/connect.log), and at midnight local time the file is closed
# and copied in the same directory but with a filename that ends in the `DatePattern` option.
#
log4j.appender.connectAppender=org.apache.log4j.DailyRollingFileAppender
log4j.appender.connectAppender.DatePattern='.'yyyy-MM-dd-HH
log4j.appender.connectAppender.File=${kafka.logs.dir}/connect.log
log4j.appender.connectAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.connectAppender.layout.ConversionPattern=${connect.log.pattern}

# Send the logs to a kafka topic
# there will be no key for the kafka records by default
log4j.appender.connectKafkaAppender=org.apache.kafka.log4jappender.KafkaLog4jAppender
log4j.appender.connectKafkaAppender.brokerList=localhost:9092
log4j.appender.connectKafkaAppender.topic=_connect-log
log4j.appender.connectKafkaAppender.compressionType=none
log4j.appender.connectKafkaAppender.ignoreExceptions=true
log4j.appender.connectKafkaAppender.syncSend=false
log4j.appender.connectKafkaAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.connectKafkaAppender.layout.ConversionPattern=${connect.log.pattern}
1 Like