Problem with writing Kafka Connect `log4j` logs to a Kafka Topic

Ops
13

was able to get it up and running :slight_smile:
config below

important:
you need the following jars in your $CONFLUENT_INSTALL_DIR/share/java/confluent-common

  • common-logging-6.1.1.jar
  • confluent-log4j-extensions-6.1.1.jar

get it from:
https://mvnrepository.com/artifact/io.confluent/common-logging
https://mvnrepository.com/artifact/io.confluent/confluent-log4j-extensions

config starts here

log4j.rootLogger=DEBUG, stdout, connectAppender

# Send the logs to the console.
#
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout

# Send the logs to a file, rolling the file at midnight local time. For example, the `File` option specifies the
# location of the log files (e.g. ${kafka.logs.dir}/connect.log), and at midnight local time the file is closed
# and copied in the same directory but with a filename that ends in the `DatePattern` option.
#
log4j.appender.connectAppender=org.apache.log4j.DailyRollingFileAppender
log4j.appender.connectAppender.DatePattern='.'yyyy-MM-dd-HH
log4j.appender.connectAppender.File=${kafka.logs.dir}/connect.log
log4j.appender.connectAppender.layout=org.apache.log4j.PatternLayout

# The `%X{connector.context}` parameter in the layout includes connector-specific and task-specific information
# in the log message, where appropriate. This makes it easier to identify those log messages that apply to a
# specific connector. Simply add this parameter to the log layout configuration below to include the contextual information.
#
connect.log.pattern=[%d] %p %m (%c:%L)%n
#connect.log.pattern=[%d] %p %X{connector.context}%m (%c:%L)%n

log4j.appender.stdout.layout.ConversionPattern=${connect.log.pattern}
log4j.appender.connectAppender.layout.ConversionPattern=${connect.log.pattern}

og4j.appender.kafka_appender=org.apache.kafka.log4jappender.KafkaLog4jAppender
log4j.appender.kafka_appender.layout=io.confluent.common.logging.log4j.StructuredJsonLayout
log4j.appender.kafka_appender.BrokerList=localhost:9092
log4j.appender.kafka_appender.Topic=k-connect-log
log4j.logger.processing=INFO, kafka_appender


log4j.logger.org.apache.kafka.connect=INFO, kafka_appender
log4j.additivity.org.apache.kafka.connect=false

log4j.logger.org.apache.zookeeper=ERROR
log4j.logger.org.reflections=ERROR
1 Like