# Problem with writing Kafka Connect \`log4j\` logs to a Kafka Topic **URL:** \ **Category:** Ops\ **Created:** [4 May 2021 10:55 UTC](https://forum.confluent.io/t/problem-with-writing-kafka-connect-log4j-logs-to-a-kafka-topic/1659 "2021-05-04T10:55:53Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![whatsupbros](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/whatsupbros/32/239_2.png) [@whatsupbros](https://forum.confluent.io/u/whatsupbros)\ **Post date:** [4 May 2021 10:55 UTC](https://forum.confluent.io/t/problem-with-writing-kafka-connect-log4j-logs-to-a-kafka-topic/1659/1 "2021-05-04T10:55:53Z")
Hi people! I need to write Kafka Connect `log4j` logs to a Kafka Topic, to be able to access them more easily. There is this good blog post on `log4j` configuration for to write Kafka Connect logs to both `stdout` and a file: [https://forum.confluent.io/t/kafka-connect-change-log-level-and-write-log-to-file/](https://forum.confluent.io/t/kafka-connect-change-log-level-and-write-log-to-file/) Also, there is another `log4j` appender type, which is called `KafkaAppender` ([Log4j – Log4j 2 Appenders](https://logging.apache.org/log4j/2.x/manual/appenders.html#KafkaAppender)). But after I added its configuration to my `connect-log4j.properties`: ```bash log4j.rootLogger=INFO, stdout, connectAppender, connectKafkaAppender #log4j.rootLogger=INFO, stdout, connectAppender log4j.logger.org.apache.zookeeper=ERROR log4j.logger.org.reflections=ERROR # The `%X{connector.context}` parameter in the layout includes connector-specific and task-specific information # in the log message, where appropriate. This makes it easier to identify those log messages that apply to a # specific connector. Simply add this parameter to the log layout configuration below to include the contextual information. # #connect.log.pattern=[%d] %p %m (%c:%L)%n connect.log.pattern=[%d] %p %X{connector.context}%m (%c:%L)%n # Send the logs to the console. # log4j.appender.stdout=org.apache.log4j.ConsoleAppender log4j.appender.stdout.layout=org.apache.log4j.PatternLayout log4j.appender.stdout.layout.ConversionPattern=${connect.log.pattern} # Send the logs to a file, rolling the file at midnight local time. For example, the `File` option specifies the # location of the log files (e.g. ${kafka.logs.dir}/connect.log), and at midnight local time the file is closed # and copied in the same directory but with a filename that ends in the `DatePattern` option. # log4j.appender.connectAppender=org.apache.log4j.DailyRollingFileAppender log4j.appender.connectAppender.DatePattern='.'yyyy-MM-dd-HH log4j.appender.connectAppender.File=${kafka.logs.dir}/connect.log log4j.appender.connectAppender.layout=org.apache.log4j.PatternLayout log4j.appender.connectAppender.layout.ConversionPattern=${connect.log.pattern} # Send the logs to a kafka topic # there will be no key for the kafka records by default log4j.appender.connectKafkaAppender=org.apache.kafka.log4jappender.KafkaLog4jAppender log4j.appender.connectKafkaAppender.brokerList=localhost:9092 log4j.appender.connectKafkaAppender.topic=_connect_log log4j.appender.connectKafkaAppender.compressionType=none log4j.appender.connectKafkaAppender.ignoreExceptions=true log4j.appender.connectKafkaAppender.syncSend=false log4j.appender.connectKafkaAppender.layout=org.apache.log4j.PatternLayout log4j.appender.connectKafkaAppender.layout.ConversionPattern=${connect.log.pattern} ``` **My Kafka Connect Worker won’t start anymore.** The topic `_connect_log` exists in my Kafka Cluster, and it is empty. The only Worker’s output, which I see in my console is as following: ```bash [2021-05-04 10:25:00,641] INFO ProducerConfig values: acks = 1 batch.size = 16384 bootstrap.servers = [localhost:9092] buffer.memory = 33554432 client.dns.lookup = use_all_dns_ips client.id = producer-1 compression.type = none connections.max.idle.ms = 540000 delivery.timeout.ms = 120000 enable.idempotence = false interceptor.classes = [] internal.auto.downgrade.txn.commit = false key.serializer = class org.apache.kafka.common.serialization.ByteArraySerializer linger.ms = 0 max.block.ms = 60000 max.in.flight.requests.per.connection = 5 max.request.size = 1048576 metadata.max.age.ms = 300000 metadata.max.idle.ms = 300000 metric.reporters = [] metrics.num.samples = 2 metrics.recording.level = INFO metrics.sample.window.ms = 30000 partitioner.class = class org.apache.kafka.clients.producer.internals.DefaultPartitioner receive.buffer.bytes = 32768 reconnect.backoff.max.ms = 1000 reconnect.backoff.ms = 50 request.timeout.ms = 30000 retries = 2147483647 retry.backoff.ms = 100 sasl.client.callback.handler.class = null sasl.jaas.config = null sasl.kerberos.kinit.cmd = /usr/bin/kinit sasl.kerberos.min.time.before.relogin = 60000 sasl.kerberos.service.name = null sasl.kerberos.ticket.renew.jitter = 0.05 sasl.kerberos.ticket.renew.window.factor = 0.8 sasl.login.callback.handler.class = null sasl.login.class = null sasl.login.refresh.buffer.seconds = 300 sasl.login.refresh.min.period.seconds = 60 sasl.login.refresh.window.factor = 0.8 sasl.login.refresh.window.jitter = 0.05 sasl.mechanism = GSSAPI security.protocol = PLAINTEXT security.providers = null send.buffer.bytes = 131072 socket.connection.setup.timeout.max.ms = 127000 socket.connection.setup.timeout.ms = 10000 ssl.cipher.suites = null ssl.enabled.protocols = [TLSv1.2, TLSv1.3] ssl.endpoint.identification.algorithm = https ssl.engine.factory.class = null ssl.key.password = null ssl.keymanager.algorithm = SunX509 ssl.keystore.certificate.chain = null ssl.keystore.key = null ssl.keystore.location = null ssl.keystore.password = null ssl.keystore.type = JKS ssl.protocol = TLSv1.3 ssl.provider = null ssl.secure.random.implementation = null ssl.trustmanager.algorithm = PKIX ssl.truststore.certificates = null ssl.truststore.location = null ssl.truststore.password = null ssl.truststore.type = JKS transaction.timeout.ms = 60000 transactional.id = null value.serializer = class org.apache.kafka.common.serialization.ByteArraySerializer (org.apache.kafka.clients.producer.ProducerConfig:0) ``` Usually, there is much more output on Worker’s startup. And if I remove the `connectKafkaAppender` related configuration from my `connect-log4j.properties`, the Worker starts normally and serves requests. Is there anything I am missing or doing wrong? **May the 4th be with you!** 💫 ---
**Author:** ![mmuehlbeyer](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/mmuehlbeyer/32/1088_2.png) [@mmuehlbeyer](https://forum.confluent.io/u/mmuehlbeyer)\ **Post date:** [4 May 2021 11:22 UTC](https://forum.confluent.io/t/problem-with-writing-kafka-connect-log4j-logs-to-a-kafka-topic/1659/2 "2021-05-04T11:22:37Z")
Hi, did you try with `log4j.rootLogger=INFO, stdout, connectAppender` _edit_: sorry of course you’ve tried 😉 just recognized your last sentence. Regards, Michael ---
**Author:** ![mmuehlbeyer](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/mmuehlbeyer/32/1088_2.png) [@mmuehlbeyer](https://forum.confluent.io/u/mmuehlbeyer)\ **Post date:** [4 May 2021 12:11 UTC](https://forum.confluent.io/t/problem-with-writing-kafka-connect-log4j-logs-to-a-kafka-topic/1659/3 "2021-05-04T12:11:06Z")
next try 😉 did you try with settings provided here: > perhaps also the “underscore topic” leads to an error? ---
**Author:** ![whatsupbros](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/whatsupbros/32/239_2.png) [@whatsupbros](https://forum.confluent.io/u/whatsupbros)\ **Post date:** [4 May 2021 12:23 UTC](https://forum.confluent.io/t/problem-with-writing-kafka-connect-log4j-logs-to-a-kafka-topic/1659/4 "2021-05-04T12:23:01Z")
hi @mmuehlbeyer! in fact, I already tried different topic names including `connect.log` and `connectlog`, and this didn’t change much for me. Do I understand it right that the main difference in the config provided by you is that `log4j.appender.kafka_appender.layout=io.confluent.common.logging.log4j.StructuredJsonLayout` is used there? Or should I take the config 1:1 and give it a try to see if the worker starts with it? ---
**Author:** ![mmuehlbeyer](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/mmuehlbeyer/32/1088_2.png) [@mmuehlbeyer](https://forum.confluent.io/u/mmuehlbeyer)\ **Post date:** [4 May 2021 12:33 UTC](https://forum.confluent.io/t/problem-with-writing-kafka-connect-log4j-logs-to-a-kafka-topic/1659/5 "2021-05-04T12:33:07Z")
hi @whatsupbros ok I see. > [@whatsupbros](#): > > Do I understand it right that the main difference in the config provided by you is that `log4j.appender.kafka_appender.layout=io.confluent.common.logging.log4j.StructuredJsonLayout` is used there? yes you’re right, I would go for both 1:1 and only changing the `log4j.appender.kafka_appender.layout` ---
**Author:** ![whatsupbros](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/whatsupbros/32/239_2.png) [@whatsupbros](https://forum.confluent.io/u/whatsupbros)\ **Post date:** [4 May 2021 13:52 UTC](https://forum.confluent.io/t/problem-with-writing-kafka-connect-log4j-logs-to-a-kafka-topic/1659/6 "2021-05-04T13:52:32Z")
Okay, I tried it to change the appender layout, and now have this error after I start the Worker: ```auto log4j:ERROR Could not instantiate class [io.confluent.common.logging.log4j.StructuredJsonLayout]. [15:47:59] java.lang.ClassNotFoundException: io.confluent.common.logging.log4j.StructuredJsonLayout at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:602) at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522) at java.base/java.lang.Class.forName0(Native Method) at java.base/java.lang.Class.forName(Class.java:340) at org.apache.log4j.helpers.Loader.loadClass(Loader.java:198) at org.apache.log4j.helpers.OptionConverter.instantiateByClassName(OptionConverter.java:327) at org.apache.log4j.helpers.OptionConverter.instantiateByKey(OptionConverter.java:124) at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:797) at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768) at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:580) at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526) at org.apache.log4j.LogManager.(LogManager.java:127) at org.slf4j.impl.Log4jLoggerFactory.(Log4jLoggerFactory.java:66) at org.slf4j.impl.StaticLoggerBinder.(StaticLoggerBinder.java:72) at org.slf4j.impl.StaticLoggerBinder.(StaticLoggerBinder.java:45) at org.slf4j.LoggerFactory.bind(LoggerFactory.java:150) at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:124) at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:417) at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:362) at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:388) at org.apache.kafka.connect.cli.ConnectDistributed.(ConnectDistributed.java:57) ``` And the worker output is the same - only the producer config is there. Using the configuration 1:1 is unfortunately not an option for me, because it seems that this is the `log4j` config for `ksqlDB` and I simply don’t have some variables used in the file set in my environment. However, I do not see anything in the configuration, which could fix my issue, since it is pretty much the same in the Kafka Connect part. Only `log4j.appender.kafka_appender.SyncSend` is set to `true`, but this is the default, which I also tried. ---
**Author:** ![mmuehlbeyer](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/mmuehlbeyer/32/1088_2.png) [@mmuehlbeyer](https://forum.confluent.io/u/mmuehlbeyer)\ **Post date:** [4 May 2021 14:00 UTC](https://forum.confluent.io/t/problem-with-writing-kafka-connect-log4j-logs-to-a-kafka-topic/1659/7 "2021-05-04T14:00:54Z")
hmm class is in classpath? perhaps one thing worth to try > **[How to handle deserialization errors using ksqlDB](https://developer.confluent.io/tutorials/handling-deserialization-errors/ksql.html)** > > In this tutorial, learn how to handle deserialization errors in a stream of events using ksqlDB, with step-by-step instructions and examples. seems the configuration is similar as the one you’ve tried perhaps you can get correct classpath and so on from the tutorial. nevertheless I’ll check the settings later in my local lab setup ---
**Author:** ![whatsupbros](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/whatsupbros/32/239_2.png) [@whatsupbros](https://forum.confluent.io/u/whatsupbros)\ **Post date:** [4 May 2021 14:07 UTC](https://forum.confluent.io/t/problem-with-writing-kafka-connect-log4j-logs-to-a-kafka-topic/1659/8 "2021-05-04T14:07:12Z")
I am not a Java Developer, and to be honest, I cannot tell if the class is in classpath or not. But what I can say is that in my case it is not `ksqlDB`, it is Kafka Connect Worker, which I am trying to start with such configuration that it writes logs of self not only to a file, but also a Kafka Topic. This configuration, which I provided in the starter message, is actually result of my own research on `log4j`, because I never did it before. So probably it can be done even easier that I am doing it there. The ultimate goal is to have the logs inside a topic. ---
**Author:** ![mmuehlbeyer](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/mmuehlbeyer/32/1088_2.png) [@mmuehlbeyer](https://forum.confluent.io/u/mmuehlbeyer)\ **Post date:** [4 May 2021 14:16 UTC](https://forum.confluent.io/t/problem-with-writing-kafka-connect-log4j-logs-to-a-kafka-topic/1659/9 "2021-05-04T14:16:45Z")
> [@whatsupbros](#): > > I am not a Java Developer to be honest, and I cannot tell if the class is in classpath or not. me neither 😉 > [@whatsupbros](#): > > But what I can say is that in my case it is not `ksqlDB` , it is Kafka Connect Worker, which I am trying to start with such configuration that it writes logs of self not only to a file, but also a Kafka Topic. understand, thought it might helpful are you running vanilla Kafka or Confluent Kafka? might the cause for the class not found > [@whatsupbros](#): > > The ultimate goal is to have the logs inside a topic. I see should be possible I guess but also need to test it by myself ---
**Author:** ![whatsupbros](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/whatsupbros/32/239_2.png) [@whatsupbros](https://forum.confluent.io/u/whatsupbros)\ **Post date:** [5 May 2021 08:23 UTC](https://forum.confluent.io/t/problem-with-writing-kafka-connect-log4j-logs-to-a-kafka-topic/1659/10 "2021-05-05T08:23:19Z")
> [@mmuehlbeyer](#): > > are you running vanilla Kafka or Confluent Kafka? I am on Confluent Platform Community in fact. But the class is still not found when I try using it. But I don’t think that the reason is in the log message layout, to be honest. Because even when the class is not found, the behaviour of the worker is the same (it hangs with the only message with the provider properties in the output). I guess there is something about the connection to the Kafka Brokers. But I cannot understand what exactly is wrong with it (and it is local installation, so there is no even network traversing and stuff). ---
**Author:** ![mmuehlbeyer](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/mmuehlbeyer/32/1088_2.png) [@mmuehlbeyer](https://forum.confluent.io/u/mmuehlbeyer)\ **Post date:** [5 May 2021 08:44 UTC](https://forum.confluent.io/t/problem-with-writing-kafka-connect-log4j-logs-to-a-kafka-topic/1659/11 "2021-05-05T08:44:04Z")
hmm I see I’ve played also around with my local setup but wasn’t able to get logs into a topic. one workaround not the best but maybe worth to think about: why not ingesting the logfiles with a file source connector with kafka connect ;)? I know if connect goes down the logs will not be loaded to kafka but perhaps it’s a proper workaround for your usecase ---
**Author:** ![whatsupbros](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/whatsupbros/32/239_2.png) [@whatsupbros](https://forum.confluent.io/u/whatsupbros)\ **Post date:** [5 May 2021 10:03 UTC](https://forum.confluent.io/t/problem-with-writing-kafka-connect-log4j-logs-to-a-kafka-topic/1659/12 "2021-05-05T10:03:47Z")
> [@mmuehlbeyer](#): > > why not ingesting the logfiles with a file source connector with kafka connect ;)? I also have this option in mind, but I’d rather use it as a last resort. Because you know, this is not a good idea to write a service log using functionality of the service itself, it looks fishy. That is why I was trying to use built in `log4j` capabilities for that, which seems to be a better option. But thank you anyway for trying to be helpful! ---
**Author:** ![mmuehlbeyer](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/mmuehlbeyer/32/1088_2.png) [@mmuehlbeyer](https://forum.confluent.io/u/mmuehlbeyer)\ **Post date:** [5 May 2021 10:54 UTC](https://forum.confluent.io/t/problem-with-writing-kafka-connect-log4j-logs-to-a-kafka-topic/1659/13 "2021-05-05T10:54:10Z")
was able to get it up and running 🙂 config below **important:** you need the following jars in your `$CONFLUENT_INSTALL_DIR/share/java/confluent-common` - common-logging-6.1.1.jar - confluent-log4j-extensions-6.1.1.jar get it from: [https://mvnrepository.com/artifact/io.confluent/common-logging](https://mvnrepository.com/artifact/io.confluent/common-logging) [https://mvnrepository.com/artifact/io.confluent/confluent-log4j-extensions](https://mvnrepository.com/artifact/io.confluent/confluent-log4j-extensions) _ **config starts here** _ ``` log4j.rootLogger=DEBUG, stdout, connectAppender # Send the logs to the console. # log4j.appender.stdout=org.apache.log4j.ConsoleAppender log4j.appender.stdout.layout=org.apache.log4j.PatternLayout # Send the logs to a file, rolling the file at midnight local time. For example, the `File` option specifies the # location of the log files (e.g. ${kafka.logs.dir}/connect.log), and at midnight local time the file is closed # and copied in the same directory but with a filename that ends in the `DatePattern` option. # log4j.appender.connectAppender=org.apache.log4j.DailyRollingFileAppender log4j.appender.connectAppender.DatePattern='.'yyyy-MM-dd-HH log4j.appender.connectAppender.File=${kafka.logs.dir}/connect.log log4j.appender.connectAppender.layout=org.apache.log4j.PatternLayout # The `%X{connector.context}` parameter in the layout includes connector-specific and task-specific information # in the log message, where appropriate. This makes it easier to identify those log messages that apply to a # specific connector. Simply add this parameter to the log layout configuration below to include the contextual information. # connect.log.pattern=[%d] %p %m (%c:%L)%n #connect.log.pattern=[%d] %p %X{connector.context}%m (%c:%L)%n log4j.appender.stdout.layout.ConversionPattern=${connect.log.pattern} log4j.appender.connectAppender.layout.ConversionPattern=${connect.log.pattern} og4j.appender.kafka_appender=org.apache.kafka.log4jappender.KafkaLog4jAppender log4j.appender.kafka_appender.layout=io.confluent.common.logging.log4j.StructuredJsonLayout log4j.appender.kafka_appender.BrokerList=localhost:9092 log4j.appender.kafka_appender.Topic=k-connect-log log4j.logger.processing=INFO, kafka_appender log4j.logger.org.apache.kafka.connect=INFO, kafka_appender log4j.additivity.org.apache.kafka.connect=false log4j.logger.org.apache.zookeeper=ERROR log4j.logger.org.reflections=ERROR ``` ---
**Author:** ![rmoff](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/rmoff/32/3_2.png) [@rmoff](https://forum.confluent.io/u/rmoff)\ **Post date:** [5 May 2021 13:20 UTC](https://forum.confluent.io/t/problem-with-writing-kafka-connect-log4j-logs-to-a-kafka-topic/1659/14 "2021-05-05T13:20:58Z")
Wow, nice job! This sounds really useful. ---
**Author:** ![whatsupbros](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/whatsupbros/32/239_2.png) [@whatsupbros](https://forum.confluent.io/u/whatsupbros)\ **Post date:** [6 May 2021 07:25 UTC](https://forum.confluent.io/t/problem-with-writing-kafka-connect-log4j-logs-to-a-kafka-topic/1659/15 "2021-05-06T07:25:01Z")
Thx! Will be able to check it tomorrow, so will let you know as soon as I have any result here. ---
**Author:** ![whatsupbros](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/whatsupbros/32/239_2.png) [@whatsupbros](https://forum.confluent.io/u/whatsupbros)\ **Post date:** [7 May 2021 11:51 UTC](https://forum.confluent.io/t/problem-with-writing-kafka-connect-log4j-logs-to-a-kafka-topic/1659/16 "2021-05-07T11:51:20Z")
Hi @mmuehlbeyer! So, I finally had a chance to check your proposal, and with some help of it, I found out the actual reason of the issue. > [@mmuehlbeyer](#): > > **important:** > you need the following jars in your `$CONFLUENT_INSTALL_DIR/share/java/confluent-common` > > - common-logging-6.1.1.jar > - confluent-log4j-extensions-6.1.1.jar > > get it from: > [https://mvnrepository.com/artifact/io.confluent/common-logging](https://mvnrepository.com/artifact/io.confluent/common-logging) > [https://mvnrepository.com/artifact/io.confluent/confluent-log4j-extensions](https://mvnrepository.com/artifact/io.confluent/confluent-log4j-extensions) First of all, about these libraries you are mentioning - for the reason unknown to me, I don’t have the mentioned libraries in my `$CONFLUENT_BASE/share/java/confluent-common` for my installation. I also checked it for the most current version of Confluent Platform Community and Enterprise (because the current is 6.1.1 and I am now still on 6.1.0) - and there no such libraries there as well under `confluent-common` directory. However, in both my installation and the most current one, I have these libraries in `$CONFLUENT_BASE/share/java/ksqldb` directory. I don’t know if it was intended to be like this, but this is the reality we have here. I don’t know though how all these libraries are loaded from `$CONFLUENT_BASE/share/java` by the Confluent Platform processes, probably @rmoff could clarify on this a little more. Nevertheless, I was quite sure that unless I use the `io.confluent.common.logging.log4j.StructuredJsonLayout` as the log message layout, I shouldn’t need these libraries, and that is why I looked for other differences in yours and mine configs. > [@mmuehlbeyer](#): > > config starts here So, I noticed, that you are using your `kafka_appender` only for particular loggers, and you didn’t add it as a root logger. And when I adjusted my config the same way, so that only Kafka Connect logger logs are written to my Kafka Topic, **it all started to work!** Everything else about the config stayed the same - the topic name starting with `_` symbol and having dash `-` symbol in the name. The bottomline is, that **you probably cannot use `KafkaLog4jAppender` for the root logger**. My final and working `connect-log4j.properties` file looks like this now: ```auto log4j.rootLogger=INFO, stdout, connectAppender log4j.logger.org.apache.kafka.connect=INFO, connectKafkaAppender log4j.logger.org.apache.zookeeper=ERROR log4j.logger.org.reflections=ERROR # The `%X{connector.context}` parameter in the layout includes connector-specific and task-specific information # in the log message, where appropriate. This makes it easier to identify those log messages that apply to a # specific connector. Simply add this parameter to the log layout configuration below to include the contextual information. # #connect.log.pattern=[%d] %p %m (%c:%L)%n connect.log.pattern=[%d] %p %X{connector.context}%m (%c:%L)%n # Send the logs to the console. # log4j.appender.stdout=org.apache.log4j.ConsoleAppender log4j.appender.stdout.layout=org.apache.log4j.PatternLayout log4j.appender.stdout.layout.ConversionPattern=${connect.log.pattern} # Send the logs to a file, rolling the file at midnight local time. For example, the `File` option specifies the # location of the log files (e.g. ${kafka.logs.dir}/connect.log), and at midnight local time the file is closed # and copied in the same directory but with a filename that ends in the `DatePattern` option. # log4j.appender.connectAppender=org.apache.log4j.DailyRollingFileAppender log4j.appender.connectAppender.DatePattern='.'yyyy-MM-dd-HH log4j.appender.connectAppender.File=${kafka.logs.dir}/connect.log log4j.appender.connectAppender.layout=org.apache.log4j.PatternLayout log4j.appender.connectAppender.layout.ConversionPattern=${connect.log.pattern} # Send the logs to a kafka topic # there will be no key for the kafka records by default log4j.appender.connectKafkaAppender=org.apache.kafka.log4jappender.KafkaLog4jAppender log4j.appender.connectKafkaAppender.brokerList=localhost:9092 log4j.appender.connectKafkaAppender.topic=_connect-log log4j.appender.connectKafkaAppender.compressionType=none log4j.appender.connectKafkaAppender.ignoreExceptions=true log4j.appender.connectKafkaAppender.syncSend=false log4j.appender.connectKafkaAppender.layout=org.apache.log4j.PatternLayout log4j.appender.connectKafkaAppender.layout.ConversionPattern=${connect.log.pattern} ``` ---
**Author:** ![whatsupbros](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/whatsupbros/32/239_2.png) [@whatsupbros](https://forum.confluent.io/u/whatsupbros)\ **Post date:** [7 May 2021 13:40 UTC](https://forum.confluent.io/t/problem-with-writing-kafka-connect-log4j-logs-to-a-kafka-topic/1659/17 "2021-05-07T13:40:25Z")
Regarding usage of `io.confluent.common.logging.log4j.StructuredJsonLayout` for the message layout - I tried this too. And you were 100% correct, that the two mentioned by you libraries are required for it: - common-logging-X.X.X.jar - confluent-log4j-extensions-X.X.X.jar Where X.X.X is the version of your Confluent Platform installation. And yes, these libraries are missing byt default in `$CONFLUENT_BASE/share/java/confluent-common` directory, however they exist in `$CONFLUENT_BASE/share/java/ksqldb`. I just copied them from there to `confluent-common` directory, restarted my Kafka Connect Workers, and they started to see the needed libraries, and they were loaded without issues. So it all started to work, and as a result, the log messages were formatted like this in my topic: ```auto $ ./bin/kafka-console-consumer \ > --bootstrap-server localhost:9092 \ > --topic _connect-log \ > --key-deserializer "org.apache.kafka.common.serialization.StringDeserializer" \ > --value-deserializer "org.apache.kafka.common.serialization.StringDeserializer" \ > --partition 0 --offset 250 --max-messages 5 {"level":"INFO","logger":"org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader","time":1620392108294,"message":Added alias 'TopicNameMatches' to plugin 'org.apache.kafka.connect.transforms.predicates.TopicNameMatches'} {"level":"INFO","logger":"org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader","time":1620392108296,"message":Added alias 'BasicAuthSecurityRestExtension' to plugin 'org.apache.kafka.connect.rest.basic.auth.extension.BasicAuthSecurityRestExtension'} {"level":"INFO","logger":"org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader","time":1620392108296,"message":Added aliases 'AllConnectorClientConfigOverridePolicy' and 'All' to plugin 'org.apache.kafka.connect.connector.policy.AllConnectorClientConfigOverridePolicy'} {"level":"INFO","logger":"org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader","time":1620392108298,"message":Added aliases 'NoneConnectorClientConfigOverridePolicy' and 'None' to plugin 'org.apache.kafka.connect.connector.policy.NoneConnectorClientConfigOverridePolicy'} {"level":"INFO","logger":"org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader","time":1620392108299,"message":Added aliases 'PrincipalConnectorClientConfigOverridePolicy' and 'Principal' to plugin 'org.apache.kafka.connect.connector.policy.PrincipalConnectorClientConfigOverridePolicy'} Processed a total of 5 messages ``` **However, there is another problem with it - because the topic message values are sadly not valid JSONs.** The reason for this, is that the log message value itself (`message` attribute of the resulting JSON) is not properly quoted, and any special characters within the message value are not escaped. @rmoff could this be a bug with the log message layout formatter? ---
**Author:** ![whatsupbros](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/whatsupbros/32/239_2.png) [@whatsupbros](https://forum.confluent.io/u/whatsupbros)\ **Post date:** [7 May 2021 15:10 UTC](https://forum.confluent.io/t/problem-with-writing-kafka-connect-log4j-logs-to-a-kafka-topic/1659/18 "2021-05-07T15:10:50Z")
As the current workaround of the mentioned above JSON serialization problem with `io.confluent.common.logging.log4j.StructuredJsonLayout`, I used these layout properties instead: ```auto log4j.appender.connectKafkaAppender.layout=org.apache.log4j.PatternLayout #log4j.appender.connectKafkaAppender.layout.ConversionPattern=${connect.log.pattern} log4j.appender.connectKafkaAppender.layout.ConversionPattern={"level":"%p","timestamp":"%d{ISO8601}","logger":"%c","message":"%X{connector.context}%m"} ``` Such configuration gives us these log records in the `_connect-log` topic: ```auto $ ./bin/kafka-console-consumer \ > --bootstrap-server localhost:9092 \ > --topic _connect-log \ > --key-deserializer "org.apache.kafka.common.serialization.StringDeserializer" \ > --value-deserializer "org.apache.kafka.common.serialization.StringDeserializer" \ > --partition 0 --offset 195 --max-messages 5 {"level":"INFO","timestamp":"2021-05-07 14:58:01,267","logger":"org.apache.kafka.connect.runtime.distributed.WorkerCoordinator","message":"[Worker clientId=connect-1, groupId=connect-cluster] Rebalance started"} {"level":"INFO","timestamp":"2021-05-07 14:58:01,313","logger":"org.apache.kafka.connect.runtime.distributed.DistributedHerder","message":"[Worker clientId=connect-1, groupId=connect-cluster] Joined group at generation 1 with protocol version 2 and got assignment: Assignment{error=0, leader='connect-1-5e12c053-635c-4489-93ef-f6689b91cefb', leaderUrl='http://127.0.1.1:8083/', offset=-1, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0"} {"level":"INFO","timestamp":"2021-05-07 14:58:01,314","logger":"org.apache.kafka.connect.runtime.distributed.DistributedHerder","message":"[Worker clientId=connect-1, groupId=connect-cluster] Starting connectors and tasks using config offset -1"} {"level":"INFO","timestamp":"2021-05-07 14:58:01,315","logger":"org.apache.kafka.connect.runtime.distributed.DistributedHerder","message":"[Worker clientId=connect-1, groupId=connect-cluster] Finished starting connectors and tasks"} {"level":"INFO","timestamp":"2021-05-07 14:58:01,368","logger":"org.apache.kafka.connect.runtime.distributed.DistributedHerder","message":"[Worker clientId=connect-1, groupId=connect-cluster] Session key updated"} Processed a total of 5 messages ``` This is also not ideal, because the log message is not encoded. But unfortunately I didn’t find a way to do this easily with `log4j` version 1, which seems to be under the hood of Kafka logging capabilities currently. In `log4j2` though, it seems to be possible to use such a pattern, which should do the job: ```auto log4j.appender.connectKafkaAppender.layout.ConversionPattern={"level":"%level","timestamp":"%date{ISO8601_OFFSET_DATE_TIME_HHCMM}","logger":"%logger","message":"%encode{%X{connector.context}%message}{JSON}"} ``` I am not sure about this, but it looks like `log4j` is being replaced with `log4j2` in this [KIP](https://cwiki.apache.org/confluence/display/KAFKA/KIP-653%3A+Upgrade+log4j+to+log4j2) and the related [Jira](https://issues.apache.org/jira/browse/KAFKA-9366). Does anybody know for which version it is planned? ---
**Author:** ![whatsupbros](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/whatsupbros/32/239_2.png) [@whatsupbros](https://forum.confluent.io/u/whatsupbros)\ **Post date:** [11 May 2021 09:45 UTC](https://forum.confluent.io/t/problem-with-writing-kafka-connect-log4j-logs-to-a-kafka-topic/1659/19 "2021-05-11T09:45:29Z")
Okay, I found an alternative way to write JSON formatted log messages with `log4j` version 1, but these additional libraries must be put to `$CONFLUENT_BASE/share/java/confluent-common` directory for it to work: - [https://mvnrepository.com/artifact/net.logstash.log4j/jsonevent-layout](https://mvnrepository.com/artifact/net.logstash.log4j/jsonevent-layout) - [https://mvnrepository.com/artifact/commons-lang/commons-lang](https://mvnrepository.com/artifact/commons-lang/commons-lang) - [https://mvnrepository.com/artifact/net.minidev/json-smart](https://mvnrepository.com/artifact/net.minidev/json-smart) - [https://mvnrepository.com/artifact/net.minidev/accessors-smart](https://mvnrepository.com/artifact/net.minidev/accessors-smart) - [https://mvnrepository.com/artifact/org.ow2.asm/asm](https://mvnrepository.com/artifact/org.ow2.asm/asm) Then, the layout configuration of the appender should look as following: ```auto log4j.appender.connectKafkaAppender.layout=net.logstash.log4j.JSONEventLayoutV1 ``` As a result, the log messages are formatted like this: ```auto $ ./bin/kafka-console-consumer \ > --bootstrap-server localhost:9092 \ > --topic _connect-log \ > --key-deserializer "org.apache.kafka.common.serialization.StringDeserializer" \ > --value-deserializer "org.apache.kafka.common.serialization.StringDeserializer" \ > --partition 0 --offset 195 --max-messages 5 {"@timestamp":"2021-05-11T09:27:50.404Z","source_host":"DWPNB7S3T273","file":"WorkerCoordinator.java","method":"onJoinPrepare","level":"INFO","line_number":"225","thread_name":"DistributedHerder-connect-1-1","@version":1,"logger_name":"org.apache.kafka.connect.runtime.distributed.WorkerCoordinator","message":"[Worker clientId=connect-1, groupId=connect-cluster] Rebalance started","class":"org.apache.kafka.connect.runtime.distributed.WorkerCoordinator","mdc":{}} {"@timestamp":"2021-05-11T09:27:50.450Z","source_host":"DWPNB7S3T273","file":"DistributedHerder.java","method":"onAssigned","level":"INFO","line_number":"1689","thread_name":"DistributedHerder-connect-1-1","@version":1,"logger_name":"org.apache.kafka.connect.runtime.distributed.DistributedHerder","message":"[Worker clientId=connect-1, groupId=connect-cluster] Joined group at generation 1 with protocol version 2 and got assignment: Assignment{error=0, leader='connect-1-dc8be253-e282-4c95-838c-fecfe893556e', leaderUrl='http:\/\/127.0.1.1:8083\/', offset=-1, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0","class":"org.apache.kafka.connect.runtime.distributed.DistributedHerder$RebalanceListener","mdc":{}} {"@timestamp":"2021-05-11T09:27:50.452Z","source_host":"DWPNB7S3T273","file":"DistributedHerder.java","method":"startWork","level":"INFO","line_number":"1216","thread_name":"DistributedHerder-connect-1-1","@version":1,"logger_name":"org.apache.kafka.connect.runtime.distributed.DistributedHerder","message":"[Worker clientId=connect-1, groupId=connect-cluster] Starting connectors and tasks using config offset -1","class":"org.apache.kafka.connect.runtime.distributed.DistributedHerder","mdc":{}} {"@timestamp":"2021-05-11T09:27:50.453Z","source_host":"DWPNB7S3T273","file":"DistributedHerder.java","method":"startWork","level":"INFO","line_number":"1244","thread_name":"DistributedHerder-connect-1-1","@version":1,"logger_name":"org.apache.kafka.connect.runtime.distributed.DistributedHerder","message":"[Worker clientId=connect-1, groupId=connect-cluster] Finished starting connectors and tasks","class":"org.apache.kafka.connect.runtime.distributed.DistributedHerder","mdc":{}} {"@timestamp":"2021-05-11T09:27:50.517Z","source_host":"DWPNB7S3T273","file":"DistributedHerder.java","method":"onSessionKeyUpdate","level":"INFO","line_number":"1578","thread_name":"KafkaBasedLog Work Thread - connect-configs","@version":1,"logger_name":"org.apache.kafka.connect.runtime.distributed.DistributedHerder","message":"[Worker clientId=connect-1, groupId=connect-cluster] Session key updated","class":"org.apache.kafka.connect.runtime.distributed.DistributedHerder$ConfigUpdateListener","mdc":{}} Processed a total of 5 messages ``` This option generates proper JSONs, which should be possible to parse without issues. Just FYI @mmuehlbeyer! And thank you again for your assistance! Nevertheless, hopefully, we’ll have `log4j2` integrated with Kafka soon, which will give us much more opportunities of configuring the loggers. I.e. `JsonTemplateLayout` and other fancy things, which would be possible to configure even to have proper Kafka Connect records with schema within the messages. ---
**Author:** ![system](https://us1.discourse-cdn.com/flex019/uploads/confluentcommunity/original/1X/c49438c90c9df282e9996fdf6971be890c71b65a.svg) [@system](https://forum.confluent.io/u/system)\ **Post date:** [18 May 2021 09:46 UTC](https://forum.confluent.io/t/problem-with-writing-kafka-connect-log4j-logs-to-a-kafka-topic/1659/20 "2021-05-18T09:46:21Z")
This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.