# Problem with writing Kafka Connect \`log4j\` logs to a Kafka Topic

**URL:** <https://forum.confluent.io/t/problem-with-writing-kafka-connect-log4j-logs-to-a-kafka-topic/1659>\
**Category:** Ops\
**Created:** [4 May 2021 10:55 UTC](https://forum.confluent.io/t/problem-with-writing-kafka-connect-log4j-logs-to-a-kafka-topic/1659 "2021-05-04T10:55:53Z")\
**Posts on this page:** 1\
**Showing post:** 16

<div class="post-metadata">

**Author:** ![whatsupbros](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/whatsupbros/32/239_2.png) [@whatsupbros](https://forum.confluent.io/u/whatsupbros)\
**Post date:** [7 May 2021 11:51 UTC](https://forum.confluent.io/t/problem-with-writing-kafka-connect-log4j-logs-to-a-kafka-topic/1659/16 "2021-05-07T11:51:20Z")

</div>

Hi @mmuehlbeyer!

So, I finally had a chance to check your proposal, and with some help of it, I found out the actual reason of the issue.

> [@mmuehlbeyer](#):
>
> **important:**  
> you need the following jars in your `$CONFLUENT_INSTALL_DIR/share/java/confluent-common`
> 
> - common-logging-6.1.1.jar
> - confluent-log4j-extensions-6.1.1.jar
> 
> get it from:  
> [https://mvnrepository.com/artifact/io.confluent/common-logging](https://mvnrepository.com/artifact/io.confluent/common-logging)  
> [https://mvnrepository.com/artifact/io.confluent/confluent-log4j-extensions](https://mvnrepository.com/artifact/io.confluent/confluent-log4j-extensions)

First of all, about these libraries you are mentioning - for the reason unknown to me, I don’t have the mentioned libraries in my `$CONFLUENT_BASE/share/java/confluent-common` for my installation.

I also checked it for the most current version of Confluent Platform Community and Enterprise (because the current is 6.1.1 and I am now still on 6.1.0) - and there no such libraries there as well under `confluent-common` directory.

However, in both my installation and the most current one, I have these libraries in `$CONFLUENT_BASE/share/java/ksqldb` directory.

I don’t know if it was intended to be like this, but this is the reality we have here. I don’t know though how all these libraries are loaded from `$CONFLUENT_BASE/share/java` by the Confluent Platform processes, probably @rmoff could clarify on this a little more.

Nevertheless, I was quite sure that unless I use the `io.confluent.common.logging.log4j.StructuredJsonLayout` as the log message layout, I shouldn’t need these libraries, and that is why I looked for other differences in yours and mine configs.

> [@mmuehlbeyer](#):
>
> config starts here

So, I noticed, that you are using your `kafka_appender` only for particular loggers, and you didn’t add it as a root logger.

And when I adjusted my config the same way, so that only Kafka Connect logger logs are written to my Kafka Topic, **it all started to work!**

Everything else about the config stayed the same - the topic name starting with `_` symbol and having dash `-` symbol in the name.

The bottomline is, that **you probably cannot use `KafkaLog4jAppender` for the root logger**.

My final and working `connect-log4j.properties` file looks like this now:

```auto
log4j.rootLogger=INFO, stdout, connectAppender

log4j.logger.org.apache.kafka.connect=INFO, connectKafkaAppender

log4j.logger.org.apache.zookeeper=ERROR
log4j.logger.org.reflections=ERROR

# The `%X{connector.context}` parameter in the layout includes connector-specific and task-specific information
# in the log message, where appropriate. This makes it easier to identify those log messages that apply to a
# specific connector. Simply add this parameter to the log layout configuration below to include the contextual information.
#
#connect.log.pattern=[%d] %p %m (%c:%L)%n
connect.log.pattern=[%d] %p %X{connector.context}%m (%c:%L)%n

# Send the logs to the console.
#
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=${connect.log.pattern}

# Send the logs to a file, rolling the file at midnight local time. For example, the `File` option specifies the
# location of the log files (e.g. ${kafka.logs.dir}/connect.log), and at midnight local time the file is closed
# and copied in the same directory but with a filename that ends in the `DatePattern` option.
#
log4j.appender.connectAppender=org.apache.log4j.DailyRollingFileAppender
log4j.appender.connectAppender.DatePattern='.'yyyy-MM-dd-HH
log4j.appender.connectAppender.File=${kafka.logs.dir}/connect.log
log4j.appender.connectAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.connectAppender.layout.ConversionPattern=${connect.log.pattern}

# Send the logs to a kafka topic
# there will be no key for the kafka records by default
log4j.appender.connectKafkaAppender=org.apache.kafka.log4jappender.KafkaLog4jAppender
log4j.appender.connectKafkaAppender.brokerList=localhost:9092
log4j.appender.connectKafkaAppender.topic=_connect-log
log4j.appender.connectKafkaAppender.compressionType=none
log4j.appender.connectKafkaAppender.ignoreExceptions=true
log4j.appender.connectKafkaAppender.syncSend=false
log4j.appender.connectKafkaAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.connectKafkaAppender.layout.ConversionPattern=${connect.log.pattern}

```

---

_[View the full topic](https://forum.confluent.io/t/problem-with-writing-kafka-connect-log4j-logs-to-a-kafka-topic/1659)._