# How to save Connect logs to file

**URL:** <https://forum.confluent.io/t/how-to-save-connect-logs-to-file/3311>\
**Category:** Kafka Connect\
**Created:** [8 November 2021 10:37 UTC](https://forum.confluent.io/t/how-to-save-connect-logs-to-file/3311 "2021-11-08T10:37:18Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![shlomi](https://avatars.discourse-cdn.com/v4/letter/s/cab0a1/32.png) [@shlomi](https://forum.confluent.io/u/shlomi)\
**Post date:** [8 November 2021 10:37 UTC](https://forum.confluent.io/t/how-to-save-connect-logs-to-file/3311/1 "2021-11-08T10:37:18Z")

</div>

Hey all,  
How do I send connect logs to files? I want to be able to send them into SumoLogic (logs service).

I’m running connect on AWS ECS (docker) and tried to add the following:

```
 - CONNECT_LOG4J_ROOT_LOGLEVEL="INFO, FILE"
 - CONNECT_LOG4J_APPENDER_FILE="org.apache.log4j.RollingFileAppender"
 - CONNECT_LOG4J_APPENDER_FILE_FILE="/var/log/kafka/connect_log"
 - CONNECT_LOG4J_APPENDER_FILE_LAYOUT=org.apache.log4j.PatternLayout
 - CONNECT_LOG4J_APPENDER_FILE_LAYOUT_CONVERSIONPATTERN=[%d] %p %m (%c:%L)%n
 - CONNECT_LOG4J_APPENDER_FILE_MAXFILESIZE=10MB
 - CONNECT_LOG4J_APPENDER_FILE_MAXBACKUPINDEX=5
 - CONNECT_LOG4J_APPENDER_FILE_APPEND=true

```

but getting the following error:

```
log4j:ERROR Could not find value for key log4j.appender.file"
log4j:ERROR Could not instantiate appender named "file"".

```

Any ideas on how to fix it?  
Thanks

---

<div class="post-metadata">

**Author:** ![mmuehlbeyer](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/mmuehlbeyer/32/1088_2.png) [@mmuehlbeyer](https://forum.confluent.io/u/mmuehlbeyer)\
**Post date:** [8 November 2021 13:51 UTC](https://forum.confluent.io/t/how-to-save-connect-logs-to-file/3311/2 "2021-11-08T13:51:31Z")

</div>

hey @shlomi

did you also adapt your  
connect-log4j.properties file?

which image are you using?

Best,  
Michael

---

<div class="post-metadata">

**Author:** ![shlomi](https://avatars.discourse-cdn.com/v4/letter/s/cab0a1/32.png) [@shlomi](https://forum.confluent.io/u/shlomi)\
**Post date:** [8 November 2021 13:55 UTC](https://forum.confluent.io/t/how-to-save-connect-logs-to-file/3311/3 "2021-11-08T13:55:53Z")

</div>

I’m using `confluentinc/cp-kafka-connect-base`. If I can I would like to not change the image and only control it with environment variables.  
Is that possible?

---

<div class="post-metadata">

**Author:** ![mmuehlbeyer](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/mmuehlbeyer/32/1088_2.png) [@mmuehlbeyer](https://forum.confluent.io/u/mmuehlbeyer)\
**Post date:** [8 November 2021 14:02 UTC](https://forum.confluent.io/t/how-to-save-connect-logs-to-file/3311/4 "2021-11-08T14:02:04Z")

</div>

need to check by myself, keep you posted

but I think one possible solution might be to map the /etc/kafka directory contents to your container

---

<div class="post-metadata">

**Author:** ![mmuehlbeyer](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/mmuehlbeyer/32/1088_2.png) [@mmuehlbeyer](https://forum.confluent.io/u/mmuehlbeyer)\
**Post date:** [8 November 2021 14:50 UTC](https://forum.confluent.io/t/how-to-save-connect-logs-to-file/3311/5 "2021-11-08T14:50:28Z")

</div>

short update:  
after a quick scan of the source code of your used docker image, I think the env vars are not evaluated during the startup of the container

but let me check something else

---

<div class="post-metadata">

**Author:** ![shlomi](https://avatars.discourse-cdn.com/v4/letter/s/cab0a1/32.png) [@shlomi](https://forum.confluent.io/u/shlomi)\
**Post date:** [8 November 2021 18:11 UTC](https://forum.confluent.io/t/how-to-save-connect-logs-to-file/3311/6 "2021-11-08T18:11:42Z")

</div>

Thanks for the info. I tried to add

```auto
COPY connect-log4j.properties /etc/kafka/connect-log4j.properties

```

to my `dockerfile` but I’m getting the error:

```
[Errno 13] Permission denied: '/etc/kafka/connect-log4j.properties'
Command [/usr/local/bin/dub template /etc/confluent/docker/log4j.properties.template /etc/kafka/connect-log4j.properties] FAILED !

```

Any advice?

---

<div class="post-metadata">

**Author:** ![mmuehlbeyer](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/mmuehlbeyer/32/1088_2.png) [@mmuehlbeyer](https://forum.confluent.io/u/mmuehlbeyer)\
**Post date:** [9 November 2021 06:36 UTC](https://forum.confluent.io/t/how-to-save-connect-logs-to-file/3311/7 "2021-11-09T06:36:46Z")

</div>

how does your dockerfile and docker run command look like?

could you share it with us?

---

<div class="post-metadata">

**Author:** ![shlomi](https://avatars.discourse-cdn.com/v4/letter/s/cab0a1/32.png) [@shlomi](https://forum.confluent.io/u/shlomi)\
**Post date:** [9 November 2021 07:10 UTC](https://forum.confluent.io/t/how-to-save-connect-logs-to-file/3311/8 "2021-11-09T07:10:33Z")

</div>

dockerfile:

```auto
FROM confluentinc/cp-kafka-connect-base

# Connectors
RUN confluent-hub install --no-prompt confluentinc/kafka-connect-s3:10.0.0
RUN confluent-hub install --no-prompt debezium/debezium-connector-postgresql:1.4.0

COPY set_env.sh /etc/set_env.sh
COPY connect-log4j.properties /etc/kafka/connect-log4j.properties

CMD ["/etc/set_env.sh"]

```

set\_env.sh:

```auto
#!/bin/bash

LOCAL_IP=$(hostname -i)

# Create secrets file for connectors
cat > /tmp/postgres.properties <<EOL
db_name=${DB_NAME}
host_name=${HOST_NAME}
db_user=${DB_USER}
db_password=${DB_PASSWORD}
sslmode=${SSLMODE}
EOL

CONNECT_REST_ADVERTISED_HOST_NAME=$LOCAL_IP /etc/confluent/docker/run

```

---

<div class="post-metadata">

**Author:** ![mmuehlbeyer](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/mmuehlbeyer/32/1088_2.png) [@mmuehlbeyer](https://forum.confluent.io/u/mmuehlbeyer)\
**Post date:** [9 November 2021 09:33 UTC](https://forum.confluent.io/t/how-to-save-connect-logs-to-file/3311/9 "2021-11-09T09:33:11Z")

</div>

hey,

I’ve tested different settings in my local env, not yet a working solution but probably a good point to start.

I’v extended the Dockerfile with the following

```auto
COPY common-logging-6.2.1.jar /usr/share/java/kafka/common-logging-6.2.1.jar
COPY confluent-log4j-extensions-6.2.1.jar /usr/share/java/kafka/confluent-log4j-extensions-6.2.1.jar
COPY kafka-connect.properties /etc/confluent/docker/kafka-connect.properties.template

```

get the jars from  
[https://mvnrepository.com/artifact/io.confluent/common-logging](https://mvnrepository.com/artifact/io.confluent/common-logging)  
[https://mvnrepository.com/artifact/io.confluent/confluent-log4j-extensions](https://mvnrepository.com/artifact/io.confluent/confluent-log4j-extensions)

and then start the container with  
` -e CONNECT_LOG4J_ROOT_LOGLEVEL="DEBUG, connectAppender, stdout" \`

for reference, similar topic but not on docker:

> [@Problem with writing Kafka Connect \`log4j\` logs to a Kafka Topic](https://forum.confluent.io/t/problem-with-writing-kafka-connect-log4j-logs-to-a-kafka-topic/1659/16):
>
> Hi @mmuehlbeyer! So, I finally had a chance to check your proposal, and with some help of it, I found out the actual reason of the issue. First of all, about these libraries you are mentioning - for the reason unknown to me, I don’t have the mentioned libraries in my $CONFLUENT\_BASE/share/java/confluent-common for my installation. I also checked it for the most current version of Confluent Platform Community and Enterprise (because the current is 6.1.1 and I am now still on 6.1.0) - and ther…

---

<div class="post-metadata">

**Author:** ![shlomi](https://avatars.discourse-cdn.com/v4/letter/s/cab0a1/32.png) [@shlomi](https://forum.confluent.io/u/shlomi)\
**Post date:** [15 November 2021 15:32 UTC](https://forum.confluent.io/t/how-to-save-connect-logs-to-file/3311/10 "2021-11-15T15:32:08Z")

</div>

Did you get to test it? what is in the `kafka-connect.properties` file?

---

<div class="post-metadata">

**Author:** ![mmuehlbeyer](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/mmuehlbeyer/32/1088_2.png) [@mmuehlbeyer](https://forum.confluent.io/u/mmuehlbeyer)\
**Post date:** [15 November 2021 16:38 UTC](https://forum.confluent.io/t/how-to-save-connect-logs-to-file/3311/11 "2021-11-15T16:38:01Z")

</div>

not yet  
need to check tomorrow

keep you posted

---

<div class="post-metadata">

**Author:** ![mmuehlbeyer](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/mmuehlbeyer/32/1088_2.png) [@mmuehlbeyer](https://forum.confluent.io/u/mmuehlbeyer)\
**Post date:** [19 November 2021 09:21 UTC](https://forum.confluent.io/t/how-to-save-connect-logs-to-file/3311/12 "2021-11-19T09:21:37Z")

</div>

no working solution till now

but one thing I would like to drop  
some times ago we’ve collected docker logs with fluent and pushed it to S3 with fluent-bit  
it might be worth to have a look  
[https://www.fluentd.org/guides/recipes/docker-logging](https://www.fluentd.org/guides/recipes/docker-logging)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex019/uploads/confluentcommunity/original/1X/c49438c90c9df282e9996fdf6971be890c71b65a.svg) [@system](https://forum.confluent.io/u/system)\
**Post date:** [19 December 2021 09:22 UTC](https://forum.confluent.io/t/how-to-save-connect-logs-to-file/3311/13 "2021-12-19T09:22:03Z")

</div>

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.