Adding appender to logging

Kafka Connect
1

I am deploying kafka-connect to kubernetes. The dockerfile is setup like so:

FROM confluentinc/cp-kafka-connect:7.9.0

COPY plugins /usr/share/java


USER root

COPY logging/connect-log4j.properties /etc/kafka/connect-log4j.properties
RUN yum install -y unzip

RUN yum update glibc-common -y

ARG NewRelic=https://download.newrelic.com/newrelic/java-agent/newrelic-agent/current/newrelic-java.zip
RUN curl -O $NewRelic
RUN unzip -o newrelic-java.zip -d /opt
USER appuser

#add this line in to increase log level
ENV  CONNECT_LOG4J_LOGGERS="org.apache.kafka.connect=FATAL"

The connect-log4j.properties file is:

log4j.rootLogger=INFO, stdout

log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern =[%d] %p %X{dbz.connectorType}|%X{dbz.connectorName}|%X{dbz.connectorContext} %m (%c)%n


# loggers from CONNECT_LOG4J_LOGGERS env variable
log4j.logger.org.apache.kafka.connect=FATAL

If I attempt to deploy the docker above using that log4j file I get an error when trying to mount it in the pod:

[Errno 13] Permission denied: ‘/etc/kafka/connect-log4j.properties’

I can upload the file later to that location just fine, the error only seems to happen when trying to insert the file using the dockerfile above.

Is there another way that I should try to add, specifically, the log4j.appender.stdout.layout.ConversionPattern value to the container in the dockerfile? An environment variable perhaps? I looked at the documentation here, but did not see anything covering ConversionPattern.

Any help would be welcome. Thanks!

2

I figured it out. Here is the solution.

The connect-log4j.properties file is:

log4j.rootLogger=INFO, stdout

log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern =[%d] %p %X{dbz.connectorType}|%X{dbz.connectorName}|%X{dbz.connectorContext} %m (%c)%n


# loggers from CONNECT_LOG4J_LOGGERS env variable
log4j.logger.org.apache.kafka.connect=FATAL

The dockerfile is:

FROM confluentinc/cp-kafka-connect:7.9.0

COPY plugins /usr/share/java
COPY logging/connect-log4j.properties /etc/kafka/overrides/connect-log4j.properties


USER root

RUN yum install -y unzip

RUN yum update glibc-common -y

ARG NewRelic=https://download.newrelic.com/newrelic/java-agent/newrelic-agent/current/newrelic-java.zip
RUN curl -O $NewRelic
RUN unzip -o newrelic-java.zip -d /opt
USER appuser


#add this line in to increase log level
ENV  CONNECT_LOG4J_LOGGERS="org.apache.kafka.connect=FATAL"
ENV  KAFKA_LOG4J_OPTS="-Dlog4j.configuration=file:/etc/kafka/overrides/connect-log4j.properties"

Basically plugging in the properties file to the environment variable KAFKA_LOG4J_OPTS

Thanks!

1 Like
3

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.