I am using confluentinc/cp-ksqldb-server:7.6.1 and wish to reconfigure the logging so that a Logstash-compatible JSON output is produced.
I tried using io.confluent.common.logging.log4j.StructuredJsonLayout, but unfortunately, it is not Logstash compatible out of the box (e.g. has a time field instead of the expected @timestamp) and is not configurable (to e.g. rename the fields).
I’ve also tried to put e.g. log4j-layout-template-json:2.17.1 on the classpath and configuring the layout in log4j.properties, but am getting the following error:
2024-07-12T15:26:39.168891619Z log4j:ERROR A "org.apache.logging.log4j.layout.template.json.JsonTemplateLayout" object is not assignable to a "org.apache.log4j.Layout" variable.
meaning that ksqldb is using the older log4j 1 version (as of version 2 the packages were changed to org.apache.logging.**).
I’d like to avoid having to provide super old jars on the classpath only to get the logging going the way I’d like it to.
Just to try it out, I tried the approach in the docs: Logging - ksqlDB Documentation , but unfortunately could not get this to work. Regardless, I would prefer my logs to be able to be sent to Logstash directly without using Kafka as an intermediary there.
Is there a suggested way to configure the ksqlDB logging appenders to produce Logstash-compatible JSON output?