How to sink message from audit log topic to elastic search

[ravvi]

I want to sink audit log topic in confluent cloud to elastic search kafka connector .
I tried to use elastic search sink connector in confluent cloud but while configuring it ,I am unable to see audit log topic in the list of topics shown in the confluent cloud UI.
Is there any way I can do this ?

[dtroiano]

Hi @ravvi and welcome!

Audit logs can’t be accessed from Confluent Cloud fully managed connectors. That may change in the future. The way to do this is to self-manage the connector. Please see the docs about this here and an example here.

HTH,
Dave

[ravvi]

Hi @dtroiano, thank you for the help , I tried that on my local
In my case,
I created a docker- compose file for kafka connect and then appending elastic sink connector config on top of that. But while running the whole process, in kafka connect log I am getting some warning,

[2023-03-17 15:37:18,609] WARN [elastic-sink|task-0] Encountered an illegal document error → Response status: ‘BAD_REQUEST’,
2023-03-17 21:07:18 Index: ‘confluent-audit-log-events’,
2023-03-17 21:07:18 Document Id: ‘confluent-audit-log-events+0+35604’.
2023-03-17 21:07:18 Ignoring and will not index record. (io.confluent.connect.elasticsearch.ElasticsearchClient:649)

I do not know why it is coming, do you have idea on this, if yes, then, It is impact the sinking of audit log in elastic search as we are planning to use it on our production

For your reference, I am attaching the docker-compose and elastic search config file

[ravvi]