I have been working with MSK Serverless and kafka-clients 3.4.0. Everything worked as it should when I used * for a topic name.
But, when I had specified a topic name and tried to produce to it: arn:aws:kafka:region:account-id:topic/cluster-name/cluster-uuid/private-topic-1
I got the following error: Caused by: org.apache.kafka.common.errors.ClusterAuthorizationException: Cluster authorization failed
So, I decided to use two terminals in parallel: 1: kafka-3.0.0 2. kafka-3.4.0
I made the same configuration for both version. The same command(kafka-console-producer) works on the version 3.0.0, but failed on the version 3.4.0 (Cluster authorization failed).
I decided to try multiple version, and conclusion is that everything works as it should up to kafka 3.0.1 version. Producer start failing with the 3.0.1 version.
Consumer works with the specified topic on the version 3.4.0. So the problem is in the producer.
As I read, MSK Serverless works with the kafka 2.8.1 version, but is there any way to fix this issue, so I can use kafka-clients 3.4.0. I don’t want to downgrade kafka-clients version on my Java project.