I’m trying to create a Kafka Producer inside a Lambda Function with Exactly-Once Delivery support enabled to push messages to MSK. Edit: MSK IAM Auth is used for security protocol between Kafka and clients However, even though (I think) I’ve setup all the configurations correctly, Producer still …

Hi, yep, seems like the issue is with AWS IAM Auth for MSK. Not sure exactly what though, because with IAM Auth and no transactions - everything works just fine, but IAM Auth + Transactions = No connection even attempted to be established on the broker side. P.S. Trying to establish IAM Auth via Py…

Possibly it has to do with the authorization policy attached to the role – there are some actions that need to be allowed for transactions to work. IAM access control - Amazon Managed Streaming for Apache Kafka How to use IAM for authentication and authorization. …

Nope, in order to test this, I`ve given kafka-cluster:* and kafka:* permissions to the role attached to the Lambda - nothing changed

@kojimba were you able to find solution for this ? I am stuck with the similar situation

@dtroiano is there any solution for this ? This happens only with transactional producer with IAM auth

@Shashicr nope, could not fix it unfortunately, only work around it. Here’s my question on stack overflow as well. apache kafka - AWS MSK Transactions Support - Stack Overflow

Since this is MSK / IAM-specific, you might also try asking in AWS re:Post .

MSK Kafka Transactions Support

Clients

dtroiano 21 February 2024 17:06 2

I’d recommend testing the transactional producer on an unauthenticated cluster (leave all other cluster configs the same) so that you can rule security config in or out as the area to debug.

Topic		Replies	Views
MSK Serverless - kafka-console-producer error with Kafka 3.4.0 version Stream Processing	1	3093	27 September 2023
Kafka Producer time out during initTransactions while one broker is down Kafka Streams	3	6250	29 September 2023
Trying to connect to an AWS MSK cluster using Confluent SDK Non-Java Clients	0	3561	13 June 2022
Kafka Producer Error: Failed to Find Transaction Coordinator and Timed Out Waiting for Operation to Finish Clients	0	931	3 June 2024
Not work Kafka Transactions client developed by Python Non-Java Clients	2	3529	24 March 2022

MSK Kafka Transactions Support

Related topics