# MSK Kafka Transactions Support **URL:** \ **Category:** Clients\ **Created:** [20 February 2024 10:43 UTC](https://forum.confluent.io/t/msk-kafka-transactions-support/10218 "2024-02-20T10:43:18Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![kojimba](https://avatars.discourse-cdn.com/v4/letter/k/ba9def/32.png) [@kojimba](https://forum.confluent.io/u/kojimba)\ **Post date:** [20 February 2024 10:43 UTC](https://forum.confluent.io/t/msk-kafka-transactions-support/10218/1 "2024-02-20T10:43:18Z")
I’m trying to create a Kafka Producer inside a Lambda Function with Exactly-Once Delivery support enabled to push messages to MSK. **Edit** : MSK IAM Auth is used for security protocol between Kafka and clients However, even though (I think) I’ve setup all the configurations correctly, Producer still can’t write messages to MSK. Producer hangs upon calling **init\_transactions()** and outputs the following debug messages in a loop: ```auto %7|1708348719.300|TXNCOORD|lambda#producer-1| [thrd:main]: Unable to query for transaction coordinator: Coordinator query timer: No brokers available for Transactions (3 broker(s) known) 2024-02-19T14:18:39.338+01:00 %7|1708348719.338|CONNECT|lambda#producer-1| [thrd:TxnCoordinator]: TxnCoordinator: broker in state TRY_CONNECT connecting 2024-02-19T14:18:39.338+01:00 %7|1708348719.338|CONNECT|lambda#producer-1| [thrd:TxnCoordinator]: TxnCoordinator: broker has no address yet: postponing connect 2024-02-19T14:18:39.800+01:00 %7|1708348719.800|CONNECT|lambda#producer-1| [thrd:main]: Cluster connection already in progress: acquire ProducerID 2024-02-19T14:18:39.800+01:00 %7|1708348719.800|PIDBROKER|lambda#producer-1| [thrd:main]: No brokers available for Transactions (3 broker(s) known) ``` I have tried changing the number of brokers (from 2 to 4 - did not work), playing around with values for the settings **transaction.state.log.replication.factor** , **transaction.state.log.min.isr** , **offsets.topic.replication.factor** (even setting them all to 1 - did not help). Advice from this thread did not help either [Problems with Amazon MSK default configuration and publishing with transactions](https://stackoverflow.com/questions/65744538/problems-with-amazon-msk-default-configuration-and-publishing-with-transactions) I have the following configs and settings for AWS MSK Cluster: - 4 Brokers across 2 availability zones - Kafka 2.8.1 - Cluster size: kafka.t3.small (I`ve checked and the same occurs on kafka.m5.large) Cluster Configs: ```auto transaction.state.log.replication.factor=3 transaction.state.log.min.isr=2 offsets.topic.replication.factor=3 min.insync.replicas=2 default.replication.factor=3 auto.create.topics.enable=true num.io.threads=8 num.network.threads=2 num.partitions=1 num.replica.fetchers=2 replica.lag.time.max.ms=30000 socket.receive.buffer.bytes=102400 socket.request.max.bytes=104857600 socket.send.buffer.bytes=102400 zookeeper.session.timeout.ms=18000 ``` Producer Configs: - Using Confluent Kafka for Python to create the Producer ```auto { "client.id": "some_id", "acks": "all", "enable.idempotence": "true", "transactional.id": "123", } ``` P.S. If I don’t use the acks, enable.idempotence and transactional.id - Producer works fine, but it beats the purpose of raising this issue in the first place. **UPD** : After digging through logs of MSK brokers, it seems that the connection is not getting established - no idea why, especially because the auth method is the same for transactional and non-transactional Producer and for non-transactional Producer the connection is getting established just fine. ---
**Author:** ![dtroiano](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/dtroiano/32/1961_2.png) [@dtroiano](https://forum.confluent.io/u/dtroiano)\ **Post date:** [21 February 2024 17:06 UTC](https://forum.confluent.io/t/msk-kafka-transactions-support/10218/2 "2024-02-21T17:06:54Z")
I’d recommend testing the transactional producer on an unauthenticated cluster (leave all other cluster configs the same) so that you can rule security config in or out as the area to debug. ---
**Author:** ![kojimba](https://avatars.discourse-cdn.com/v4/letter/k/ba9def/32.png) [@kojimba](https://forum.confluent.io/u/kojimba)\ **Post date:** [22 February 2024 08:17 UTC](https://forum.confluent.io/t/msk-kafka-transactions-support/10218/3 "2024-02-22T08:17:31Z")
Hi, yep, seems like the issue is with AWS IAM Auth for MSK. Not sure exactly what though, because with IAM Auth and no transactions - everything works just fine, but IAM Auth + Transactions = No connection even attempted to be established on the broker side. P.S. Trying to establish IAM Auth via Python and aws-msk-iam-sasl-signer library ---
**Author:** ![dtroiano](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/dtroiano/32/1961_2.png) [@dtroiano](https://forum.confluent.io/u/dtroiano)\ **Post date:** [22 February 2024 14:32 UTC](https://forum.confluent.io/t/msk-kafka-transactions-support/10218/4 "2024-02-22T14:32:41Z")
Possibly it has to do with the authorization policy attached to the role – there are some actions that need to be allowed for transactions to work. > **[IAM access control - Amazon Managed Streaming for Apache Kafka](https://docs.aws.amazon.com/msk/latest/developerguide/iam-access-control.html)** > > How to use IAM for authentication and authorization. ---
**Author:** ![kojimba](https://avatars.discourse-cdn.com/v4/letter/k/ba9def/32.png) [@kojimba](https://forum.confluent.io/u/kojimba)\ **Post date:** [22 February 2024 14:52 UTC](https://forum.confluent.io/t/msk-kafka-transactions-support/10218/5 "2024-02-22T14:52:45Z")
Nope, in order to test this, I`ve given kafka-cluster:\* and kafka:\* permissions to the role attached to the Lambda - nothing changed ---
**Author:** ![Shashicr](https://avatars.discourse-cdn.com/v4/letter/s/73ab20/32.png) [@Shashicr](https://forum.confluent.io/u/Shashicr)\ **Post date:** [23 August 2024 18:04 UTC](https://forum.confluent.io/t/msk-kafka-transactions-support/10218/6 "2024-08-23T18:04:40Z")
@kojimba were you able to find solution for this ? I am stuck with the similar situation ---
**Author:** ![Shashicr](https://avatars.discourse-cdn.com/v4/letter/s/73ab20/32.png) [@Shashicr](https://forum.confluent.io/u/Shashicr)\ **Post date:** [23 August 2024 18:05 UTC](https://forum.confluent.io/t/msk-kafka-transactions-support/10218/7 "2024-08-23T18:05:25Z")
@dtroiano is there any solution for this ? This happens only with transactional producer with IAM auth ---
**Author:** ![kojimba](https://avatars.discourse-cdn.com/v4/letter/k/ba9def/32.png) [@kojimba](https://forum.confluent.io/u/kojimba)\ **Post date:** [24 August 2024 06:05 UTC](https://forum.confluent.io/t/msk-kafka-transactions-support/10218/8 "2024-08-24T06:05:30Z")
@Shashicr nope, could not fix it unfortunately, only work around it. Here’s my question on stack overflow as well. [apache kafka - AWS MSK Transactions Support - Stack Overflow](https://stackoverflow.com/questions/78021233/aws-msk-transactions-support/78033598#78033598) ---
**Author:** ![dtroiano](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/dtroiano/32/1961_2.png) [@dtroiano](https://forum.confluent.io/u/dtroiano)\ **Post date:** [26 August 2024 18:21 UTC](https://forum.confluent.io/t/msk-kafka-transactions-support/10218/9 "2024-08-26T18:21:17Z")
Since this is MSK / IAM-specific, you might also try asking in AWS [re:Post](https://repost.aws/tags/questions/TAhybM_7u2QN2GsWOc4aJVDg).