Kafka Producer Error: Failed to Find Transaction Coordinator and Timed Out Waiting for Operation to Finish

mican · 3 June 2024 11:04

Hi everyone,

I’m encountering a couple of issues while using the Confluent Kafka Python client to produce messages with transactions enabled. Here are the details of the problem:

from confluent_kafka import Producer, KafkaError
import uuid

conf = {
    "bootstrap.servers": "10.10.10.165:9092"
    "transactional.id": f"transaction_{str(uuid.uuid4())}",
    "enable.idempotence": True,
    "debug": "all"
}

producer = Producer(conf)

def delivery_report(err, msg):
    if err is not None:
        print(f'Message delivery failed: {err}')
    else:
        print(f'Message delivered to {msg.topic()} [{msg.partition()}]')

try:
    producer.init_transactions()
    producer.begin_transaction()
    producer.produce('my-topic', key='key', value='value', callback=delivery_report)
    producer.commit_transaction()
except KafkaError as e:
    print(f'Error: {e}')
    producer.abort_transaction()

version:
confluent_kafka(‘2.3.0’, 33751040)
confluent_kafka.libversion(‘2.3.0’, 33751295)

logs:

%7|1717417496.187|BRKMAIN|rdkafka#producer-1| [thrd::0/internal]: :0/internal: Enter main broker thread
%7|1717417496.189|BROKER|rdkafka#producer-1| [thrd:main]: TxnCoordinator: Added new broker with NodeId -1
%7|1717417496.190|BRKMAIN|rdkafka#producer-1| [thrd:TxnCoordinator]: TxnCoordinator: Enter main broker thread
%7|1717417496.190|STATE|rdkafka#producer-1| [thrd:TxnCoordinator]: TxnCoordinator: Broker changed state INIT -> TRY_CONNECT
%7|1717417496.190|BROADCAST|rdkafka#producer-1| [thrd:TxnCoordinator]: Broadcasting state change
%7|1717417496.190|CONNECT|rdkafka#producer-1| [thrd:TxnCoordinator]: TxnCoordinator: broker in state TRY_CONNECT connecting
%7|1717417496.190|CONNECT|rdkafka#producer-1| [thrd:TxnCoordinator]: TxnCoordinator: broker has no address yet: postponing connect
%7|1717417496.190|CONNECT|rdkafka#producer-1| [thrd:TxnCoordinator]: TxnCoordinator: broker in state TRY_CONNECT connecting
%7|1717417496.190|CONNECT|rdkafka#producer-1| [thrd:TxnCoordinator]: TxnCoordinator: broker has no address yet: postponing connect
%7|1717417496.206|BROKER|rdkafka#producer-1| [thrd:app]: 10.10.10.165:9092/bootstrap: Added new broker with NodeId -1
%7|1717417496.206|CONNECT|rdkafka#producer-1| [thrd:app]: 10.10.10.165:9092/bootstrap: Selected for cluster connection: bootstrap servers added (broker has 0 connection attempt(s))
%7|1717417496.206|INIT|rdkafka#producer-1| [thrd:app]: librdkafka v2.3.0 (0x20300ff) rdkafka#producer-1 initialized (builtin.features gzip,snappy,ssl,sasl,regex,lz4,sasl_gssapi,sasl_plain,sasl_scram,plugins,zstd,sasl_oauthbearer,http,oidc, SSL ZLIB SNAPPY ZSTD CURL SASL_SCRAM SASL_OAUTHBEARER PLUGINS HDRHISTOGRAM, debug 0xfffff)
%7|1717417496.206|CONF|rdkafka#producer-1| [thrd:app]: Client configuration:
%7|1717417496.206|CONF|rdkafka#producer-1| [thrd:app]:   client.software.name = confluent-kafka-python
%7|1717417496.206|CONF|rdkafka#producer-1| [thrd:app]:   client.software.version = 2.3.0-rdkafka-2.3.0
%7|1717417496.206|CONF|rdkafka#producer-1| [thrd:app]:   metadata.broker.list = 10.10.10.165:9092
%7|1717417496.206|CONF|rdkafka#producer-1| [thrd:app]:   debug = generic,broker,topic,metadata,feature,queue,msg,protocol,cgrp,security,fetch,interceptor,plugin,consumer,admin,eos,mock,assignor,conf,all
%7|1717417496.206|CONF|rdkafka#producer-1| [thrd:app]:   error_cb = 00007FFA2CCB1140
%7|1717417496.206|CONF|rdkafka#producer-1| [thrd:app]:   opaque = 000002A8A8F03D80
%7|1717417496.206|CONF|rdkafka#producer-1| [thrd:app]:   transactional.id = test
%7|1717417496.206|CONF|rdkafka#producer-1| [thrd:app]:   enable.idempotence = true
%7|1717417496.206|CONF|rdkafka#producer-1| [thrd:app]:   dr_msg_cb = 00007FFA2CCAE760
%7|1717417496.207|BRKMAIN|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap: Enter main broker thread
%7|1717417496.207|CONNECT|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap: Received CONNECT op
%7|1717417496.208|STATE|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap: Broker changed state INIT -> TRY_CONNECT
%7|1717417496.208|BROADCAST|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: Broadcasting state change
%7|1717417496.208|CONNECT|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap: broker in state TRY_CONNECT connecting
%7|1717417496.208|STATE|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap: Broker changed state TRY_CONNECT -> CONNECT
%7|1717417496.208|BROADCAST|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: Broadcasting state change
%7|1717417496.210|CONNECT|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap: Connecting to ipv4#10.10.10.165:9092 (plaintext) with socket 536
%7|1717417496.221|TXNAPI|rdkafka#producer-1| [thrd:app]: Transactional API called: init_transactions (in txn state Init, idemp state Init, API timeout 120000)
%7|1717417496.224|TXNSTATE|rdkafka#producer-1| [thrd:main]: Transaction state change Init -> WaitPID
%7|1717417496.224|IDEMPSTATE|rdkafka#producer-1| [thrd:main]: Idempotent producer state change Init -> RequestPID
%7|1717417496.224|TXN|rdkafka#producer-1| [thrd:main]: Starting PID FSM timer (fire immediately): Starting idempotent producer
%7|1717417496.230|CONNECT|rdkafka#producer-1| [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 26ms: acquire ProducerID
%7|1717417496.230|PIDBROKER|rdkafka#producer-1| [thrd:main]: No brokers available for Transactions (2 broker(s) known)
%7|1717417496.230|TXNCOORD|rdkafka#producer-1| [thrd:main]: Unable to query for transaction coordinator: Acquire PID: No brokers available for Transactions (2 broker(s) known)
%7|1717417496.244|CONNECT|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap: Connected to ipv4#10.10.10.165:9092
%7|1717417496.244|CONNECTED|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap: Connected (#1)
%7|1717417496.244|FEATURE|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap: Updated enabled protocol features +ApiVersion to ApiVersion
%7|1717417496.244|STATE|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap: Broker changed state CONNECT -> APIVERSION_QUERY
%7|1717417496.244|BROADCAST|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: Broadcasting state change
%7|1717417496.244|SEND|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap: Sent ApiVersionRequest (v3, 66 bytes @ 0, CorrId 1)
%7|1717417496.265|RECV|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap: Received ApiVersionResponse (v3, 438 bytes, CorrId 1, rtt 20.66ms)
%7|1717417496.265|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap: Broker API support:
%7|1717417496.265|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:   ApiKey Produce (0) Versions 0..9
%7|1717417496.265|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:   ApiKey Fetch (1) Versions 0..15
%7|1717417496.265|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:   ApiKey ListOffsets (2) Versions 0..8
%7|1717417496.265|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:   ApiKey Metadata (3) Versions 0..12
%7|1717417496.265|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:   ApiKey LeaderAndIsr (4) Versions 0..7
%7|1717417496.265|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:   ApiKey StopReplica (5) Versions 0..4
%7|1717417496.265|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:   ApiKey UpdateMetadata (6) Versions 0..8
%7|1717417496.265|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:   ApiKey ControlledShutdown (7) Versions 0..3
%7|1717417496.265|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:   ApiKey OffsetCommit (8) Versions 0..8
%7|1717417496.265|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:   ApiKey OffsetFetch (9) Versions 0..8
%7|1717417496.265|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:   ApiKey FindCoordinator (10) Versions 0..4
%7|1717417496.265|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:   ApiKey JoinGroup (11) Versions 0..9
%7|1717417496.265|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:   ApiKey Heartbeat (12) Versions 0..4
%7|1717417496.265|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:   ApiKey LeaveGroup (13) Versions 0..5
%7|1717417496.265|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:   ApiKey SyncGroup (14) Versions 0..5
%7|1717417496.265|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:   ApiKey DescribeGroups (15) Versions 0..5
%7|1717417496.265|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:   ApiKey ListGroups (16) Versions 0..4
%7|1717417496.265|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:   ApiKey SaslHandshake (17) Versions 0..1
%7|1717417496.265|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:   ApiKey ApiVersion (18) Versions 0..3
%7|1717417496.265|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:   ApiKey CreateTopics (19) Versions 0..7
%7|1717417496.265|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:   ApiKey DeleteTopics (20) Versions 0..6
%7|1717417496.265|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:   ApiKey DeleteRecords (21) Versions 0..2
%7|1717417496.265|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:   ApiKey InitProducerId (22) Versions 0..4
%7|1717417496.265|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:   ApiKey OffsetForLeaderEpoch (23) Versions 0..4
%7|1717417496.265|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:   ApiKey AddPartitionsToTxn (24) Versions 0..4
%7|1717417496.277|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:   ApiKey AddOffsetsToTxn (25) Versions 0..3
%7|1717417496.277|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:   ApiKey EndTxn (26) Versions 0..3
%7|1717417496.277|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:   ApiKey WriteTxnMarkers (27) Versions 0..1
%7|1717417496.277|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:   ApiKey TxnOffsetCommit (28) Versions 0..3
%7|1717417496.277|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:   ApiKey DescribeAcls (29) Versions 0..3
%7|1717417496.277|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:   ApiKey CreateAcls (30) Versions 0..3
%7|1717417496.277|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:   ApiKey DeleteAcls (31) Versions 0..3
%7|1717417496.277|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:   ApiKey DescribeConfigs (32) Versions 0..4
%7|1717417496.277|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:   ApiKey AlterConfigs (33) Versions 0..2
%7|1717417496.277|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:   ApiKey AlterReplicaLogDirs (34) Versions 0..2
%7|1717417496.277|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:   ApiKey DescribeLogDirs (35) Versions 0..4
%7|1717417496.277|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:   ApiKey SaslAuthenticate (36) Versions 0..2
%7|1717417496.277|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:   ApiKey CreatePartitions (37) Versions 0..3
%7|1717417496.277|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:   ApiKey CreateDelegationToken (38) Versions 0..3
%7|1717417496.277|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:   ApiKey RenewDelegationToken (39) Versions 0..2
%7|1717417496.277|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:   ApiKey ExpireDelegationToken (40) Versions 0..2
%7|1717417496.277|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:   ApiKey DescribeDelegationToken (41) Versions 0..3
%7|1717417496.277|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:   ApiKey DeleteGroups (42) Versions 0..2
%7|1717417496.277|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:   ApiKey ElectLeadersRequest (43) Versions 0..2
%7|1717417496.277|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:   ApiKey IncrementalAlterConfigsRequest (44) Versions 0..1
%7|1717417496.277|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:   ApiKey AlterPartitionReassignmentsRequest (45) Versions 0..0
%7|1717417496.277|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:   ApiKey ListPartitionReassignmentsRequest (46) Versions 0..0
%7|1717417496.277|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:   ApiKey OffsetDeleteRequest (47) Versions 0..0
%7|1717417496.277|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:   ApiKey DescribeClientQuotasRequest (48) Versions 0..1
%7|1717417496.277|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:   ApiKey AlterClientQuotasRequest (49) Versions 0..1
%7|1717417496.277|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:   ApiKey DescribeUserScramCredentialsRequest (50) Versions 0..0
%7|1717417496.277|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:   ApiKey AlterUserScramCredentialsRequest (51) Versions 0..0
%7|1717417496.277|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:   ApiKey AlterIsrRequest (56) Versions 0..3
%7|1717417496.277|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:   ApiKey UpdateFeaturesRequest (57) Versions 0..1
%7|1717417496.277|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:   ApiKey EnvelopeRequest (58) Versions 0..0
%7|1717417496.277|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:   ApiKey DescribeCluster (60) Versions 0..0
%7|1717417496.277|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:   ApiKey DescribeProducers (61) Versions 0..0
%7|1717417496.277|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:   ApiKey DescribeTransactions (65) Versions 0..0
%7|1717417496.277|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:   ApiKey ListTransactions (66) Versions 0..0
%7|1717417496.277|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:   ApiKey AllocateProducerIds (67) Versions 0..0
%7|1717417496.277|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:  Feature MsgVer1: Produce (2..2) supported by broker
%7|1717417496.277|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:  Feature MsgVer1: Fetch (2..2) supported by broker
%7|1717417496.277|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap: Enabling feature MsgVer1
%7|1717417496.277|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:  Feature MsgVer2: Produce (3..3) supported by broker
%7|1717417496.277|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:  Feature MsgVer2: Fetch (4..4) supported by broker
%7|1717417496.277|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap: Enabling feature MsgVer2
%7|1717417496.277|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:  Feature ApiVersion: ApiVersion (0..0) supported by broker
%7|1717417496.277|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap: Enabling feature ApiVersion
%7|1717417496.277|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:  Feature BrokerGroupCoordinator: FindCoordinator (0..0) supported by broker
%7|1717417496.277|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap: Enabling feature BrokerGroupCoordinator
%7|1717417496.277|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:  Feature BrokerBalancedConsumer: FindCoordinator (0..0) supported by broker
%7|1717417496.277|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:  Feature BrokerBalancedConsumer: OffsetCommit (1..2) supported by broker
%7|1717417496.277|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:  Feature BrokerBalancedConsumer: OffsetFetch (1..1) supported by broker
%7|1717417496.277|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:  Feature BrokerBalancedConsumer: JoinGroup (0..0) supported by broker
%7|1717417496.277|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:  Feature BrokerBalancedConsumer: SyncGroup (0..0) supported by broker
%7|1717417496.277|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:  Feature BrokerBalancedConsumer: Heartbeat (0..0) supported by broker
%7|1717417496.277|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:  Feature BrokerBalancedConsumer: LeaveGroup (0..0) supported by broker
%7|1717417496.277|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap: Enabling feature BrokerBalancedConsumer
%7|1717417496.277|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:  Feature ThrottleTime: Produce (1..2) supported by broker
%7|1717417496.277|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:  Feature ThrottleTime: Fetch (1..2) supported by broker
%7|1717417496.277|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap: Enabling feature ThrottleTime
%7|1717417496.277|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:  Feature Sasl: JoinGroup (0..0) supported by broker
%7|1717417496.277|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap: Enabling feature Sasl
%7|1717417496.277|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:  Feature SaslHandshake: SaslHandshake (0..0) supported by broker
%7|1717417496.277|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap: Enabling feature SaslHandshake
%7|1717417496.278|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:  Feature LZ4: FindCoordinator (0..0) supported by broker
%7|1717417496.278|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap: Enabling feature LZ4
%7|1717417496.278|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:  Feature OffsetTime: ListOffsets (1..1) supported by broker
%7|1717417496.278|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap: Enabling feature OffsetTime
%7|1717417496.278|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:  Feature IdempotentProducer: InitProducerId (0..0) supported by broker
%7|1717417496.278|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap: Enabling feature IdempotentProducer
%7|1717417496.278|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:  Feature ZSTD: Produce (7..7) supported by broker
%7|1717417496.278|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:  Feature ZSTD: Fetch (10..10) supported by broker
%7|1717417496.278|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap: Enabling feature ZSTD
%7|1717417496.278|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:  Feature SaslAuthReq: SaslHandshake (1..1) supported by broker
%7|1717417496.278|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap:  Feature SaslAuthReq: SaslAuthenticate (0..1) supported by broker
%7|1717417496.278|APIVERSION|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap: Enabling feature SaslAuthReq
%7|1717417496.278|FEATURE|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap: Updated enabled protocol features to MsgVer1,ApiVersion,BrokerBalancedConsumer,ThrottleTime,Sasl,SaslHandshake,BrokerGroupCoordinator,LZ4,OffsetTime,MsgVer2,IdempotentProducer,ZSTD,SaslAuthReq
%7|1717417496.278|STATE|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap: Broker changed state APIVERSION_QUERY -> UP
%7|1717417496.278|BROADCAST|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: Broadcasting state change
%7|1717417496.278|METADATA|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap: Request metadata for brokers only: connected
%7|1717417496.278|SEND|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap: Sent MetadataRequest (v12, 26 bytes @ 0, CorrId 2)
%7|1717417496.299|RECV|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap: Received MetadataResponse (v12, 58 bytes, CorrId 2, rtt 21.22ms)
%7|1717417496.299|METADATA|rdkafka#producer-1| [thrd:main]: 10.10.10.165:9092/bootstrap: ===== Received metadata: connected =====
%7|1717417496.299|METADATA|rdkafka#producer-1| [thrd:main]: 10.10.10.165:9092/bootstrap: ClusterId: 1PpQ1tDSQ_2zghAiLo8XFw, ControllerId: 1
%7|1717417496.299|METADATA|rdkafka#producer-1| [thrd:main]: 10.10.10.165:9092/bootstrap: 1 brokers, 0 topics
%7|1717417496.299|METADATA|rdkafka#producer-1| [thrd:main]: 10.10.10.165:9092/bootstrap:   Broker #0/1: 10.10.10.165:9092 NodeId 1
%7|1717417496.299|UPDATE|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/bootstrap: NodeId changed from -1 to 1
%7|1717417496.299|UPDATE|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/1: Name changed from 10.10.10.165:9092/bootstrap to 10.10.10.165:9092/1
%7|1717417496.299|LEADER|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/1: Mapped 0 partition(s) to broker
%7|1717417496.299|STATE|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/1: Broker changed state UP -> UPDATE
%7|1717417496.299|BROADCAST|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: Broadcasting state change
%7|1717417496.299|BROADCAST|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: Broadcasting state change
%7|1717417496.299|STATE|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/1: Broker changed state UPDATE -> UP
%7|1717417496.299|BROADCAST|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: Broadcasting state change
%7|1717417496.299|CLUSTERID|rdkafka#producer-1| [thrd:main]: 10.10.10.165:9092/1: ClusterId update "" -> "1PpQ1tDSQ_2zghAiLo8XFw"
%7|1717417496.299|CONTROLLERID|rdkafka#producer-1| [thrd:main]: 10.10.10.165:9092/1: ControllerId update -1 -> 1
%7|1717417496.299|BROADCAST|rdkafka#producer-1| [thrd:main]: Broadcasting state change
%7|1717417496.299|TXNCOORD|rdkafka#producer-1| [thrd:main]: Querying for transaction coordinator: Acquire PID
%7|1717417496.311|SEND|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/1: Sent FindCoordinatorRequest (v2, 28 bytes @ 0, CorrId 3)
%7|1717417496.332|RECV|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/1: Received FindCoordinatorResponse (v2, 51 bytes, CorrId 3, rtt 20.17ms)
%7|1717417496.332|TXNCOORD|rdkafka#producer-1| [thrd:main]: Failed to find transaction coordinator: COORDINATOR_NOT_AVAILABLE: The coordinator is not available.
%7|1717417496.733|TXNCOORD|rdkafka#producer-1| [thrd:main]: Querying for transaction coordinator: Coordinator query timer
%7|1717417496.733|SEND|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/1: Sent FindCoordinatorRequest (v2, 28 bytes @ 0, CorrId 4)
%7|1717417496.757|RECV|rdkafka#producer-1| [thrd:10.10.10.165:9092/bootstrap]: 10.10.10.165:9092/1: Received FindCoordinatorResponse (v2, 51 bytes, CorrId 4, rtt 23.97ms)
%7|1717417496.757|TXNCOORD|rdkafka#producer-1| [thrd:main]: Failed to find transaction coordinator: COORDINATOR_NOT_AVAILABLE: The coordinator is not 
Process finished with exit code -1

Error in python:

cimpl.KafkaException: KafkaError{code=_TIMED_OUT,val=-185,str=“Timed out waiting for operation to finish, retry call to resume”}

Any insights or suggestions on what might be causing these issues and how to resolve them would be greatly appreciated. Thank you

Topic		Replies	Views
Not work Kafka Transactions client developed by Python Non-Java Clients	2	3486	24 March 2022
Confluent Cloud error: Broker may not be available Confluent Cloud	11	7428	13 July 2022
Kafka Producer time out during initTransactions while one broker is down Kafka Streams	3	4876	29 September 2023
Error consuming messages from .core application Clients	0	2771	2 November 2022
Error 13/13 brokers are down Non-Java Clients	0	36	16 September 2024

Kafka Producer Error: Failed to Find Transaction Coordinator and Timed Out Waiting for Operation to Finish

Related topics