I have a Kafka cluster running with topic created, now just want to start the Kafka Connect using bin/connect-distributed.sh but it seems to stuck at Session key updated (org.apache.kafka.connect.runtime.distributed.DistributedHerder:1577)
I have tried using the bin/connect-standalone.sh on the local Kafka and the messages are coming through fine.
The cluster is a AWS MSK if that makes a difference.
Your help is much appreciated.
extract of the last few console stack:
[2021-02-12 02:24:06,226] INFO [Producer clientId=producer-3] Cluster ID: MgEwEU0sRoyTUOkZzkzc1Q (org.apache.kafka.clients.Metadata:279)
[2021-02-12 02:24:06,234] INFO Started o.e.j.s.ServletContextHandler@3d98d138{/,null,AVAILABLE} (org.eclipse.jetty.server.handler.ContextHandler:916)
[2021-02-12 02:24:06,234] INFO REST resources initialized; server is started and ready to handle requests (org.apache.kafka.connect.runtime.rest.RestServer:319)
[2021-02-12 02:24:06,235] INFO Kafka Connect started (org.apache.kafka.connect.runtime.Connect:57)
[2021-02-12 02:24:06,297] INFO [Consumer clientId=consumer-connect-cluster-3, groupId=connect-cluster] Resetting offset for partition connect-configs-0 to offset 0. (org.apache.kafka.clients.consumer.internals.SubscriptionState:397)
[2021-02-12 02:24:06,335] INFO Finished reading KafkaBasedLog for topic connect-configs (org.apache.kafka.connect.util.KafkaBasedLog:164)
[2021-02-12 02:24:06,336] INFO Started KafkaBasedLog for topic connect-configs (org.apache.kafka.connect.util.KafkaBasedLog:166)
[2021-02-12 02:24:06,336] INFO Started KafkaConfigBackingStore (org.apache.kafka.connect.storage.KafkaConfigBackingStore:280)
[2021-02-12 02:24:06,336] INFO [Worker clientId=connect-1, groupId=connect-cluster] Herder started (org.apache.kafka.connect.runtime.distributed.DistributedHerder:290)
[2021-02-12 02:24:06,401] INFO [Worker clientId=connect-1, groupId=connect-cluster] Cluster ID: MgEwEU0sRoyTUOkZzkzc1Q (org.apache.kafka.clients.Metadata:279)
[2021-02-12 02:24:06,402] INFO [Worker clientId=connect-1, groupId=connect-cluster] Discovered group coordinator b-1.msk.433and.c4.kafka.ap-southeast-2.amazonaws.com:9094 (id: 2147483646 rack: null) (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:841)
[2021-02-12 02:24:06,410] INFO [Worker clientId=connect-1, groupId=connect-cluster] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator:225)
[2021-02-12 02:24:06,410] INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540)
[2021-02-12 02:24:06,458] INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540)
[2021-02-12 02:24:09,465] INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=1, memberId='connect-1-9bf2bd2c-1049-41ad-80d7-982efddc22c0', protocol='sessioned'} (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596)
[2021-02-12 02:24:09,492] INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully synced group in generation Generation{generationId=1, memberId='connect-1-9bf2bd2c-1049-41ad-80d7-982efddc22c0', protocol='sessioned'} (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:753)
[2021-02-12 02:24:09,493] INFO [Worker clientId=connect-1, groupId=connect-cluster] Joined group at generation 1 with protocol version 2 and got assignment: Assignment{error=0, leader='connect-1-9bf2bd2c-1049-41ad-80d7-982efddc22c0', leaderUrl='http://192.168.0.147:8083/', offset=8, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder:1688)
[2021-02-12 02:24:09,493] WARN [Worker clientId=connect-1, groupId=connect-cluster] Catching up to assignment's config offset. (org.apache.kafka.connect.runtime.distributed.DistributedHerder:1094)
[2021-02-12 02:24:09,494] INFO [Worker clientId=connect-1, groupId=connect-cluster] Current config state offset -1 is behind group assignment 8, reading to end of config log (org.apache.kafka.connect.runtime.distributed.DistributedHerder:1157)
[2021-02-12 02:24:09,828] INFO [Worker clientId=connect-1, groupId=connect-cluster] Finished reading to end of log and updated config snapshot, new config log offset: 8 (org.apache.kafka.connect.runtime.distributed.DistributedHerder:1161)
[2021-02-12 02:24:09,828] INFO [Worker clientId=connect-1, groupId=connect-cluster] Starting connectors and tasks using config offset 8 (org.apache.kafka.connect.runtime.distributed.DistributedHerder:1215)
[2021-02-12 02:24:09,828] INFO [Worker clientId=connect-1, groupId=connect-cluster] Finished starting connectors and tasks (org.apache.kafka.connect.runtime.distributed.DistributedHerder:1243)
[2021-02-12 02:24:09,906] INFO [Worker clientId=connect-1, groupId=connect-cluster] Session key updated (org.apache.kafka.connect.runtime.distributed.DistributedHerder:1577)
