Schema Registry fails to start

Im using Confluent kafka 6.1.0 with JDK 8 and trying to start the services with confluent local services start but getting “Schema Registry failed to start” with the error:
io.confluent.kafka.schemaregistry.exceptions.SchemaRegistryInitializationException: io.confluent.kafka.schemaregistry.exceptions.SchemaRegistryTimeoutException: Timed out waiting for join group to complete
Caused by: io.confluent.kafka.schemaregistry.exceptions.SchemaRegistryTimeoutException: Timed out waiting for join group to complete

This is a standalone local env. Any idea what might be causing this error?

Can you please ensure connectivity to Kafka Brokers? Also can you please validate
kafkastore.bootstrap.servers points to the correct broker address?

Yes the settings are:
confluent-6.1.0/etc/schema-registry/schema-registry.properties
kafkastore.bootstrap.servers=PLAINTEXT://:9092

confluent-6.1.0/etc/kafka/server.properties
listeners=PLAINTEXT://:9092

hey @hjconfka
any errors in brokers logs?

best,
michael

Just ran the start service again with the following:
$ confluent local services start
The local commands are intended for a single-node development environment only,
NOT for production usage. https://docs.confluent.io/current/cli/index.html

Using CONFLUENT_CURRENT: /tmp/confluent.184303
Starting ZooKeeper
ZooKeeper is [UP]
Starting Kafka
Kafka is [UP]
Starting Schema Registry
Schema Registry is [UP]
Starting Kafka REST
Kafka REST is [UP]
Starting Connect
Error: Connect failed to start

$ confluent local services status
The local commands are intended for a single-node development environment only,
NOT for production usage. https://docs.confluent.io/current/cli/index.html

Using CONFLUENT_CURRENT: /tmp/confluent.184303
Connect is [DOWN]
Control Center is [DOWN]
Kafka is [DOWN]
Kafka REST is [UP]
ksqlDB Server is [DOWN]
Schema Registry is [UP]
ZooKeeper is [UP]

in the connect.log:
org.apache.kafka.connect.errors.ConnectException: Failed to connect to and describe Kafka cluster. Check worker’s broker connection and security properties.
at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:72)
at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:53)
at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:95)
at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:78)
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Call(callName=listNodes, deadlineMs=1678115641688, tries=1, nextAllowedTryMs=1678115641789) timed out at 1678115641689 after 1 attempt(s)
at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:66)
… 3 more
Caused by: org.apache.kafka.common.errors.TimeoutException: Call(callName=listNodes, deadlineMs=1678115641688, tries=1, nextAllowedTryMs=1678115641789) timed out at 1678115641689 after 1 attempt(s)
Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: listNodes

hey
as the Kafka broker does not start I would recommend
to start checking the log of the broker.
Then proceed further to the other services.

best,
michael

Hi Michael,
Thx for your quick response. Here are the errors I see in the server.log:
[2023-03-06 07:11:43,559] INFO [Admin Manager on Broker 0]: Error processing alter configs request for resource ConfigResource(type=TOPIC, name=‘_confluent_balancer_broker_samples’), config ArrayBuffer(AlterConfigOp{opType=SET, configEntry=ConfigEntry(name=cleanup.policy, value=delete, source=UNKNOWN, isSensitive=false, isReadOnly=false, synonyms=)}, AlterConfigOp{opType=SET, configEntry=ConfigEntry(name=retention.ms, value=12000000, source=UNKNOWN, isSensitive=false, isReadOnly=false, synonyms=)}) (kafka.server.AdminManager)
org.apache.kafka.common.errors.UnknownTopicOrPartitionException: The topic ‘_confluent_balancer_broker_samples’ does not exist.
[2023-03-06 07:11:43,564] ERROR Skip updating configuration of topic _confluent_balancer_broker_samples due to exception. (com.linkedin.kafka.cruisecontrol.SbkTopicUtils)
java.lang.RuntimeException: org.apache.kafka.common.errors.UnknownTopicOrPartitionException: The topic ‘_confluent_balancer_broker_samples’ does not exist.
at com.linkedin.kafka.cruisecontrol.SbkTopicUtils.ensureTopicCreated(SbkTopicUtils.java:121)
at com.linkedin.kafka.cruisecontrol.SbkTopicUtils.checkTopicPropertiesMaybeCreate(SbkTopicUtils.java:53)
at com.linkedin.kafka.cruisecontrol.monitor.sampling.KafkaSampleStore.configure(KafkaSampleStore.java:150)
at com.linkedin.kafka.cruisecontrol.config.KafkaCruiseControlConfig.getConfiguredInstance(KafkaCruiseControlConfig.java:1367)
at io.confluent.databalancer.startup.StartupComponents$Builder.build(StartupComponents.java:91)
at io.confluent.databalancer.startup.CruiseControlStartable.startUp(CruiseControlStartable.java:73)
at io.confluent.databalancer.ConfluentDataBalanceEngine.doStart(ConfluentDataBalanceEngine.java:425)
at io.confluent.databalancer.ConfluentDataBalanceEngine.doStart(ConfluentDataBalanceEngine.java:412)
at io.confluent.databalancer.ConfluentDataBalanceEngine.lambda$onActivation$3(ConfluentDataBalanceEngine.java:151)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at com.linkedin.kafka.cruisecontrol.common.KafkaCruiseControlThreadFactory.lambda$newThread$1(KafkaCruiseControlThreadFactory.java:43)
at java.lang.Thread.run(Thread.java:748)

[2023-03-06 07:11:48,118] ERROR [KafkaServer id=0] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
java.lang.NoSuchMethodError: org.apache.kafka.connect.util.KafkaBasedLog.(Ljava/lang/String;Ljava/util/Map;Ljava/util/Map;Ljava/util/function/Supplier;Lorg/apache/kafka/connect/util/Callback;Lorg/apache/kafka/common/utils/Time;Ljava/util/function/Consumer;)V
at io.confluent.license.LicenseStore.createKafkaBasedLog(LicenseStore.java:201)
at io.confluent.license.LicenseStore.setupAndCreateKafkaBasedLog(LicenseStore.java:175)
at io.confluent.license.LicenseStore.(LicenseStore.java:120)
at io.confluent.license.validator.ConfluentLicenseValidator$KafkaLicenseStore.(ConfluentLicenseValidator.java:382)
at io.confluent.license.validator.ConfluentLicenseValidator.createLicenseStore(ConfluentLicenseValidator.java:217)
at io.confluent.license.validator.ConfluentLicenseValidator.start(ConfluentLicenseValidator.java:122)
at kafka.server.KafkaServer.startup(KafkaServer.scala:677)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:44)
at kafka.Kafka$.main(Kafka.scala:82)
at kafka.Kafka.main(Kafka.scala)


[2023-03-06 07:11:51,236] ERROR Exiting Kafka. (kafka.server.KafkaServerStartable)

WARN [Producer clientId=confluent-telemetry-reporter-local-producer] Connection to node 0 (/127.0.1.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

hmm anything else in the logs?
not getting the exact error from the snippets above.

which confluent version did you use? 6.1?

best,
michael

Hi Michael, yes it is 6.1.0 installed on a Linux OS. I’m not seeing any other errors in the server.log. Sometimes NoSuchMethodError indicates the wrong method signature which might mean older/wrong class file or possibly wrong jar file - just guessing at this point :slight_smile: .

Hi @hjconfka
basically if you don’t have any constraints forcing you to use confluent 6.1
I would at least recommend to use Confluent platform 7.3.x

best,
michael

Thx Micheal, I’ll try it with 7.3.2 and update this thread…

1 Like

Thx Michael, 7.3.2 seems to be working well with JDK 11.
I started using the Spool Dir Connector which is working but can’t seem to find the docs for any specific Configuration Properties. Any idea where I might be able to find it…

ok cool

you’re talking about this one?

best,
michael

Yes, but for some reason they’ve decided to put it here instead:
https://docs.confluent.io/kafka-connectors/spooldir/current/connectors/csv_source_connector.html#file-system