Confluent local services start not working well

pieterbreed · 4 May 2023 16:44

Hi All,

When I run `confluent local services start` it takes about 90 seconds for the command to return a failure …

… which is incorrect, because zookeeper did in fact start. Here is the output of the first invocation:

$ time confluent local services start
The local commands are intended for a single-node development environment only, NOT for production usage. See more: https://docs.confluent.io/current/cli/index.html
As of Confluent Platform 8.0, Java 8 is no longer supported.

Using CONFLUENT_CURRENT: /tmp/confluent.924759
Starting ZooKeeper
Error: ZooKeeper failed to start

real    1m32.857s
user    0m0.095s
sys     0m0.104s

However status shows me zookeeper is running:

$ time confluent local services status
The local commands are intended for a single-node development environment only, NOT for production usage. See more: https://docs.confluent.io/current/cli/index.html
As of Confluent Platform 8.0, Java 8 is no longer supported.

Using CONFLUENT_CURRENT: /tmp/confluent.924759
Connect is [DOWN]
Control Center is [DOWN]
Kafka is [DOWN]
Kafka REST is [DOWN]
ksqlDB Server is [DOWN]
Schema Registry is [DOWN]
ZooKeeper is [UP]

real    0m2.805s
user    0m0.062s
sys     0m0.015s

again

$ time confluent local services start
The local commands are intended for a single-node development environment only, NOT for production usage. See more: https://docs.confluent.io/current/cli/index.html
As of Confluent Platform 8.0, Java 8 is no longer supported.

Using CONFLUENT_CURRENT: /tmp/confluent.924759
ZooKeeper is [UP]
Starting Kafka
Error: Kafka failed to start

real    1m32.846s
user    0m0.112s
sys     0m0.079s

$ confluent local services status
The local commands are intended for a single-node development environment only, NOT for production usage. See more: https://docs.confluent.io/current/cli/index.html
As of Confluent Platform 8.0, Java 8 is no longer supported.

Using CONFLUENT_CURRENT: /tmp/confluent.924759
Connect is [DOWN]
Control Center is [DOWN]
Kafka is [UP]
Kafka REST is [DOWN]
ksqlDB Server is [DOWN]
Schema Registry is [DOWN]
ZooKeeper is [UP]

Now both zookeeper and kafka is reported as UP, but again the start command timed out with a failure.

If I repeatedly repeat these steps it “works” for ZooKeeper, kafka, Schema Registry & Kafka REST.

This stops working by the time I get to connect

$ time confluent local services start
The local commands are intended for a single-node development environment only, NOT for production usage. See more: https://docs.confluent.io/current/cli/index.html
As of Confluent Platform 8.0, Java 8 is no longer supported.

Using CONFLUENT_CURRENT: /tmp/confluent.924759
ZooKeeper is [UP]
Kafka is [UP]
Schema Registry is [UP]
Kafka REST is [UP]
Starting Connect
Error: Connect failed to start

real    1m32.899s
user    0m0.178s
sys     0m0.093s


$ time confluent local services status
The local commands are intended for a single-node development environment only, NOT for production usage. See more: https://docs.confluent.io/current/cli/index.html
As of Confluent Platform 8.0, Java 8 is no longer supported.

Using CONFLUENT_CURRENT: /tmp/confluent.924759
Connect is [DOWN]
Control Center is [DOWN]
Kafka is [UP]
Kafka REST is [UP]
ksqlDB Server is [DOWN]
Schema Registry is [UP]
ZooKeeper is [UP]

real    0m2.665s
user    0m0.051s
sys     0m0.028s

… so I can’t actually get the whole stack going like this.

Here is some debug environment info:

$ java -version
openjdk version "11" 2018-09-25
OpenJDK Runtime Environment 18.9 (build 11+28)
OpenJDK 64-Bit Server VM 18.9 (build 11+28, mixed mode)

$ uname -a
Linux malaika 6.1.19-gentoo-x86_64 #1 SMP PREEMPT_DYNAMIC Sat Mar 18 17:57:46 SAST 2023 x86_64 AMD Ryzen 9 5900HX with Radeon Graphics AuthenticAMD GNU/Linux

$ echo $CONFLUENT_HOME
/home/pieter/Downloads/confluent/confluent

$ which confluent
/home/pieter/Downloads/confluent/confluent/bin/confluent

All of this relates to confluent-7.4.0.

OneCricketeer · 15 May 2023 23:55

What types of errors do you get when you run the actual start scripts, in order?

zookeeper-server-start
kafka-server-start
schema-registry-start
connect-distributed
ksql-server-start
control center … (forget the script)

Do you actually have enough memory to start all these processes?

If you only want to start Kafka, you should run the first two on their own, as you would in a real, production environment.

pieterbreed · 16 May 2023 08:34

My config files are unmodified as per the confluent tar file.

Zookeeper

$ bin/zookeeper-server-start etc/kafka/zookeeper.properties
...
2023-05-16 10:06:13,802] INFO Snapshot taken in 1 ms (org.apache.zookeeper.server.ZooKeeperServer)
[2023-05-16 10:06:13,810] INFO PrepRequestProcessor (sid:0) started, reconfigEnabled=false (org.apache.zookeeper.server.PrepRequestProcessor)
[2023-05-16 10:06:13,810] INFO zookeeper.request_throttler.shutdownTimeout = 10000 (org.apache.zookeeper.server.RequestThrottler)
[2023-05-16 10:06:13,822] INFO Using checkIntervalMs=60000 maxPerMinute=10000 maxNeverUsedIntervalMs=0 (org.apache.zookeeper.server.ContainerManager)
[2023-05-16 10:06:13,823] INFO ZooKeeper audit is enabled. (org.apache.zookeeper.audit.ZKAuditProvider)

starts up within a second or so.

kafka

$ bin/kafka-server-start etc/kafka/server.properties
...
2023-05-16 10:06:13,802] INFO Snapshot taken in 1 ms (org.apache.zookeeper.server.ZooKeeperServer)
[2023-05-16 10:06:13,810] INFO PrepRequestProcessor (sid:0) started, reconfigEnabled=false (org.apache.zookeeper.server.PrepRequestProcessor)
[2023-05-16 10:06:13,810] INFO zookeeper.request_throttler.shutdownTimeout = 10000 (org.apache.zookeeper.server.RequestThrottler)
[2023-05-16 10:06:13,822] INFO Using checkIntervalMs=60000 maxPerMinute=10000 maxNeverUsedIntervalMs=0 (org.apache.zookeeper.server.ContainerManager)
[2023-05-16 10:06:13,823] INFO ZooKeeper audit is enabled. (org.apache.zookeeper.audit.ZKAuditProvider)

starts up within 2 seconds, no ERROR in output.

schema registry

$ bin/schema-registry-start etc/schema-registry/schema-registry.properties
...
[2023-05-16 10:13:23,838] INFO HV000001: Hibernate Validator 6.1.7.Final (org.hibernate.validator.internal.util.Version:21)
[2023-05-16 10:13:23,993] INFO Started o.e.j.s.ServletContextHandler@5918c260{/,null,AVAILABLE} (org.eclipse.jetty.server.handler.ContextHandler:921)
[2023-05-16 10:13:24,000] INFO Started o.e.j.s.ServletContextHandler@77e7246b{/ws,null,AVAILABLE} (org.eclipse.jetty.server.handler.ContextHandler:921)
[2023-05-16 10:13:24,010] INFO Started NetworkTrafficServerConnector@730d2164{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8081} (org.eclipse.jetty.server.AbstractConnector:333)
[2023-05-16 10:13:24,010] INFO Started @2747ms (org.eclipse.jetty.server.Server:415)
[2023-05-16 10:13:24,010] INFO Schema Registry version: 7.4.0 commitId: 3870d17aa92a8da9bc9272a2ed9ac1678bb58ce6 (io.confluent.kafka.schemaregistry.rest.SchemaRegistryMain:47)
[2023-05-16 10:13:24,011] INFO Server started, listening for requests... (io.confluent.kafka.schemaregistry.rest.SchemaRegistryMain:49)

… starts within a second or so…

connect

I don’t know if I started connect properly, so not sure if this is a user-error or something else.

$ bin/connect-distributed etc/kafka-connect-replicator/replicator.properties
...
[2023-05-16 10:15:51,900] INFO Added aliases 'AllConnectorClientConfigOverridePolicy' and 'All' to plugin 'org.apache.kafka.connect.connector.policy.AllConnectorClientConfigOverridePolicy' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:474)
[2023-05-16 10:15:51,900] INFO Added aliases 'NoneConnectorClientConfigOverridePolicy' and 'None' to plugin 'org.apache.kafka.connect.connector.policy.NoneConnectorClientConfigOverridePolicy' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:474)
[2023-05-16 10:15:51,900] INFO Added aliases 'PrincipalConnectorClientConfigOverridePolicy' and 'Principal' to plugin 'org.apache.kafka.connect.connector.policy.PrincipalConnectorClientConfigOverridePolicy' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:474)
[2023-05-16 10:15:51,960] ERROR Stopping due to error (org.apache.kafka.connect.cli.ConnectDistributed:88)
org.apache.kafka.common.config.ConfigException: Invalid value io.confluent.connect.replicator.util.ByteArrayConverter for configuration key.converter: Class io.confluent.connect.replicator.util.ByteArrayConverter could not be found.
        at org.apache.kafka.common.config.ConfigDef.parseType(ConfigDef.java:766)
        at org.apache.kafka.common.config.ConfigDef.parseValue(ConfigDef.java:512)
        at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:505)
        at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:112)
        at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:132)
        at org.apache.kafka.connect.runtime.WorkerConfig.<init>(WorkerConfig.java:507)
        at org.apache.kafka.connect.runtime.distributed.DistributedConfig.<init>(DistributedConfig.java:568)
        at org.apache.kafka.connect.runtime.distributed.DistributedConfig.<init>(DistributedConfig.java:563)
        at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:97)
        at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:82)
pieter@malaika ~/Downloads/confluent/confluent-7.4.0 $ bin/connect-distributed etc/kafka-connect-replicator/replicator.properties

So

at least zookeeper, kafka & schema registry starts near as immediately when started manually, so it seems the confluent cli script somehow does not work correctly on this system.
I am not certain what to do with the error given by connect and would appreciate a pointer for fixing it.
I use the confluent local service ... environment for dev for confluent.cloud systems, where we do make use of schema registry, ksql & the UI and such. It would be lovely if I could get those to work.
I have 64Gb of RAM and a 12 core CPU. I think it should be sufficient yes.

OneCricketeer · 17 May 2023 19:43

Thanks, yes, most of those seem fine

Class io.confluent.connect.replicator.util.ByteArrayConverter could not be found

For this, you need to use connect-avro-distributed.properties, not replicator.properties… The latter is for using with connect-standalone.sh

for confluent.cloud systems

confluent local CLI is not for the cloud, only localhost. Did you mean ccloud, perhaps?

Your other alternative for local development is Docker Compose, which would be more reproducible across OS’s (I noticed you were using Gentoo, which I don’t have experience with)

gingerale · 15 December 2023 20:41

Hi @pieterbreed, I have the same problem. How did you fix it?

Topic		Replies	Views
Error: ZooKeeper failed to start Kafka Connect	2	505	24 February 2024
Confluent local services start command stuck , no display Command Line tools	0	1100	30 September 2023
Issue in running kafka in wsl Ops	3	3168	20 July 2021
Unable to start zookeeper and kafka service Ops	6	14829	2 October 2021
.confluent/home/etc/kafka/zookeeper.properties: no such file or directory Command Line tools	0	3669	29 July 2021