Kafka Connect: Cannot complete request because of a conflicting operation (e.g. worker rebalance)

Kafka Connect
1

Thank you for your help.

I’m running a Kafka Node (computer #1) and Connect (Sink) on a separate machine (computer #2).

On computer #2 the following is up and running:

  • Connect (“connect-avro-distributed.properties”)
  • Schema Registry (“schema-registry.properties”)
  • KSQLDB
  • KSQL CLI
  • MySQL w/CLI

Now, that I have Connect running and acknowledged by the Broker (computer #1) as a member of the group, when I attempt to create a connector in the schema-registry I get the following error in the CLI:

{"error_code":409,"message":"Cannot complete request because of a conflicting operation (e.g. worker rebalance)"}

The following message occurs in the Connect log:

INFO AbstractConfig values: 
(org.apache.kafka.common.config.AbstractConfig:372)

I found @rmoff blog on the topic:

So, I altered the rest.advertised.host.name in my connect-avro-distributed.properties (see below). I’m still getting the error.

  • Did I alter the correct properties file (connect-avro-distributed.properties)?
  • Did I misunderstand? Is the rest.advertised.host.name suppose to be the IP Address of computer #2 (Connect)? See below properties.
  • I’m wondering if the conflict is with a configuration on the Broker (computer #1)?

The Following are my configurations for computer #2 properties:

---- Computer #2 Properties (CONNECT) -----

***** schema-registry.properties *****

listeners=http://0.0.0.0:8081

kafkastore.connection.url=ip.address.computer2:2181

# kafkastore.bootstrap.servers=ip.address.computer2:9092 (COMMENTED OUT)

kafkastore.topic=_schemas

debug=false

***** connect-avro-distributed.properties *****

bootstrap.servers=ip.address.computer1:9092

group.id=connect-cluster

key.converter=io.confluent.connect.avro.AvroConverter
key.converter.schema.registry.url=http://localhost:8081
value.converter=io.confluent.connect.avro.AvroConverter
value.converter.schema.registry.url=http://localhost:8081

config.storage.topic=connect-configs
offset.storage.topic=connect-offsets
status.storage.topic=connect-statuses

config.storage.replication.factor=1
offset.storage.replication.factor=1
status.storage.replication.factor=1

internal.key.converter=org.apache.kafka.connect.json.JsonConverter
internal.value.converter=org.apache.kafka.connect.json.JsonConverter
internal.key.converter.schemas.enable=false
internal.value.converter.schemas.enable=false

rest.host.name=ip.address.computer2
rest.port=8083

rest.advertised.host.name=ip.address.computer2
rest.advertised.port=8083

plugin.path=share/java,…

***** Connector Configuration *****

sudo curl -X PUT http://localhost:8083/connectors/sink-jdbc-mysql-01/config \
     -H "Content-Type: application/json" -d '{
    "connector.class"                    : "io.confluent.connect.jdbc.JdbcSinkConnector",
    "connection.url"                     : "jdbc:mysql://localhost:3306/test",
    "topics"                             : "demo_altered",
    "key.converter"                      : "org.apache.kafka.connect.storage.StringConverter",
    "connection.user"                    : “username”,
    "connection.password"                : “password”,
    "auto.create"                        : true,
    "auto.evolve"                        : true,
    "insert.mode"                        : "upsert",
    "pk.mode"                            : "record_key",
    "pk.fields"                          : "MESSAGE_KEY"
}'

Thank you for your help.

2

Hi! A few questions to try and clarify what’s happening here:

Are you running Kafka Connect on the first machine too?

What’s the full command you’re using to launch Kafka Connect? The config file to alter is the one that you reference when you launch the Kafka Connect worker.

BTW re.

I think you mean the Kafka Connect worker, right?

This looks like you’ve changed your Schema Registry to point to ZooKeeper on the second computer, instead of Kafka broker on the first - is that deliberate?

3

@rmoff
Thank you for your help.

Are you running Kafka Connect on the first machine too?

Yes, I had Kafka Connect running on the first machine; It was only for setting up the original environment on machine #1.

I stopped Connect on machine #1, and that appears to have cleared up the original issue; however, the change appears to have revealed another issue. When I create the worker I get the following error:

curl: (7) Failed to connect to localhost port 8083: Connection refused

  • Is this a config issue (i.e. “connect-avro-distributed.properties”)?

rest.host.name=ip.address.computer2
rest.port=8083

rest.advertised.host.name=ip.address.computer2
rest.advertised.port=8083

I tried commenting out the “rest.host.name” and “rest.port”. That allows me to create the worker, but ksql “show connectors;” shows the “WARNING (0/1 Tasks Running)”.

  • Is running Kafka Connect on the same machine with the Broker a “no-no”? Or does it require a different configuration to do so?

What’s the full command you’re using to launch Kafka Connect? The config file to alter is the one that you reference when you launch the Kafka Connect worker.

bin/connect-distributed etc/schema-registry/connect-avro-distributed.properties

So I’m good there.

I think you mean the Kafka Connect worker, right?

Yes, Kafka Connect worker. Sorry about that.

This looks like you’ve changed your Schema Registry to point to ZooKeeper on the second computer, instead of Kafka broker on the first - is that deliberate?

Sorry about that… that’s a typo on my part in the post… the properties file actually has the following:

kafkastore.connection.url=ip.address.computer1:2181

4

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.