Kafka Connect Failed Removal

Kafka Connect
1

I am running a kafka connect cluster where I am frequently creating and removing connectors through the REST api.
During the deletion flow I first delete the connector and then right away delete the topic in the kafka cluster.
From what I see connector tasks don’t seem to be deleted and keeping running sending constantly logs that the topic doesn’t exist ( Received unknown topic or partition error in fetch for partition…).
I took a look at the connector logs at around the time I send the deletion request and I see the following:

 [2021-06-28T02:06:46.430+03:00] Successfully processed removal of connector '4_8WnI12yxKp2itGksZuuP6A'
[2021-06-28T02:06:46.431+03:00][Worker clientId=connect-1, groupId=connect-cluster] Connector 4_8WnI12yxKp2itGksZuuP6A config removed
[2021-06-28T02:06:46.455+03:00]Successfully processed removal of connector '4_8WnI12yxKp2itGksZuuP6A'
[2021-06-28T02:06:46.431+03:00][Worker clientId=connect-1, groupId=connect-cluster] Connector 4_8WnI12yxKp2itGksZuuP6A config removed
[2021-06-28T02:06:46.455+03:00][Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group
[2021-06-28T02:06:46.455+03:00][Worker clientId=connect-1, groupId=connect-cluster] Rebalance started
[2021-06-28T02:06:46.457+03:00][Worker clientId=connect-1, groupId=connect-cluster] Rebalance started
[2021-06-28T02:06:46.457+03:00][Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group
[2021-06-28T02:06:46.462+03:00][Consumer clientId=connector-consumer-4_8WnI12yxKp2itGksZuuP6A-8, groupId=connect-4_8WnI12yxKp2itGksZuuP6A] Received unknown topic or partition error in fetch for partition 4_8WnI12yxKp2itGksZuuP6A-8
[2021-06-28T02:06:46.463+03:00][Consumer clientId=connector-consumer-4_8WnI12yxKp2itGksZuuP6A-8, groupId=connect-4_8WnI12yxKp2itGksZuuP6A] Received unknown topic or partition error in fetch for partition 4_8WnI12yxKp2itGksZuuP6A-8

It is unclear to me why after the removal the tasks return and keep running.
Since we are deleting connectors frequently we are getting a lot of orphan tasks and our logs are filled with logs that the topic is missing (auto create topic is set to false in our kafka cluster)
Is there something that I can configure so the deletions will succeed?

2

@Jed I’m wondering if you could try waiting for confirmation of the Connector delete before proceeding with the topic removal. I’m not very familiar with the code, but I wouldn’t be surprised if there was a bit of a delay in the removal of the consumer from the time the DEL HTTP call is processed. The log message Successfully processed removal of connector, might be misleading in this case. Maybe you could try confirming the removal on a polling loop on the GET /connectors/{name} URL before proceeding with the topic deletion.

https://docs.confluent.io/platform/current/connect/references/restapi.html#connectors

3

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.