"Ghost" tasks remaining after deleting connectors

Kafka Connect
1

We’ve noticed some issues with some of our JDBC connectors. It looks like tasks are still running even though their associated connectors have been deleted. When looking at our connect-offsets topic, we’re seeing offsets being committed for connectors that no longer exist. We also looked at the JMX metrics and saw the task metrics still existed.
This is causing issues with duplicate messages getting submitted to our topics, because some of these connectors had similar queries.
A restart of the entire Connect cluster got rid of all the extra tasks as only existing connectors got their tasks restarted, but we still need to know how this happened and an easier way to fix it if it does happen again.

2

A few questions: :slight_smile:

  • How many workers do you have in your cluster?
  • Do you have other workers running that are maybe using the same internal topics?
  • How are you deleting the connectors?
3
  • We have two workers that are running in a single distributed cluster.
  • No other clusters exist to be using the same internal topics.
  • We use a GUI that integrates with Connect’s REST API.
4

What happens if you make the DELETE call directly to the REST API?

After running the DELETE, what’s the output of this - does it include the connectors?

curl -s "http://localhost:8083/connectors?expand=info&expand=status
5

I was able to recreate it. I created two new connectors, let them run for a little bit, then deleted them. One I deleted through our GUI, the other I directly called the REST API. The JMX metrics are currently showing both tasks as still running, but the connectors are gone.
Calling the connectors endpoint of the REST API as you suggested does not show the connectors I deleted.

6

I just noticed that now I’m seeing duplicates in other connector-populated topics now, from connectors I have not deleted or touched. This only started after I deleted those two connectors earlier.

When I’m looking in JMX, it looks like the tasks for connectors are running on both workers at once instead of only one.

7

Hi, do you have any further suggestions on how I might be able to diagnose this issue?

8

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.