Connect Timeout error when starting tasks post upgrade

Hi,

We have java sink connectors recently upgraded kafka connect from 5.3.1 to 7.3.1.

Post upgrade, we have noticed intermittent Timeout errors when starting the connector tasks, resulting connector failure. Stacktrace below:

Event: [Worker clientId=connect-1, groupId=tst-entitlements-sap-customer-sink-connector] Failed to reconfigure connector’s tasks (tst-entitlements-sap-customer-sink-connector), retrying after backoff:

Stacktrace: org.apache.kafka.connect.runtime.rest.errors.ConnectRestException: IO Error trying to forward REST request: java.net.SocketTimeoutException: Connect Timeout | at org.apache.kafka.connect.runtime.rest.RestClient.httpRequest(RestClient.java:144) | at org.apache.kafka.connect.runtime.distributed.DistributedHerder.lambda$reconfigureConnector$32(DistributedHerder.java:1607) | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) | at java.base/java.lang.Thread.run(Thread.java:829) |Caused by: java.util.concurrent.ExecutionException: java.net.SocketTimeoutException: Connect Timeout | at org.eclipse.jetty.client.util.FutureResponseListener.getResult(FutureResponseListener.java:118) | at org.eclipse.jetty.client.util.FutureResponseListener.get(FutureResponseListener.java:101) | at org.eclipse.jetty.client.HttpRequest.send(HttpRequest.java:732) | at org.apache.kafka.connect.runtime.rest.RestClient.httpRequest(RestClient.java:125) | … 6 more |Caused by: java.net.SocketTimeoutException: Connect Timeout | at org.eclipse.jetty.io.ManagedSelector$Connect.run(ManagedSelector.java:955) | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) | at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) | … 3 more

Source: org.apache.kafka.connect.runtime.distributed.DistributedHerder

Can you please advice why this timeout is happening? Or at least what exactly it’s trying to do that’s timing out?

Many thanks,

Joel