Source connector InterruptedException

Kafka Connect
1

Hi all, I put this question on the mailing list also, so sorry for any spam.

When my source connector has some exception when polling should I catch it and throw an InterruptedException?

The connector source task API has the following method:
public abstract List<SourceRecord> poll() throws InterruptedException;

However it seems to me that every time the connector throws the InterruptedException the connector task goes to unassigned state and stays there…

my source task implementation is something like this:

@Override
public List<SourceRecord> poll() throws InterruptedException {
  try {
         return getSourceRecords();
        } catch (SomeException e) {
          throw new InterruptedException(e.getMessage());
        }
}

Is this the correct way of doing this? I see nothing on documentation regarding this.

2

The InterruptedException exception is in the method signature because it is another thread that effectively stops the task and hence an interrupted exception might occur. Ideally, the poll() method should catch any errors that occur during each execution and always return to the caller — whether to indicate that new records have to be written to Kafka or just null to indicate that nothing needs to be done.

Regardless, the method should always return successfully so the runtime can decide whether another poll() call should execute :+1:t2:

3 Likes
3

Cool, thank you for answering.

You said it should return null, shouldn’t it return an empty list instead?

4

Returning null or an empty list of SourceRecord will indicate the same thing to the runtime. With this rule in mind, as a performance best practice, it is always better to avoid creating unnecessary overhead on-heap to minimize the GC cycle when they occur :wink:

1 Like
5

Thank you,

Yes, I was wandering if the connect deal with null.

6

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.