Rebalancing tasks when new kafka connect is started

Kafka Connect
#1

Hi

Could you help me to understand how rebalancing is working?
I have next scenario:

  1. kafka-connect-1 is started
  2. add connector with 4 tasks via POST /connectors
  3. checking status via GET /connectors/myConnector/status
    i can see my connector and 4 tasks for it
  4. i am starting kafka-connect-2
  5. checking status again and can see that all tasks still on kafka-connect-1
  6. now if i do DELETE /connectors/myConnector/ then POST /connectors
    and check status again then i can see that now 4 tasks are spread between 2 connects

My questions are

  • is there any way to trigger rebalance automatically after new kafka-connect joining cluster?
  • is this del+add connector is correct way to trigger rebalance manually?

p.s.

  • kafka-connect-1 and kafka-connect-2 in the same cluster
  • restapi
#2

Hi @Sirius_Dour , you can find some details in this blog post: https://www.confluent.io/blog/incremental-cooperative-rebalancing-in-kafka/

Can you try adding the configuration property
connect.protocol=eager

and see if that makes it work as you expect?

Keep us posted!

2 Likes
#3

Hi @gianlucanatali

Looks like it is working as i wanted:
i added connect.protocol=eager in my .properties file for kafka-connect-1 and kafka-connect-2
i started kafka-connect-1 (cube) and added connector with 4 tasks on it
then i just started kafka-connect-2 (smart) and checked status
image
i can see that all tasks are rebalanced between these 2 nodes

Thank you!

#4

I have read https://www.confluent.io/blog/incremental-cooperative-rebalancing-in-kafka/
And i have additional questions: so my kafka-connect-1 and kafka-connect-2 are just “workers”. So it mean that case that i described is exact representation of 1. A new worker joins (Figure 1). and it says that during 2-d rebalance some tasks should be moved to this new worker (kafka-connect-2/smart in my case) and other tasks should not be interrupted. But it isn’t happen: without connect.protocol=eager property all tasks remains assigned to 1-st worker (cube).
image
(on the left status from worker-1 on the right status from worker-2)

And this property forces my worker to use Eager Rebalancing. Am i correct?
Why Incremental Cooperative Rebalancing from this Figure 1 is not working?
Will this rebalance trigger tasks revocations and interruptions as result?

I tried to find more details about this property but found only this
https://docs.confluent.io/platform/current/installation/configuration/connect/index.html#connect.protocol

#5

Maybe my test wasn’t very accurate: i realized that there was 3 connectors but i checked status only for one of them.
Now i repeated this test with only 1 connector onboard and without this property
here is the result
image
look like 2 tasks moved to new worker

recheck with all 3 connectors (without property) also shows correct results if check status of each connector:
after worker-2 joined i got next picture:

  • worker-1:
    4x connector-1 + 2x connector-3 = 6 tasks in total
  • worker-2:
    4x connector-2 + 1x connector-3 = 5 tasks in total

it looks balanced (based on amount of on each worker)

recheck with all 3 connectors (with property) shows these results:

  • worker-1:
    2x connector-1 + 2x connector-2 + 1x connector-3 = 5 tasks in total
  • worker-2:
    2x connector-1 + 2x connector-2 + 2x connector-3 = 6 tasks in total

this is also balanced but in more strict way

#6

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.