Will topics still be available while their partitions are being reassigned?

Ops
1

Hi,
lets assume there is a Kafka cluster with 4 Brokers (B1 - B4), and a Kafka Topic (T) with replication factor 2.
At first all replicas of T are on B1 and B2. Then, a partition reassignment is started that moves all of T’s replicas to brokers B3 and B4.

Is that reassignment “transparent” for any consumers and producers working with that topic?
Or is there a sort of down time in which no messages can be written to, or read from the topic?

I guess that moving only some (lets say half) of the partitions to the new brokers would make it so at least the other partitions can still be used by clients, but is that actually true? (or is the whole topic somehow “locked” for the duration of the reassignment?)

Thanks to anyone who can shed a bit of light on this :slight_smile:

1 Like
2

Hi,

did you check the documentation already?
There are some examples listed:
https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-4.ReassignPartitionsTool

and some explanation in stackoverflow.

should be yes :slight_smile:

3

I feel when the reassignment happens , producers may get Topic/Partition not available exception during the reassignment which will be hardly for 5 secs. As producers has the retry mechanism it will be retried and succeds.

4

Kafka will add aditional replicas first until they are in sync. Once they are in sync Kafka will trigger a elect one of the new replicas as a leader. Therefore your in sync replicas will not shrink during the reassignment (unless Kafka is being overwhelmed - for instance because you are not throttling the process).

Accordingly there is no down time to be expected - except for the leader reelection at the end of the reassignment - during which producers will need to wait for a bit (depending on their configured acks).