i have 3 separate kubernetes clusters ( 1 in first dc, 2 in second dc, 3 in third ) and 3 separate kafka clusters ( 1 in first dc, 2 in second dc, 3 in third dc ) same as kubernetes
so in normal mode our applications works on Kubernetes in 1 dc and connect to kafka in first dc, and in case of datacenter failure, we provision services on Kubernetes which working in 2 or 3 dc and connect to kafka cluster in 2 or 3 dc
This 3 datacenter locates in the same city but on different buildings and avg latency between dc is 0.400 ms with very rare spikes up to 1,8ms, so i think of it like AZ in AWS
What the best way to build high availability,high speed kafka cluster in multi-dc mode which can work even if any datacenter going offline? i think about using broker.rack feature and deploy kafka brokers in this fashion
DC1- kafka-01
DC2- kafka-02
DC3- kafka-03
then use replication factor =2, so even any kafka ( or dc ) goes offline all data will be here and cluster remains working
Or it’s better to use Apache Kafka MirrorMaker and have a separately cluster in each DC? and in case of failure just connect to another kafka cluster