To fix this issue, I changed the ID of all brokers to 101, 102 and 103 and for all controllers 1, 2 and 3. Then added all the controllers to the quorum list for brokers and started the services like:
KAFKA_CONTROLLER_QUORUM_VOTERS: '1@vskafka-controller-1:9093,2@vskafka-controller-2:9093,3@vskafka-controller-3:9093'
Its started working well now.