I have a question for scaling down kafka. Our current kafka scaling up strategy involves, adding more brokers and partitions and more consumers to the consumer group when traffic will increase and message production rate increases. But, as far as I know we can increase partition count per topic but we cannot decrease the partitions count because that will create data loss. So, ideally if any one wants to scale down, how should one achieve that ?
It’s not really possible without abstracting the storage somehow. If you remove a broker, then you first need to remove all replicas from it, otherwise you’ll end up increasing your out of sync replicas, causing issues with producer acks and min ISR settings
You can compare how MSK Serverless and Aiven Kafka work, as their services can scale to zero, per my understanding