Tried several ways of resetting a Kafka topic:
A) briefly setting retention to 1 millisecond - this doesn’t work well until logs are shrunk which by default is something like 60 hours and checking every 5 seconds is very expensive
B) delete the topic and create topic again with the same name.
There are two ways of doing that.
First I tried deleting the topic and then checking broker metadata periodically until it doesn’t show the topic anymore. That doesn’t work well with multiple brokers because for some reason it takes awhile for other brokers to update the metadata and as the result create topic fails saying topic already exists
The other way is to delete and continuously try to create the topic until it stops failing with “topic already exists”. This worked for us 99% of the time and is relatively fast (500ms - 5s). However it has recently become less reliable as for some reason one of the brokers doesn’t let go of the deleted topic.
In general, what’s the best way to reset topics?
In particular, what could be a reason for topic to take over 15 minutes to get deleted? I rebooted all Kafka nodes and zookeeper nodes but it didn’t help.