Tried several ways of resetting a Kafka topic:
A) briefly setting retention to 1 millisecond - this doesn’t work well until logs are shrunk which by default is something like 60 hours and checking every 5 seconds is very expensive
B) delete the topic and create topic again with the same name.
There are two ways of doing that.
First I tried deleting the topic and then checking broker metadata periodically until it doesn’t show the topic anymore. That doesn’t work well with multiple brokers because for some reason it takes awhile for other brokers to update the metadata and as the result create topic fails saying topic already exists
The other way is to delete and continuously try to create the topic until it stops failing with “topic already exists”. This worked for us 99% of the time and is relatively fast (500ms - 5s). However it has recently become less reliable as for some reason one of the brokers doesn’t let go of the deleted topic.
In general, what’s the best way to reset topics?
In particular, what could be a reason for topic to take over 15 minutes to get deleted? I rebooted all Kafka nodes and zookeeper nodes but it didn’t help.
Hi @1stSolo - the second approach of deleting and re-creating a topic will probably be the way to go out of the options you’ve presented here.
Another approach would be to version your topic names i.e.topic_name_v1, then when you want to change, simply create topic_name_v2 and point your clients at the new topic. Then afterward you can delete the previous topic and don’t have to worry about the time it takes to complete.
What would be the reason for one of the brokers to hold on to the deleted topic for so long? How can I speed it up?
I tried isolating the issue and instead of using kafka api client decided to try in Control Center instead.
- create a topic “Test1”, 10 partitions
- delete topic “Test1”.
- try to create topic “Test1”, 10 partitions - get “topic Test1 already exists” error.
I have
delete.topics.enabled=true,
log.segment.delete.delay.ms=1000
file.delete.delay.ms=1000
The topic appears to go away after being deleted in Control Center but creating it again fails.
What other properties or other configuration should I change? Seems like the only way to clear it up is to restart the brokers and zookeeper.
BTW Zookeeper is on another instance than kafka.
Also when I log into ZK pod and do ls it still shows the deleted topic in ZK instance. Restarting ZK seems to release it. Is there some configuration in kafka or ZK that would fix this?