We would like to have strict control over the topics in our system and therefore prohibit the automatic creation of topics via the broker property auto.create.topics.enable=false.
Unfortunately, this means that the automatic creation of internal Kafka topics, such as __consumer_offsets, no longer seems to work.
Is there a possibility to allow the creation of internal topics by the broker itself? Or do I have to create the topics myself in this case? The problem here, however, is that I don’t know which internal topics need to be available.
Thank you very much for your help.
This seems to be handled gracefully by Kafka 3.9 (KRaft mode). I tested auto.create.topics.enable=false and see that __consumer_offsets will get created for me automatically when I first consume. I see this logged when the topic is needed:
Sent auto-creation request for Set(__consumer_offsets) to the
active controller. (kafka.server.DefaultAutoTopicCreationManager)
I see some logic here that looks like special handling for internal topics though I’m not seeing __consumer_offsets handled directly here. I think that this is the right vicinity to be looking though.
What problematic behavior are you observing? Which version of Kafka and are you using KRaft or ZooKeeper?
It happend on a Kafka 3.9 System with KRaft (two processes, one with process.roles=controller and one with process.roles=broker).
I guess that I should do some testing to validate the behaviour. Should be no big deal with docker in place. Thanks giving me some hints!