slowly making progress on setting up a new cluster, but now I have new questions…
I (want to) run controller and broker on two containers on a each box (total 3 servers → 3 brokers, 3 controllers).
The first steps after bringing up the containers is
create uuid (done9
format storage with “bin/kafka-storage format -t q1Sh-9_ISia_zwGINzRvyQ -c etc/kafka/kraft/controller.properties”
And thats where I am confused again…
Do I need (individual) log.dirs on all 6 containers or only on the three brokers?
The format command asks for a controller.properties file but I don’t use that file at all since I pass in everything via environment variables. Can I/Do I need to create a dummy one to run the format command (either pass one in or just create it temporarily for this)?
In my initial attempt (only having log.dirs in broker, not having run format) it looked like the storage was being used just fine (with Cluster UUID passed in and brokerIDs assigned), so is manual formating even necessary ?
-rw-r–r–. 1 462151 462151 249 Nov 15 14:38 bootstrap.checkpoint
-rw-r–r–. 1 462151 462151 0 Nov 15 14:38 cleaner-offset-checkpoint
drwxr-xr-x. 2 462151 462151 4096 Nov 15 14:59 __cluster_metadata-0
-rw-r–r–. 1 462151 462151 4 Nov 15 14:59 log-start-offset-checkpoint
-rw-r–r–. 1 462151 462151 122 Nov 15 14:38 meta.properties
-rw-r–r–. 1 462151 462151 4 Nov 15 14:59 recovery-point-offset-checkpoint
-rw-r–r–. 1 462151 462151 0 Nov 15 14:38 replication-offset-checkpoint
So the first question was easy to test, providing one directory for each kafka container instance (ie per role) is working just fine.
The controller writes less but similar files so I assume individual directories it is.
I also see that broker and controller create meta.properties files, but my hope that I could use those was in vain:
/bin/kafka-storage format -t UUID -c /data/cpkafka-data/meta.properties Exception in thread "main" org.apache.kafka.common.config.ConfigException: Missing required configuration zookeeper.connect which has no default value. at kafka.server.KafkaConfig.validateValues(KafkaConfig.scala:2299) at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:2290) at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:1638) at kafka.tools.StorageTool$.$anonfun$execute$1(StorageTool.scala:71) at scala.Option.flatMap(Option.scala:283) at kafka.tools.StorageTool$.execute(StorageTool.scala:71) at kafka.tools.StorageTool$.main(StorageTool.scala:52) at kafka.tools.StorageTool.main(StorageTool.scala)
So I still either need to create one manually or don’t if the storage does not need to be formatted in the first place…
Both container (type)s seem to run fine (without topics for now), so it does not look like I am missing a step.
I’ll try to read up on the formatting step.
So I created a single properties file based on the environment parameters and I got
/bin/kafka-storage format -t UUID -c /data/cpkafka-data/server.properties.controller
Log directory /data/cpkafka-data is already formatted. Use --ignore-formatted to ignore this directory and format the others.
That kind of looks like its auto formatting after all.
I mean I am happy with that but confused;)
