Kafka Broker shuts down often

We are running a dev environment on Windows with Kafka Broker and our own Kafka Producer and Consumer implementation using Java.

After running the broker for a day or so, the broker just goes down. Due to logs folder getting filled up. We manage by cleaning up the tmp\kafka-logs folder and then restarting Kafka Broker.

Obviously, this is not an option in production. Please suggest how we could fix this. We are running Kafka on windows.

I googled this and some solutions suggested were using Kafka as a docker container or running it on Linux. We could also have an automatic batch file to periodically restart Kafka/clean up the logs. But that is not a good idea at all in production.

Please suggest if there is any config change or any other solution to solve this.

Hi @aiyer100

welcome to the forum :slight_smile:

did you already check the log files of the broker?

In general it should not be necessary to restart the Kafka broker periodically.

Basically I think a proper planning for you production system is needed.
I would recommend to run the dev (and maybe integration) system with the same config and structure - despite of the hardware sizing the same power might not be needed for dev and int

Furthermore I would recommend also to run the whole Kafka stack on Linux
Perhaps it might be good to get a step back and think about the needs (size, performance,integration) of your Kafka env and the plan accordingly.

Best,
michael