I have installed Confluent for Kubernetes (CFK). I have 3 replicas of Kafka broker with dataVolumeCapacity: 3Gi. I have configured log.retention.bytes: 1073741824 (1Gi) and log.retention.ms: 7200000 (2 hours) in each Kafka broker but the logs are not being deleted.
So after the 3Gi is reached, I getting this error: java.io.IOException: No space left on device and the pod is in CrashLoopBackOff status. How can I set up the apache broker to delete the logs? The property log.retention.hours has the default value 168 and I not able to change it because it is ready only.
hi @enomoto and welcome 
did you check your setting of
log.cleanup.policy?
some further reading:
best,
michael
Hi @mmuehlbeyer, I have created another cluster with log.retention.hours=1 and log.cleanup.policy=delete but unfortunately the log is not being delete. I have created a pod to get pvc data0-kafka-0 size. As you can see in the screenshot, the size of pvc data0-kafka-0 is 478.7M. I really don’t know which configuration is missing.
This the Kafka image and tag that I’m using: confluentinc/cp-server:7.0.1
hi @enomoto
there also some other factor which influence the deletion of the logfiles and the segments.
there is a nice blog post about this topic at strimzi
best,
michael