# Kafka Broker -java.io.IOException: No space left on device

**URL:** <https://forum.confluent.io/t/kafka-broker-java-io-ioexception-no-space-left-on-device/4837>\
**Category:** Containers\
**Created:** [21 April 2022 20:04 UTC](https://forum.confluent.io/t/kafka-broker-java-io-ioexception-no-space-left-on-device/4837 "2022-04-21T20:04:33Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![enomoto](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/enomoto/32/1834_2.png) [@enomoto](https://forum.confluent.io/u/enomoto)\
**Post date:** [21 April 2022 20:04 UTC](https://forum.confluent.io/t/kafka-broker-java-io-ioexception-no-space-left-on-device/4837/1 "2022-04-21T20:04:33Z")

</div>

I have installed Confluent for Kubernetes (CFK). I have 3 replicas of Kafka broker with dataVolumeCapacity: 3Gi. I have configured log.retention.bytes: 1073741824 (1Gi) and log.retention.ms: 7200000 (2 hours) in each Kafka broker but the logs are not being deleted.  
So after the 3Gi is reached, I getting this error: java.io.IOException: No space left on device and the pod is in CrashLoopBackOff status. How can I set up the apache broker to delete the logs? The property log.retention.hours has the default value 168 and I not able to change it because it is ready only.

 ![Screen Shot 2022-04-21 at 17.57.30](https://us1.discourse-cdn.com/flex019/uploads/confluentcommunity/original/2X/9/906d8edd8d4c22a6cbd709cac8d8fc941baed2c0.jpeg)

---

<div class="post-metadata">

**Author:** ![mmuehlbeyer](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/mmuehlbeyer/32/1088_2.png) [@mmuehlbeyer](https://forum.confluent.io/u/mmuehlbeyer)\
**Post date:** [22 April 2022 09:06 UTC](https://forum.confluent.io/t/kafka-broker-java-io-ioexception-no-space-left-on-device/4837/2 "2022-04-22T09:06:55Z")

</div>

hi @enomoto and welcome 🙂

did you check your setting of  
log.cleanup.policy?

some further reading:

> **[Kafka Log Retention and Cleanup Policies](https://medium.com/@sunny_81705/kafka-log-retention-and-cleanup-policies-c8d9cb7e09f8)**
>
> Apache Kafka provides two types of Retention Policies.

best,  
michael

---

<div class="post-metadata">

**Author:** ![enomoto](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/enomoto/32/1834_2.png) [@enomoto](https://forum.confluent.io/u/enomoto)\
**Post date:** [23 April 2022 01:43 UTC](https://forum.confluent.io/t/kafka-broker-java-io-ioexception-no-space-left-on-device/4837/3 "2022-04-23T01:43:02Z")

</div>

Hi @mmuehlbeyer, I have created another cluster with log.retention.hours=1 and log.cleanup.policy=delete but unfortunately the log is not being delete. I have created a pod to get pvc data0-kafka-0 size. As you can see in the screenshot, the size of pvc data0-kafka-0 is 478.7M. I really don’t know which configuration is missing.

This the Kafka image and tag that I’m using: **confluentinc/cp-server:7.0.1**

 ![Screen Shot 2022-04-22 at 22.38.02](https://us1.discourse-cdn.com/flex019/uploads/confluentcommunity/original/2X/1/105e12c21cafd7b044d57be4d54520483862006a.jpeg)

---

<div class="post-metadata">

**Author:** ![mmuehlbeyer](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/mmuehlbeyer/32/1088_2.png) [@mmuehlbeyer](https://forum.confluent.io/u/mmuehlbeyer)\
**Post date:** [25 April 2022 07:45 UTC](https://forum.confluent.io/t/kafka-broker-java-io-ioexception-no-space-left-on-device/4837/4 "2022-04-25T07:45:42Z")

</div>

hi @enomoto

there also some other factor which influence the deletion of the logfiles and the segments.

there is a nice blog post about this topic at strimzi

> **[Deep dive into Apache Kafka storage internals: segments, rolling and retention](https://strimzi.io/blog/2021/12/17/kafka-segment-retention/)**
>
> Strimzi provides a way to run an Apache Kafka cluster on Kubernetes in various deployment configurations.

best,  
michael