Cp kafka keep increasing wsl disk space and failing

Hi,
I am using docker desktop with windows 10 with WSL2.

Kafka is starting fine and I can able to use it without any issue. But it keeps taking WSL disk space, even though no messages send to the topic. and the sample test msg also very few kb. It is occupying 250GB in 26hours, then the Kafka container is getting terminated by saying needs more dis space.

But when i run docker system df, it looks like containers are normal and taking very less space. I see the overlay disk space is increasing.

Please advise

I am using the following docker-compose yml.

version: '2.1'

services:
  zoo:
    image: confluentinc/cp-zookeeper:7.0.1
    hostname: zoo
    container_name: zoo
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_SERVER_ID: 1
      ZOOKEEPER_SERVERS: zoo:2888:3888

  kafka:
    image: confluentinc/cp-kafka:7.0.1
    hostname: kafka
    container_name: kafka
    ports:
      - "9092:9092"
      - "9999:9999"
    environment:
      KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_INTERNAL://kafka:19092,LISTENER_DOCKER_EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_DOCKER_INTERNAL:PLAINTEXT,LISTENER_DOCKER_EXTERNAL:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_DOCKER_INTERNAL
      KAFKA_ZOOKEEPER_CONNECT: "zoo:2181"
      KAFKA_BROKER_ID: 1
      KAFKA_LOG4J_LOGGERS: "kafka.controller=ERROR,kafka.producer.async.DefaultEventHandler=ERROR,state.change.logger=ERROR"
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_JMX_PORT: 9999
      KAFKA_JMX_HOSTNAME: ${DOCKER_HOST_IP:-127.0.0.1}
      KAFKA_AUTHORIZER_CLASS_NAME: kafka.security.authorizer.AclAuthorizer
      KAFKA_ALLOW_EVERYONE_IF_NO_ACL_FOUND: "true"
    depends_on:
      - zoo

Hi @shahul.dhasthagir

welcome to the forum :slight_smile:

did you check the size of vhdx files in
C:\<username>\AppData\Local\Docker\wsl?

what does

docker system df

say?

best,
michael

Hi Michael,

Thanks for the prompt reply. Please find the details below

`C:\<username>\AppData\Local\Docker\wsl`? - 244GB

docker system df

TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 19 12 9.179GB 8.019GB (87%)
Containers 13 13 1.136GB 0B (0%)
Local Volumes 27 17 7.078GB 136MB (1%)
Build Cache 154 0 820.8MB 820.8MB

Hi,

Please also note. The disk memory is getting released after i removed the Kafka container (docker rm ) and execute the below script

wsl --shutdown
optimize-vhd -Path .\ext4.vhdx -Mode full

And currently

C:\<username>\AppData\Local\Docker\wsl ? - 19GB

docker system df

TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          19        9         9.179GB   8.508GB (92%)
Containers      10        10        1.157GB   0B (0%)
Local Volumes   27        12        5.806GB   211.2MB (3%)
Build Cache     154       0         820.8MB   820.8MB

hmm strange
seems to be related to wsl, not 100% sure what’s the issue here.

which windows version are you using?

what’s the output of
wsl --list --verbose

best,
michael

Please find the details below

 OS Name	Microsoft Windows 10 Pro
  Version	10.0.19043 Build 19043

wsl --list --verbose

  NAME                   STATE           VERSION
* docker-desktop         Running         2
  docker-desktop-data    Running         2

thanks I see
did you already try with another distro like Ubuntu?

best,
Michael

No, I didn’t try. My use case is to work with the docker desktop.