Troubleshoot slow kafka writes

Hello. I have 3-node kafka cluster with settings: rf=3, num_partitions=50, min_isr=2. For deploying brokers I use docker image confluentinc/cp-kafka:7.4.0 and confluentinc/cp-zookeeper:7.4.0 on three bare metal servers with SSD.

My application experiences very high latency when producing messages to topic - around 30 ms per message. Application written in java using spring boot 3.2.2 and spring-kafka dependency (org.springframework.kafka:spring-kafka:3.1.1) which transitively uses org.apache.kafka:kafka-clients:3.6.1. Producer uses default settings (except key/value.serializer, ofc), which implies and acks=all.

Producer and brokers sits in same physical gigabit LAN.

Please help to find out what exactly is the problem.

I also export metrics from producer and brokers into prometheus and have ability to build fancy graphs:


This graph uses metric kafka_network_RequestMetrics_999thPercentile{name="LocalTimeMs", request="Produce"}