Hello. I have 3-node kafka cluster with settings: rf=3, num_partitions=50, min_isr=2. For deploying brokers I use docker image confluentinc/cp-kafka:7.4.0 and confluentinc/cp-zookeeper:7.4.0 on three bare metal servers with SSD.
My application experiences very high latency when producing messages to topic - around 30 ms per message. Application written in java using spring boot 3.2.2 and spring-kafka dependency (org.springframework.kafka:spring-kafka:3.1.1) which transitively uses org.apache.kafka:kafka-clients:3.6.1. Producer uses default settings (except key/value.serializer, ofc), which implies linger.ms=0 and acks=all.
Producer and brokers sits in same physical gigabit LAN.
Please help to find out what exactly is the problem.
I also export metrics from producer and brokers into prometheus and have ability to build fancy graphs:
