Hi Team,
We are running the producer-perf-test with throughput 60000,I could see latency spikes in between,Can you please let us know the cause of it.
I am running kafka cluster with 3 brokers(each broker with 5 cpu,57 GB RAM, heap allocated with 24 GB RAM).
Kafka topic created with 15 partition.cpu,memory,disks are normal.
kafka-producer-perf-test.sh --topic TEST --num-records 300000000 --throughput 60000 --producer-props bootstrap.servers=kafka-0.kafka-headless.kafka.svc.cluster.local:9092 batch.size=1000 acks=1 linger.ms=100000 request.timeout.ms=300000 --record-size 1000
254377 records sent, 50875.4 records/sec (48.52 MB/sec), 609.9 ms avg latency, 2761.0 ms max latency.
241437 records sent, 48287.4 records/sec (46.05 MB/sec), 635.3 ms avg latency, 2847.0 ms max latency.
252584 records sent, 50516.8 records/sec (48.18 MB/sec), 617.1 ms avg latency, 2828.0 ms max latency.
253213 records sent, 50642.6 records/sec (48.30 MB/sec), 609.9 ms avg latency, 2676.0 ms max latency.
244751 records sent, 48950.2 records/sec (46.68 MB/sec), 628.6 ms avg latency, 2745.0 ms max latency.
251521 records sent, 50304.2 records/sec (47.97 MB/sec), 612.5 ms avg latency, 2788.0 ms max latency.
252985 records sent, 50597.0 records/sec (48.25 MB/sec), 613.8 ms avg latency, 2669.0 ms max latency.
245492 records sent, 49098.4 records/sec (46.82 MB/sec), 628.1 ms avg latency, 2716.0 ms max latency.
248181 records sent, 49636.2 records/sec (47.34 MB/sec), 623.0 ms avg latency, 2696.0 ms max latency.
250278 records sent, 50055.6 records/sec (47.74 MB/sec), 22.2 ms avg latency, 216.0 ms max latency.
247574 records sent, 49514.8 records/sec (47.22 MB/sec), 21.6 ms avg latency, 209.0 ms max latency.
251750 records sent, 50350.0 records/sec (48.02 MB/sec), 35.3 ms avg latency, 210.0 ms max latency.
250285 records sent, 50057.0 records/sec (47.74 MB/sec), 20.8 ms avg latency, 168.0 ms max latency.
248663 records sent, 49732.6 records/sec (47.43 MB/sec), 19.9 ms avg latency, 156.0 ms max latency.
250760 records sent, 50152.0 records/sec (47.83 MB/sec), 31.1 ms avg latency, 138.0 ms max latency.
250925 records sent, 50185.0 records/sec (47.86 MB/sec), 30.1 ms avg latency, 152.0 ms max latency.
248241 records sent, 49648.2 records/sec (47.35 MB/sec), 22.0 ms avg latency, 168.0 ms max latency.
249948 records sent, 49989.6 records/sec (47.67 MB/sec), 48.9 ms avg latency, 207.0 ms max latency.
251635 records sent, 50327.0 records/sec (48.00 MB/sec), 23.1 ms avg latency, 184.0 ms max latency.
249459 records sent, 49891.8 records/sec (47.58 MB/sec), 18.1 ms avg latency, 135.0 ms max latency.
249519 records sent, 49903.8 records/sec (47.59 MB/sec), 119.3 ms avg latency, 2600.0 ms max latency.
253220 records sent, 50644.0 records/sec (48.30 MB/sec), 152.2 ms avg latency, 2463.0 ms max latency.
242689 records sent, 48537.8 records/sec (46.29 MB/sec), 233.1 ms avg latency, 2571.0 ms max latency.
247993 records sent, 49598.6 records/sec (47.30 MB/sec), 325.9 ms avg latency, 2569.0 ms max latency.
253490 records sent, 50698.0 records/sec (48.35 MB/sec), 612.4 ms avg latency, 2448.0 ms max latency.