Slow performance on kafka

how is your people performance on kafka ?

i have sometimes a high ms on producer on apps like 5-10 sec!

this isi internal benchmark

im running 3.10 kraft mode k8s 3 node cluster

kafka-producer-perf-test.sh --topic test --num-records 1000000 \
--throughput -1 --producer-props bootstrap.servers=localhost:9092 \ batch.size=1000 acks=1 linger.ms=100000 buffer.memory=4294967296 \ compression.type=text request.timeout.ms=300000 --record-size 1000
15825 records sent, 3163.7 records/sec (3.02 MB/sec), 2167.5 ms avg latency, 3632.0 ms max latency.
15872 records sent, 3159.9 records/sec (3.01 MB/sec), 5220.2 ms avg latency, 8348.0 ms max latency.
50704 records sent, 10130.7 records/sec (9.66 MB/sec), 10231.6 ms avg latency, 12277.0 ms max latency.
38640 records sent, 7728.0 records/sec (7.37 MB/sec), 14150.6 ms avg latency, 16483.0 ms max latency.
53936 records sent, 10761.4 records/sec (10.26 MB/sec), 17935.6 ms avg latency, 20375.0 ms max latency.
59504 records sent, 11877.0 records/sec (11.33 MB/sec), 21704.8 ms avg latency, 24171.0 ms max latency.
48016 records sent, 8287.2 records/sec (7.90 MB/sec), 25574.3 ms avg latency, 28991.0 ms max latency.
66736 records sent, 13347.2 records/sec (12.73 MB/sec), 31376.3 ms avg latency, 32609.0 ms max latency.
108944 records sent, 17183.6 records/sec (16.39 MB/sec), 33180.1 ms avg latency, 36541.0 ms max latency.
110240 records sent, 22048.0 records/sec (21.03 MB/sec), 38206.1 ms avg latency, 39271.0 ms max latency.
229248 records sent, 45849.6 records/sec (43.73 MB/sec), 39345.1 ms avg latency, 39520.0 ms max latency.
63360 records sent, 11212.2 records/sec (10.69 MB/sec), 39541.4 ms avg latency, 43723.0 ms max latency.
1000000 records sent, 15269.040493 records/sec (14.56 MB/sec), 32191.74 ms avg latency, 43900.00 ms max latency, 37720 ms 50th, 43774 ms 95th, 43889 ms 99th, 43897 ms 99.9th.

Wow, this is an very low batch.size a very high buffer.memory and ludicrous linger.ms. The latter setting basically allows to wait up to 100 seconds until a batch is full and allocates 4 GB Ram to buffer messages in the meantime. To reduce the maximal latency reducing linger.ms to something like 50 should help a lot.

The thing that is still puzzling me a bit however is how it can take up to 40 seconds to produce a batch with only 1000 bytes batch.size. How much RAM does your machine have? Is it swapping maybe?

I would expect much better performance with default parameters and then even more throughput but higher latency with the ones in recommended in How to optimize your Kafka producer for throughput using Confluent

If you try it, I would be happy about an update. :slight_smile:

1 Like