Benchmark with confluent platform - local instance

Tools Graphical tools
1

Hi,

I am planning to test / benchmark kafka performance (with Confluent platform) by changing configuration: number of clusters/brokers, replication ration and so on. Can You give me some material / github examples / source code with benchmarking scenario? Is there option to do this on cloud (for free :D)?

2

hey @TinjakM

welcome :slight_smile:

check the following:

https://cwiki.apache.org/confluence/display/KAFKA/Performance+testing

3

@mmuehlbeyer Thank you on response. I have confluent 7.5.2 local confluent platform. When I run for example kafka-topics shell script I can create with replication factor 1, but when I want replication factor 3, I must have 3 brokers. I change bootstrap.servers property in confluent/etc/kafka/producer.properties to localhost:9092,localhost:9093,localhost:9094. But, it gives me error message number of brokers a still one. Is this constraint to this local confluent platform:
sudo sh ./kafka-topics --create --topic test --bootstrap-server localhost:9092,localhost:9093,localhost:9094 -partitions 6 --replication-factor 3
Error while executing topic command : Replication factor: 3 larger than available brokers: 1.

4

hi @TinjakM
you need to start 3 brokers to achieve this.

something like this GitHub - inomera/kafka-3-cluster

best,
michael

5

Thank on last reply, @mmuehlbeyer. Since that, I have established cluster with 3 brokers. I have tested on kafka-producer-test shell script, it works:
./bin/kafka-producer-perf-test.sh --topic test --num-records 5000 --record-size 100 --throughput -1 --producer-props “acks=1” --producer-props “batch.size=8196” --producer-props “bootstrap.servers=localhost:9092,localhost:9093,localhost:9094” --print-metrics
5000 records sent, 7800.312012 records/sec (0.74 MB/sec), 41.72 ms avg latency, 515.00 ms max latency, 42 ms 50th, 71 ms 95th, 73 ms 99th, 75 ms 99.9th.
There is next problem. :smile: How I will collect these data to graph in respect to: record-size, batch-size and so on. From Kafka - performance testing that author has way to export to .csv files.

P.S With execution of producer-test command, i have option --print-metrics but it gives me all in terminal, I have no use from them in this format.