Performance Issues with confluent kafka vs older cluster

mmuehlbeyer · 17 February 2025 12:50

need to check this by myself which metrics are useful to detect that issue

is it possible for you to share the config you’re running?
I’d like to test a bit by myself.

as said seems to be network related between the brokers

Rand · 17 February 2025 13:55

Config:
To reiterate, we run RHEL 801, selinux, local fw are on.
Containers run with podman under a dedicated user with proper permissions and ulimits.
Containers run as app_user (hardcoded in container) [vs old Kafka/ZK Combo which runs root internally)

These are the startup commands:

controller

    KAFKA_TYPE="controller"
    KAFKA_INSTANCE="${KAFKA_TYPE}-${node_number}"
    KAFKA_LOG_PATH="$basedir/data/cpkafka-${KAFKA_TYPE}_logs"
    KAFKA_LOG="${KAFKA_LOG_PATH}/${KAFKA_INSTANCE}.json"
    podman unshare chown 1000:1000 -R $KAFKA_LOG_PATH
    podman unshare chown 1000:1000 -R $basedir/data/cpkafka-${KAFKA_TYPE}
    podman unshare chown 1000:1000 -R $basedir/monitor

    podman run -d \
    --name $KAFKA_INSTANCE \
    -h=host0 \
    -p 9092:9092 \
    -p 8082:8091 \
    -v $basedir/data/cpkafka-controller:/data/cpkafka-data:Z \
    -v $basedir/keystore:/etc/kafka/secrets:z \
    -v $basedir/monitor:/monitor:z \
    -e KAFKA_NODE_ID=1 \
    -e KAFKA_PROCESS_ROLES='controller' \
    -e KAFKA_LISTENER_SECURITY_PROTOCOL_MAP='CONTROLLER:SSL, SSLINTERNAL:SSL' \
    -e KAFKA_CONTROLLER_QUORUM_VOTERS='1@host0:9092,2@host1:9092,3@host2:9092' \
    -e KAFKA_INTER_BROKER_LISTENER_NAME='SSLINTERNAL' \
    -e KAFKA_CONTROLLER_LISTENER_NAMES='CONTROLLER' \
    -e KAFKA_LISTENERS='CONTROLLER://:9092' \
    -e CLUSTER_ID='CID' \
    -e KAFKA_LISTENER_NAME_CONTROLLER_SSL_TRUSTSTORE_LOCATION='/etc/kafka/secrets/tsi.pfx' \
    -e KAFKA_LISTENER_NAME_CONTROLLER_SSL_TRUSTSTORE_PASSWORD='xyz' \
    -e KAFKA_LISTENER_NAME_CONTROLLER_SSL_TRUSTSTORE_TYPE='PKCS12' \
    -e KAFKA_LISTENER_NAME_CONTROLLER_SSL_KEYSTORE_LOCATION='/etc/kafka/secrets/host0.pfx' \
    -e KAFKA_LISTENER_NAME_CONTROLLER_SSL_KEYSTORE_PASSWORD='xyz' \
    -e KAFKA_LISTENER_NAME_CONTROLLER_SSL_KEYSTORE_TYPE='PKCS12' \
    -e KAFKA_LISTENER_NAME_CONTROLLER_SSL_CLIENT_AUTH='required' \
    -e KAFKA_SSL_KEYSTORE_FILENAME='host0.pfx' \
    -e KAFKA_SSL_KEYSTORE_CREDENTIALS='keystore.crd' \
    -e KAFKA_SSL_KEY_CREDENTIALS='key.crd' \
    -e KAFKA_SSL_TRUSTSTORE_LOCATION='/etc/kafka/secrets/tsi.pfx' \
    -e KAFKA_SSL_TRUSTSTORE_CREDENTIALS='truststore.crd' \
    -e KAFKA_SSL_CLIENT_AUTH='required' \
    -e KAFKA_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM="" \
    -e KAFKA_ssl_keyStore_type='PKCS12' \
    -e KAFKA_ssl_trustStore_type='PKCS12' \
    -e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 \
    -e KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS=0 \
    -e KAFKA_TRANSACTION_STATE_LOG_MIN_ISR=1 \
    -e KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR=1 \
    -e KAFKA_JMX_PORT=18091 \
    -e KAFKA_JMX_HOSTNAME=${KAFKA_INSTANCE} \
    -e KAFKA_OPTS="-javaagent:/monitor/jmx_prometheus_javaagent-1.1.0.jar=8091:/monitor/${KAFKA_TYPE}_config.yml" \
    -e KAFKA_LOG4J_ROOT_LOGLEVEL='WARN' \
    -e KAFKA_LOG4J_TOOLS_LOGLEVEL='WARN' \
    -e KAFKA_LOG4J_LOGGERS="kafka=WARN,kafka.controller=WARN,kafka.log.LogCleaner=WARN,state.change.logger=WARN,kafka.producer.async.DefaultEventHandler=WARN" \
    --log-driver=k8s-file \
    --log-opt=path=${KAFKA_LOG} \
    --log-opt=max-size=100mb \
     $kafka_image

broker

    KAFKA_TYPE="broker"
    KAFKA_INSTANCE="${KAFKA_TYPE}-${node_number}"
    KAFKA_LOG_PATH="$basedir/data/cpkafka-${KAFKA_TYPE}_logs"
    KAFKA_LOG="${KAFKA_LOG_PATH}/${KAFKA_INSTANCE}.json"
    podman unshare chown 1000:1000 -R $KAFKA_LOG_PATH
    podman unshare chown 1000:1000 -R $basedir/data/cpkafka-${KAFKA_TYPE}
    podman unshare chown 1000:1000 -R $basedir/monitor

    podman run -d  --restart always \
    --name $KAFKA_INSTANCE \
    -h=host0 \
    -p 9093:9093 \
    -p 9094:9094 \
    -p 9095:9095 \
    -p 8081:8091 \
    -v $basedir/data/cpkafka-broker:/data/cpkafka-data:Z \
    -v $basedir/keystore:/etc/kafka/secrets:z \
    -v $basedir/monitor:/monitor:z \
    -e KAFKA_NODE_ID=4 \
    -e KAFKA_PROCESS_ROLES='broker' \
    -e KAFKA_LISTENER_SECURITY_PROTOCOL_MAP='SSL:SSL,CONTROLLER:SSL,SSLINTERNAL:SSL, PLAIN:PLAINTEXT' \
    -e KAFKA_LISTENERS='SSLINTERNAL://:9093,SSL://:9095,PLAIN://:9094' \
    -e KAFKA_CONTROLLER_QUORUM_VOTERS='1@host0:9092,2@host1:9092,3@host2:9092' \
    -e KAFKA_INTER_BROKER_LISTENER_NAME='SSLINTERNAL' \
    -e KAFKA_SECURITY_PROTOCOL='SSL' \
    -e KAFKA_ADVERTISED_LISTENERS='SSLINTERNAL://host0:9093,SSL://:9095,PLAIN://:9094' \
    -e KAFKA_CONTROLLER_LISTENER_NAMES='CONTROLLER' \
    -e CLUSTER_ID='CID' \
    -e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 \
    -e KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS=0 \
    -e KAFKA_TRANSACTION_STATE_LOG_MIN_ISR=1 \
    -e KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR=1 \
    -e KAFKA_LOG_DIRS='/data/cpkafka-data' \
    -e KAFKA_SSL_KEYSTORE_FILENAME='host0.pfx' \
    -e KAFKA_SSL_KEYSTORE_CREDENTIALS='keystore.crd' \
    -e KAFKA_SSL_KEY_CREDENTIALS='key.crd' \
    -e KAFKA_SSL_TRUSTSTORE_FILENAME='tsi.pfx' \
    -e KAFKA_SSL_TRUSTSTORE_CREDENTIALS='truststore.crd' \
    -e KAFKA_SSL_CLIENT_AUTH='required' \
    -e KAFKA_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM="" \
    -e KAFKA_ssl_keyStore_type='PKCS12' \
    -e KAFKA_ssl_trustStore_type='PKCS12' \
    -e KAFKA_JMX_PORT=18091 \
    -e KAFKA_JMX_HOSTNAME=${KAFKA_INSTANCE} \
    -e KAFKA_OPTS="-javaagent:/monitor/jmx_prometheus_javaagent-1.1.0.jar=8091:/monitor/${KAFKA_TYPE}_config.yml" \
    -e KAFKA_LOG4J_ROOT_LOGLEVEL='WARN' \
    -e KAFKA_LOG4J_TOOLS_LOGLEVEL='WARN' \
    -e KAFKA_LOG4J_LOGGERS="kafka=WARN,kafka.controller=WARN,kafka.log.LogCleaner=WARN,state.change.logger=WARN,kafka.producer.async.DefaultEventHandler=WARN" \
    --log-driver=k8s-file \
    --log-opt=path=${KAFKA_LOG} \
    --log-opt=max-size=100mb \
     $kafka_image

Network -
The actual network is identical between old and new, so it cannot be network itself.
The only thing different (besides the actual image, Kafka version and KRaft vs ZK) is that

Your image runs as app_user (instead of root), but I checked ulimits and provide proper access to files via unshare command
I run with dedicated ports (-p 9092:9092) instead of allowing everything

> CONTAINER ID  IMAGE                                                       COMMAND               CREATED     STATUS      PORTS                                 NAMES
> 5aad61137c9f  localrepo/humio/zookeeper:latest        /bin/sh /run.sh       3 days ago  Up 3 days                                                             zookeeper-0
> 4d058b0181c5  localrepo/humio/kafka:latest            /bin/sh /run.sh       3 days ago  Up 3 days                                                             kafka-0
> 365d986658b6  localrepo/confluentinc/cp-kafka:latest  /etc/confluent/do...  3 days ago  Up 3 days   0.0.0.0:8082->8091/tcp, 0.0.0.0:9092->9092/tcp            controller-0
> e8274f734429  localrepo/confluentinc/cp-kafka:latest  /etc/confluent/do...  3 days ago  Up 4 hours  0.0.0.0:8081->8091/tcp, 0.0.0.0:9093-9095->9093-9095/tcp  broker-0

There is nothing that gets denied on SeLinux, we turned it off as well with no difference, so I don’t think that is it. Same for the local firewall.

No clue what it can be except Kafka itself, but I assume that release is working fine for thousands of other ppl.
Very very odd.

Edit - Just ran kafka-e2e-latency cause i had hoped it would print more details on the various stages…
It looks ok except every 10s or so (usually, sometimes its different) we get peaks (like at 225000, 230000) which explains the 22ms 99.9th percentile. But that does not tell me why…

222000  1.903323
223000  2.3260859999999997
224000  1.376086
225000  **11.132087**
226000  2.1862649999999997
227000  4.28301
228000  3.138185
229000  2.5233670000000004
230000  **9.439168**
231000  2.2690770000000002
232000  2.7217710000000004
233000  2.6167860000000003
..
298000  2.697682
299000  2.292109
Avg latency: 3.7426 ms
Percentiles: 50th = 2, 99th = 8, 99.9th = 22

Rand · 17 February 2025 16:49

Got a tip that it might be related to the way we call podman (passing individual port instead of host network, potentially slower), will give it a try and post feedback.

Rand · 18 February 2025 08:47

Well, thats definitly looking better now

> /bin/kafka-producer-perf-test --topic perf-test --throughput -1 --num-records 3000000 --record-size 1024 --producer-props acks=all bootstrap.servers=host0:9094,host1:9094,host2:9094
> 139681 records sent, 27930.6 records/sec (27.28 MB/sec), 599.9 ms avg latency, 2878.0 ms max latency.
> 176691 records sent, 35338.2 records/sec (34.51 MB/sec), 1057.8 ms avg latency, 6989.0 ms max latency.
> 193044 records sent, 38608.8 records/sec (37.70 MB/sec), 811.7 ms avg latency, 2919.0 ms max latency.
> 179370 records sent, 35874.0 records/sec (35.03 MB/sec), 570.0 ms avg latency, 2452.0 ms max latency.
> 180139 records sent, 36020.6 records/sec (35.18 MB/sec), 385.2 ms avg latency, 3051.0 ms max latency.
> 189703 records sent, 37925.4 records/sec (37.04 MB/sec), 1361.5 ms avg latency, 11807.0 ms max latency.
> 175965 records sent, 35186.0 records/sec (34.36 MB/sec), 591.5 ms avg latency, 10890.0 ms max latency.
> 192210 records sent, 38434.3 records/sec (37.53 MB/sec), 787.3 ms avg latency, 11442.0 ms max latency.
> 199380 records sent, 39796.4 records/sec (38.86 MB/sec), 705.3 ms avg latency, 16541.0 ms max latency.
> 169005 records sent, 33801.0 records/sec (33.01 MB/sec), 518.6 ms avg latency, 21348.0 ms max latency.
> 185825 records sent, 37157.6 records/sec (36.29 MB/sec), 416.2 ms avg latency, 3380.0 ms max latency.
> 158490 records sent, 31698.0 records/sec (30.96 MB/sec), 464.3 ms avg latency, 3281.0 ms max latency.
> 160196 records sent, 32020.0 records/sec (31.27 MB/sec), 1081.5 ms avg latency, 31466.0 ms max latency.
> 205818 records sent, 41163.6 records/sec (40.20 MB/sec), 1500.0 ms avg latency, 32729.0 ms max latency.
> 207855 records sent, 41479.7 records/sec (40.51 MB/sec), 1415.4 ms avg latency, 26552.0 ms max latency.
> 181650 records sent, 36330.0 records/sec (35.48 MB/sec), 516.5 ms avg latency, 3018.0 ms max latency.
> 93744 records sent, 15431.1 records/sec (15.07 MB/sec), 1121.0 ms avg latency, 11059.0 ms max latency.
> 5025 records sent, 913.3 records/sec (0.89 MB/sec), 11139.6 ms avg latency, 15479.0 ms max latency.
> 450 records sent, 87.9 records/sec (0.09 MB/sec), 15394.8 ms avg latency, 20124.0 ms max latency.
> 4875 records sent, 881.6 records/sec (0.86 MB/sec), 17440.9 ms avg latency, 20904.0 ms max latency.
> 3000000 records sent, 29257.446020 records/sec (28.57 MB/sec), 877.77 ms avg latency, 32729.00 ms max latency, 215 ms 50th, 2995 ms 95th, 11680 ms 99th, 30136 ms 99.9th.

There’s a weird drop off at the end (and I see it every now and then in longer tests) but I assume thats more a storage issue than a kafka issue, but will take a look at memory metrics to rule that out.

This is with
podman run -d --restart always --net=host \
instead of individual ports, just for reference.

With this I consider this resolved, many thanks for your help.
Will just post one followup q to the jmx thread.

Thanks,
take care

mmuehlbeyer · 18 February 2025 12:12

great

interesting root cause

system · 25 February 2025 12:12

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Kafka producer latency is high Ops	0	3360	22 June 2022
Slow performance on kafka Ops	16	10382	11 November 2022
Producer Performance - Differences Clients	2	3231	1 March 2022
Troubleshoot slow kafka writes Ops	0	812	25 March 2024
Kafka producer performance issue in spring boot 2.7 Java Clients	1	1722	21 August 2023

Performance Issues with confluent kafka vs older cluster

Related topics