Enable JMX for prometheus

Rand · 6 February 2025 12:05

Hi,

@mmuehlbeyer
In my very first topic you provided a config including the JMX setup but unfortunatly this got deleted.
Can you reshare this?

I tried simply adding KAFKA_JMX_Port and Hostname (and allow the port for the container), but when I query it via curl I only get

curl: (52) Empty reply from server

So I guess I am missing something
Thanks

mmuehlbeyer · 6 February 2025 13:05

Hi @Rand
I guess you’re looking for this one

gist.github.com

https://gist.github.com/mmuehlbeyer/3c5e3bc5275a6fc3eb937a9e0cb2b593

docker multinode.md

download all the files and spread across the node
adapt and change according to your existing hostnames
**replace vm0[1-3]**


if running on a single node create entries in /etc/hosts. 

put the kafka_config.yml in a subdirectory "volumes"
download https://github.com/sknop/kafka-docker-composer/blob/master/volumes/jmx_prometheus_javaagent-0.20.0.jar and put it to volumes
as well.

kafka_config.yml

---
startDelaySeconds: 120
lowercaseOutputName: true
lowercaseOutputLabelNames: true
blacklistObjectNames:
  - "kafka.consumer:type=*,id=*"
  - "kafka.consumer:type=*,client-id=*"
  - "kafka.consumer:type=*,client-id=*,node-id=*"
  - "kafka.producer:type=*,id=*"
  - "kafka.producer:type=*,client-id=*"

This file has been truncated. show original

vm01.yml

services:
    controller-1:
        image: confluentinc/cp-kafka:7.7.0
        hostname: vm01
        container_name: controller-1
        environment:
            KAFKA_NODE_ID: 1
            CLUSTER_ID: Nk018hRAQFytWskYqtQduw
            KAFKA_PROCESS_ROLES: controller
            KAFKA_LISTENERS: CONTROLLER://controller-1:19091

This file has been truncated. show original

There are more than three files. show original

Best,
Michael

Rand · 6 February 2025 13:42

Yes, that was it, thanks.
I’ll have look:)

Rand · 7 February 2025 09:38

@mmuehlbeyer
Is there a reason not to use the latest version - that would be 1.1.0 (2 releases newer)?

mmuehlbeyer · 7 February 2025 13:58

no specific reason, just a copy paste one

colleagues tested with
1.1.0 in conjunction with K8s therefore I think you should be fine.
(jmx-monitoring-stacks/jmxexporter-prometheus-grafana/cfk at main · confluentinc/jmx-monitoring-stacks · GitHub)

best,
michael

Rand · 11 February 2025 12:11

@mmuehlbeyer

I’ve now added the jmx exporter to one broker, and it seems to work, i see the startup info on broker start.

However when running kafka-topics inside to recreate the test topic (or bascially when running any command) it seesm to try to start the Exporter again despite it still running?

I’m using your config basically…

sh-4.4$ ./kafka-topics \
> --create \
> --topic perf-test \
> --partitions 16 \
> --replication-factor 3 \
> --config retention.ms=86400000 \
> --config min.insync.replicas=2 \
> --bootstrap-server host0:9094,host1:9094,host2:9094
2025-02-11 12:05:59.503 | main | INFO | io.prometheus.jmx.JavaAgent | Starting ...
2025-02-11 12:05:59.693 | main | INFO | io.prometheus.jmx.JavaAgent | HTTP enabled [true]
2025-02-11 12:05:59.694 | main | INFO | io.prometheus.jmx.JavaAgent | HTTP host:port [0.0.0.0:8091]
2025-02-11 12:05:59.694 | main | INFO | io.prometheus.jmx.JavaAgent | OpenTelemetry enabled [false]
Failed to start Prometheus JMX Exporter ...

java.net.BindException: Address already in use
        at java.base/sun.nio.ch.Net.bind0(Native Method)
        at java.base/sun.nio.ch.Net.bind(Unknown Source)
        at java.base/sun.nio.ch.ServerSocketChannelImpl.netBind(Unknown Source)
        at java.base/sun.nio.ch.ServerSocketChannelImpl.bind(Unknown Source)
        at java.base/sun.nio.ch.ServerSocketAdaptor.bind(Unknown Source)
        at jdk.httpserver/sun.net.httpserver.ServerImpl.<init>(Unknown Source)
        at jdk.httpserver/sun.net.httpserver.HttpServerImpl.<init>(Unknown Source)
        at jdk.httpserver/sun.net.httpserver.DefaultHttpServerProvider.createHttpServer(Unknown Source)
        at jdk.httpserver/com.sun.net.httpserver.HttpServer.create(Unknown Source)
        at io.prometheus.metrics.exporter.httpserver.HTTPServer$Builder.buildAndStart(HTTPServer.java:263)
        at io.prometheus.jmx.common.http.HTTPServerFactory.createHTTPServer(HTTPServerFactory.java:125)
        at io.prometheus.jmx.JavaAgent.premain(JavaAgent.java:84)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.base/java.lang.reflect.Method.invoke(Unknown Source)
        at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(Unknown Source)
        at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallPremain(Unknown Source)

Prometheus JMX Exporter exiting
sh-4.4$ ./kafka-topics --list --bootstrap-server host0:9094,host1:9094,host2:9094
2025-02-11 12:06:37.464 | main | INFO | io.prometheus.jmx.JavaAgent | Starting ...
2025-02-11 12:06:37.653 | main | INFO | io.prometheus.jmx.JavaAgent | HTTP enabled [true]
2025-02-11 12:06:37.653 | main | INFO | io.prometheus.jmx.JavaAgent | HTTP host:port [0.0.0.0:8091]
2025-02-11 12:06:37.653 | main | INFO | io.prometheus.jmx.JavaAgent | OpenTelemetry enabled [false]
Failed to start Prometheus JMX Exporter ...

java.net.BindException: Address already in use
        at java.base/sun.nio.ch.Net.bind0(Native Method)
        at java.base/sun.nio.ch.Net.bind(Unknown Source)
        at java.base/sun.nio.ch.ServerSocketChannelImpl.netBind(Unknown Source)
        at java.base/sun.nio.ch.ServerSocketChannelImpl.bind(Unknown Source)
        at java.base/sun.nio.ch.ServerSocketAdaptor.bind(Unknown Source)
        at jdk.httpserver/sun.net.httpserver.ServerImpl.<init>(Unknown Source)
        at jdk.httpserver/sun.net.httpserver.HttpServerImpl.<init>(Unknown Source)
        at jdk.httpserver/sun.net.httpserver.DefaultHttpServerProvider.createHttpServer(Unknown Source)
        at jdk.httpserver/com.sun.net.httpserver.HttpServer.create(Unknown Source)
        at io.prometheus.metrics.exporter.httpserver.HTTPServer$Builder.buildAndStart(HTTPServer.java:263)
        at io.prometheus.jmx.common.http.HTTPServerFactory.createHTTPServer(HTTPServerFactory.java:125)
        at io.prometheus.jmx.JavaAgent.premain(JavaAgent.java:84)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.base/java.lang.reflect.Method.invoke(Unknown Source)
        at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(Unknown Source)
        at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallPremain(Unknown Source)

Prometheus JMX Exporter exiting

Also it does not seem to exec the actual command maybe due to the error? (ie no new topic created, not all topics listed)?

Rand · 11 February 2025 12:18

Also, is there a Grafana dashboard related to your kafka_yaml.file?

I found one at Grafana’s but thats outdated (Angular) and using different metrics.
I can adjust o/c but if there is a mathcing version that you know off…

(Edit - removed the solved marker so this wont close on us while there are still things to discuss - or me ask you about:p)

mmuehlbeyer · 11 February 2025 12:45

did you check if there is something else running on that port?

mmuehlbeyer · 11 February 2025 12:45

I think the one in the shared confluent jmx github repo should fit

Rand · 11 February 2025 12:47

The JMX exporter is running on that port Its working fine, I see metrics in prometheus .

Why is started again when i run kafka topics is the question

mmuehlbeyer · 11 February 2025 13:04

ah sorry was a bit confused

mmh probably as the KAFKA_OPTS:
are taken/evaluated as well by kafka-topics command

in detail kafka-topics uses kafka-run-class which has the following

exec "$JAVA" $KAFKA_HEAP_OPTS $KAFKA_JVM_PERFORMANCE_OPTS $KAFKA_GC_LOG_OPTS $KAFKA_JMX_OPTS $KAFKA_LOG4J_CMD_OPTS -cp "$CLASSPATH" $KAFKA_OPTS "$@

Rand · 11 February 2025 13:08

Thats makes sense, so a simple fix is to reset the shells value for the current session, works fine

export KAFKA_OPTS=“”
sh-4.4$ ./kafka-topics --create --topic perf-test --partitions 16 --replication-factor 3 --config retention.ms=86400000 --config min.insync.replicas=2 --bootstrap-server host0:9094,host1:9094,host2:9094
Created topic perf-test.

Rand · 18 February 2025 08:51

So, with us now passing through network as a whole I am now exposing two JMX ports to the outside,

one in JMX_PORT, and a different one in the KAFKA_OPTS

-e KAFKA_JMX_PORT=18091 \
-e KAFKA_JMX_HOSTNAME=host0 \
-e KAFKA_OPTS="-javaagent:/monitor/jmx_prometheus_javaagent-1.1.0.jar=8091:/monitor/broker_config.yml" \

The Kafka_opt is the one which actually provides data, and JMX Port is used to activate JMX in the first place normally ö-
do I really need both? Or will using the Opts in this way automatically activate JMX without me specifically setting the port?

Thanks

Rand · 18 February 2025 14:39

@mmuehlbeyer Any comment on this to bring this to a close?
Thanks

mmuehlbeyer · 18 February 2025 19:30

let me check
will come back latest thursday

mmuehlbeyer · 20 February 2025 06:10

just tested @Rand

you could omit the KAFKA_JMX_PORT
at least for me it was working just with the KAFKA_OPTS setting and then exposing the port via docker config.

best,
michael

Rand · 20 February 2025 06:41

Brilliant, thanks a lot:)

Topic		Replies	Views
How to enable JMX port in Confluent Cloud? Confluent Cloud	3	6429	15 April 2021
Kafka - random high port Containers	6	1133	23 January 2024
Prometheus not publishing broker metrics Containers	2	3128	29 December 2022
Kafka Monitoring and Metrics Using JMX Confluent Cloud	11	5119	22 August 2022
Kafka Monitoring Confluent Cloud	9	4543	7 September 2022

Enable JMX for prometheus

Related topics