Hi Team,
I am trying to monitor and evaluate a data pipeline with Kafka, KsqlDB, Kafka Connect, PostgreSQL in a docker environment. My objective is to stream data from source connector, do some stream analysis by joining two topics, and then dump it into the sink database (PostgreSQL in our case). I am using Voluble Data Generator as a source connector, and JDBC sink connector as a sink connector. I have KsqlDB for stream processing, and Kafka Connect embedded in it (I was having some issue when Kafka Connect is deployed separately from KsqlDB). Now I want to setup a monitoring stack to monitor the architecture with the help of Prometheus and Grafana. I have been successfully able to setup the monitoring for Kafka-broker. However, I am having some issue in scraping the metrics from Kafka Connect and KsqlDB. My ultimate objective is to evaluate the architecture and find out the following metrics:
- End to End latency of the whole stack (time since the data is generated from the source connector to the time it is actually dumped into the PostgreSQL)
- Performance vs Throughput (I want to see how performance of the stack varies as I change the throughput of my data being generated by the source connector)
- If there are any duplicate messages being generated by the system (I am not sure what graph will be useful for this)
I have a “jmx-exporter” directory which has all the config.yaml files downloaded from jmx_exporter/example_configs at master · prometheus/jmx_exporter · GitHub. The folder also have the “jmx_prometheus_javaagent-0.17.0.jar” JAR file.
The following is my docker-compose file:
version: '3'
services:
postgres:
image: quay.io/debezium/postgres:9.6
ports:
- "5432:5432"
environment:
- POSTGRES_USER=postgresuser
- POSTGRES_PASSWORD=postgrespw
- POSTGRES_DB=inventory
networks:
- kafka
zookeeper:
image: confluentinc/cp-zookeeper:7.2.0
logging:
driver: local
ports:
- 2181:2181
- 1999:1999
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
EXTRA_ARGS: -javaagent:/usr/share/jmx_exporter/jmx_prometheus_javaagent-0.17.0.jar=1999:/usr/share/jmx_exporter/zookeeper.yaml
volumes:
- ./jmx-exporter:/usr/share/jmx_exporter/
- zookeeper-jmx-volume:/jmx-exporter
networks:
- kafka
kafka:
image: confluentinc/cp-kafka:7.2.0
logging:
driver: local
ports:
- 29094:29092
- 29103:29101
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_LISTENERS: PLAINTEXT://:9092,PLAINTEXT_HOST://:29092
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://127.0.0.1:29094
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
CONFLUENT_TOPIC_REPLICATION_FACTOR: 1
EXTRA_ARGS: -javaagent:/usr/share/jmx_exporter/jmx_prometheus_javaagent-0.17.0.jar=29101:/usr/share/jmx_exporter/kafka-broker.yaml
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
networks:
- kafka
volumes:
- ./jmx-exporter:/usr/share/jmx_exporter/
- kafka-jmx-volume:/jmx-exporter
depends_on:
- zookeeper
schema-registry:
image: confluentinc/cp-schema-registry:7.2.0
hostname: schema-registry
container_name: schema-registry
depends_on:
- zookeeper
- kafka
ports:
- "8081:8081"
environment:
SCHEMA_REGISTRY_HOST_NAME: schema-registry
SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: "PLAINTEXT://kafka:9092"
networks:
- kafka
ksqldb-server:
image: confluentinc/ksqldb-server:0.27.2
hostname: ksqldb-server
container_name: ksqldb-server
depends_on:
- kafka
- schema-registry
ports:
- "8088:8088"
- "9012:9012"
- "1169:1169"
volumes:
- ./confluent-hub-components/confluentinc-kafka-connect-jdbc:/usr/share/kafka/plugins/jdbc
- "./confluent-hub-components/debezium-debezium-connector-mysql:/usr/share/kafka/plugins/debezium-debezium-connector-mysql"
- "./confluent-hub-components/mdrogalis-voluble:/usr/share/kafka/plugins/mdrogalis-voluble"
- ./jmx-exporter:/usr/share/kafka/plugins/jmx_exporter/
- ksqldb-jmx-volume:/jmx-exporter
environment:
# KSQL_JMX_OPTS: "-Djava.rmi.server.hostname=localhost -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=3001 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.rmi.port=3001"
KSQL_LISTENERS: "http://0.0.0.0:8088"
KSQL_BOOTSTRAP_SERVERS: "kafka:9092"
KSQL_CONNECT_REST_ADVERTISED_HOST_NAME: 'ksqldb'
KSQL_KSQL_SCHEMA_REGISTRY_URL: "http://schema-registry:8081"
KSQL_KSQL_LOGGING_PROCESSING_STREAM_AUTO_CREATE: "true"
KSQL_KSQL_LOGGING_PROCESSING_TOPIC_AUTO_CREATE: "true"
# Configuration to embed Kafka Connect support.
KSQL_CONNECT_GROUP_ID: "ksql-connect-cluster"
KSQL_CONNECT_BOOTSTRAP_SERVERS: "kafka:9092"
KSQL_CONNECT_KEY_CONVERTER: "org.apache.kafka.connect.storage.StringConverter"
KSQL_CONNECT_VALUE_CONVERTER: "io.confluent.connect.avro.AvroConverter"
KSQL_CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL: "http://schema-registry:8081"
KSQL_CONNECT_CONFIG_STORAGE_TOPIC: "_ksql-connect-configs"
KSQL_CONNECT_OFFSET_STORAGE_TOPIC: "_ksql-connect-offsets"
KSQL_CONNECT_STATUS_STORAGE_TOPIC: "_ksql-connect-statuses"
KSQL_CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: 1
KSQL_CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: 1
KSQL_CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: 1
KSQL_CONNECT_PLUGIN_PATH: "/usr/share/kafka/plugins"
# KSQL_CONNECT_EXTRA_ARGS: -javaagent:/usr/share/kafka/plugins/jmx_exporter/jmx_prometheus_javaagent-0.17.0.jar=1169:/usr/share/kafka/plugins/jmx_exporter/kafka-connect.yaml
KSQL_CONNECT_KAFKA_OPTS: -javaagent:/usr/share/kafka/plugins/jmx_exporter/jmx_prometheus_javaagent-0.17.0.jar=1169:/usr/share/kafka/plugins/jmx_exporter/kafka-connect.yaml
KSQL_CONNECT_JMXHOST: "0.0.0.0"
KSQL_CONNECT_JMXPORT: 9012
networks:
- kafka
ksqldb-cli:
image: confluentinc/ksqldb-cli:0.27.2
container_name: ksqldb-cli
depends_on:
- kafka
- ksqldb-server
entrypoint: /bin/sh
tty: true
networks:
- kafka
prometheus:
image: prom/prometheus:v2.1.0
logging:
driver: local
links:
- ksqldb-server
volumes:
- ./prometheus/:/etc/prometheus/
- prometheus-volume:/prometheus
command:
- '--config.file=/etc/prometheus/prometheus.yml'
- '--storage.tsdb.path=/prometheus'
- '--web.console.libraries=/usr/share/prometheus/console_libraries'
- '--web.console.templates=/usr/share/prometheus/consoles'
ports:
- 9090:9090
networks:
- kafka
restart: always
grafana:
image: grafana/grafana
depends_on:
- prometheus
logging:
driver: local
volumes:
- grafana-volume:/var/lib/grafana
- ./grafana/provisioning/:/etc/grafana/provisioning/
networks:
- kafka
environment:
- "GF_SECURITY_ADMIN_USER=kafka"
- "GF_SECURITY_ADMIN_PASSWORD=kafka"
- "GF_USERS_ALLOW_SIGN_UP=false"
ports:
- 3000:3000
volumes:
kafka-jmx-volume:
grafana-volume:
prometheus-volume:
zookeeper-jmx-volume:
ksqldb-jmx-volume:
networks:
kafka:
external: false
Note: I am successfully able to scrape metrics and able to visualize it in the grafana and prometheus for Kafka Broker from the endpoint “localhost:29103” for Kafka and “localhost:1999” for Zookeeper.
How to scrape the metrics similarly for KsqlDB and Kafka Connect. Are there different dashboards available for both components?
P.S: It would be also very helpful if someone can give me tips on how to get the plots for all the metrics I am actually interested in. 