Monitoring KsqlDB and Kafka Connect using Prometheus and Grafana with Docker

Hi Team,

I am trying to monitor and evaluate a data pipeline with Kafka, KsqlDB, Kafka Connect, PostgreSQL in a docker environment. My objective is to stream data from source connector, do some stream analysis by joining two topics, and then dump it into the sink database (PostgreSQL in our case). I am using Voluble Data Generator as a source connector, and JDBC sink connector as a sink connector. I have KsqlDB for stream processing, and Kafka Connect embedded in it (I was having some issue when Kafka Connect is deployed separately from KsqlDB). Now I want to setup a monitoring stack to monitor the architecture with the help of Prometheus and Grafana. I have been successfully able to setup the monitoring for Kafka-broker. However, I am having some issue in scraping the metrics from Kafka Connect and KsqlDB. My ultimate objective is to evaluate the architecture and find out the following metrics:

  1. End to End latency of the whole stack (time since the data is generated from the source connector to the time it is actually dumped into the PostgreSQL)
  2. Performance vs Throughput (I want to see how performance of the stack varies as I change the throughput of my data being generated by the source connector)
  3. If there are any duplicate messages being generated by the system (I am not sure what graph will be useful for this)

I have a “jmx-exporter” directory which has all the config.yaml files downloaded from jmx_exporter/example_configs at master · prometheus/jmx_exporter · GitHub. The folder also have the “jmx_prometheus_javaagent-0.17.0.jar” JAR file.

The following is my docker-compose file:

version: '3'

services:
 
  
  postgres:
    image: quay.io/debezium/postgres:9.6
    ports:
      - "5432:5432"
    environment:
      - POSTGRES_USER=postgresuser
      - POSTGRES_PASSWORD=postgrespw
      - POSTGRES_DB=inventory 
    networks:
      - kafka

  zookeeper:
    image: confluentinc/cp-zookeeper:7.2.0
    logging:
      driver: local
    ports:
      - 2181:2181
      - 1999:1999
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
      EXTRA_ARGS: -javaagent:/usr/share/jmx_exporter/jmx_prometheus_javaagent-0.17.0.jar=1999:/usr/share/jmx_exporter/zookeeper.yaml
    volumes:
      - ./jmx-exporter:/usr/share/jmx_exporter/
      - zookeeper-jmx-volume:/jmx-exporter
    networks:
      - kafka

  kafka:
    image: confluentinc/cp-kafka:7.2.0
    logging:
      driver: local
    ports:
      - 29094:29092
      - 29103:29101
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENERS: PLAINTEXT://:9092,PLAINTEXT_HOST://:29092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://127.0.0.1:29094
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1   
      CONFLUENT_TOPIC_REPLICATION_FACTOR: 1
      EXTRA_ARGS: -javaagent:/usr/share/jmx_exporter/jmx_prometheus_javaagent-0.17.0.jar=29101:/usr/share/jmx_exporter/kafka-broker.yaml
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
    networks:
      - kafka
    volumes:
      - ./jmx-exporter:/usr/share/jmx_exporter/
      - kafka-jmx-volume:/jmx-exporter
    depends_on:
      - zookeeper
  
  schema-registry:
    image: confluentinc/cp-schema-registry:7.2.0
    hostname: schema-registry
    container_name: schema-registry
    depends_on:
      - zookeeper
      - kafka
    ports:
      - "8081:8081"
    environment:
      SCHEMA_REGISTRY_HOST_NAME: schema-registry
      SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: "PLAINTEXT://kafka:9092"
    networks:
      - kafka
  
  ksqldb-server:
    image: confluentinc/ksqldb-server:0.27.2
    hostname: ksqldb-server
    container_name: ksqldb-server
    depends_on:
      - kafka
      - schema-registry
    ports:
      - "8088:8088"
      - "9012:9012"
      - "1169:1169"
    volumes:
      - ./confluent-hub-components/confluentinc-kafka-connect-jdbc:/usr/share/kafka/plugins/jdbc
      - "./confluent-hub-components/debezium-debezium-connector-mysql:/usr/share/kafka/plugins/debezium-debezium-connector-mysql"
      - "./confluent-hub-components/mdrogalis-voluble:/usr/share/kafka/plugins/mdrogalis-voluble"
      - ./jmx-exporter:/usr/share/kafka/plugins/jmx_exporter/
      - ksqldb-jmx-volume:/jmx-exporter
    environment:
      # KSQL_JMX_OPTS: "-Djava.rmi.server.hostname=localhost -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=3001 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.rmi.port=3001"
      KSQL_LISTENERS: "http://0.0.0.0:8088"
      KSQL_BOOTSTRAP_SERVERS: "kafka:9092"
      KSQL_CONNECT_REST_ADVERTISED_HOST_NAME: 'ksqldb'
      KSQL_KSQL_SCHEMA_REGISTRY_URL: "http://schema-registry:8081"
      KSQL_KSQL_LOGGING_PROCESSING_STREAM_AUTO_CREATE: "true"
      KSQL_KSQL_LOGGING_PROCESSING_TOPIC_AUTO_CREATE: "true"
      # Configuration to embed Kafka Connect support.
      KSQL_CONNECT_GROUP_ID: "ksql-connect-cluster"
      KSQL_CONNECT_BOOTSTRAP_SERVERS: "kafka:9092"
      KSQL_CONNECT_KEY_CONVERTER: "org.apache.kafka.connect.storage.StringConverter"
      KSQL_CONNECT_VALUE_CONVERTER: "io.confluent.connect.avro.AvroConverter"
      KSQL_CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL: "http://schema-registry:8081"
      KSQL_CONNECT_CONFIG_STORAGE_TOPIC: "_ksql-connect-configs"
      KSQL_CONNECT_OFFSET_STORAGE_TOPIC: "_ksql-connect-offsets"
      KSQL_CONNECT_STATUS_STORAGE_TOPIC: "_ksql-connect-statuses"
      KSQL_CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: 1
      KSQL_CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: 1
      KSQL_CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: 1
      KSQL_CONNECT_PLUGIN_PATH: "/usr/share/kafka/plugins"
      # KSQL_CONNECT_EXTRA_ARGS: -javaagent:/usr/share/kafka/plugins/jmx_exporter/jmx_prometheus_javaagent-0.17.0.jar=1169:/usr/share/kafka/plugins/jmx_exporter/kafka-connect.yaml
      KSQL_CONNECT_KAFKA_OPTS: -javaagent:/usr/share/kafka/plugins/jmx_exporter/jmx_prometheus_javaagent-0.17.0.jar=1169:/usr/share/kafka/plugins/jmx_exporter/kafka-connect.yaml
      KSQL_CONNECT_JMXHOST: "0.0.0.0"
      KSQL_CONNECT_JMXPORT: 9012
    networks:
      - kafka

  ksqldb-cli:
    image: confluentinc/ksqldb-cli:0.27.2
    container_name: ksqldb-cli
    depends_on:
      - kafka
      - ksqldb-server
    entrypoint: /bin/sh
    tty: true
    networks:
      - kafka

  prometheus:
    image: prom/prometheus:v2.1.0
    logging:
      driver: local
    links:
      - ksqldb-server
    volumes:
      - ./prometheus/:/etc/prometheus/
      - prometheus-volume:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--web.console.libraries=/usr/share/prometheus/console_libraries'
      - '--web.console.templates=/usr/share/prometheus/consoles'
    ports:
      - 9090:9090
    networks:
      - kafka
    restart: always
  
  grafana:
    image: grafana/grafana
    depends_on:
      - prometheus
    logging:
      driver: local
    volumes:
      - grafana-volume:/var/lib/grafana
      - ./grafana/provisioning/:/etc/grafana/provisioning/
    networks:
      - kafka
    environment:
      - "GF_SECURITY_ADMIN_USER=kafka"
      - "GF_SECURITY_ADMIN_PASSWORD=kafka"
      - "GF_USERS_ALLOW_SIGN_UP=false"
    ports:
      - 3000:3000

volumes:
  kafka-jmx-volume:
  grafana-volume:
  prometheus-volume:
  zookeeper-jmx-volume:
  ksqldb-jmx-volume:

networks:
  kafka:
    external: false

Note: I am successfully able to scrape metrics and able to visualize it in the grafana and prometheus for Kafka Broker from the endpoint “localhost:29103” for Kafka and “localhost:1999” for Zookeeper.

How to scrape the metrics similarly for KsqlDB and Kafka Connect. Are there different dashboards available for both components?

P.S: It would be also very helpful if someone can give me tips on how to get the plots for all the metrics I am actually interested in. :slight_smile:

Hi @amanlonare

short question:
basically the jmx endpoint for ksqldb on port 9012 is responding with some infos?

best
michael