Grafna - promtail - kafka integration

Confluent Cloud
1

I am running into the below error when trying to connect to a Kafka cluster/topic in Confluent Cloud using Promtail

level=error ts=2022-08-05T23:20:02.398245259Z caller=main.go:117 msg="error creating promtail" error="failed to make kafka target manager: error creating kafka client: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)"

Below is my scrape config in Promtail.

- job_name: kafka
  kafka:
    brokers:
    - [redacted].confluent.cloud:9092
    - https://[redacted].confluent.cloud:443
    topics:
    - [redacted]
    group_id: kafka-promtail-cg
    authentication: 
      type: sasl
      sasl_config:
        mechanism: PLAIN
        user: [redacted]
        password: [redacted]
    labels:
      job: kafka

Has anybody run into similar issues before or have been able to use Promtail to read data from a Kafka topic hosted in Confluent Cloud?

Thanks.

2

hey @gautamg

not sure if it fits your need but did you consider:

best,
michael

3

Thank you for the link but I am looking for Promtail integration.

I have been able to integrate with Datadog, Prometheus+Grafana for the Kafka metrics in the past but am now looking for strictly Promtail integration.

Promtail gives the ability to read from any Kafka topics using the consumer strategy unlike the ones mentioned in the link above.

4

I was able to resolve this on my end since opening the ticket.

The missing attribute in the promtail configuration file for the kafka scraper config that was required to get working is

use_tls: true
1 Like
5

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.