How to get offering wise topics, partitions, offsets, disk etc.. using kafka default commands?

Ops
1

I am new to Kafka and trying to find output of below points.

  1. How many topics does an offering have?
  • How should we collect and store this information, so we can use in the following tasks.
  1. How many partitions does an offering have?
  • Iterating over the topics and sumin up the number of partitions
  1. How much disk space is an offering using?
  2. What is the current offset for a topic/partition?
  • the existing scripts (kafka-run-class.sh) don’t support SASL/SSL unfortunately
  1. How many connection to the cluster is a client responsible?

Can someone help here to get these output?

2

Hi,

to list your topics

kafka-topics.sh --bootstrap-server=localhost:9092 --list

to describe the topics and partitons

kafka-topics.sh --describe --topic <yourtopic> --bootstrap-server localhost:9092

Monitoring
have look at

https://docs.confluent.io/platform/current/kafka/monitoring.html

possibility to monitor active connections.

Talking about ‘offering’ what exactly are you talking about?

3

We are providing shared services and kafka is one of them and may customers using it, customers = offerings.
We want to filter things separately as per offerings.
At initial question using list of topics as per offerings need to print other stuff like partitions, offsets, disk space etc…

Is this possible to get?

4

@sdewarde The concept of an “offering” seems very specific to your use case. Maybe you can clarify your question. Kafka brokers are deployed in clusters. Are you trying to determine this information from a single cluster or across clusters? Are the Kafka clusters managed by you or are you using a managed service?

5

@rick Offering in the sense customers using kafka which are in clusters and managed by us.
We have other customers project and they use Kafka from us, they have multiple topics being created with similar prefixes and read/write access provided to them etc…
In our project management want to get list of everything, like how many topics per customer is using and their total partitions, disk size and replications.
I hope you understood what I mean exactly now.

6

It seems that you’ll have to implement your own aggregations of the data you get from the standard tools. There is nothing built into Kafka, that I’m aware of, to support building and monitoring of a multi-tenant solution like you’re describing.

7

Sure, Thanks Rick, trying something similar to as below.

Get all the topic info

Might be helpful for later commands

kfk-topics-describe-all > /tmp/kafka-topics

Get the list of offerings

clients = getClients()
for client in clients
do
topics = getClientTopics(client)
clientTotalPartitions = 0
clientTotalDisk = 0
clientMessageOffset = 0
for topic topics
do
partitionCount = topic.getPartitionCount()
clientTotalPartitions += $partitionCount
for i in 1…$partitionCount
do
clientTotalDisk += topic.getPartitionSize()
clientMessageOffset += topic.getPartitionOffset()
done
done
# Print the stats
print $client $clientTotalPartitions $clientTotalDisk $clientMessageOffset
done