How do I count the number of messages in a Kafka topic?

rmoff · 3 December 2020 14:53

I want to know how many messages are in the topic, how can I do this?

shree · 16 February 2021 08:46

@rmoff don’t we all want to know the answer to this
But I am certainly interested to know what others are doing to handle this.

rmoff · 16 February 2021 09:04

Here are a few options: Counting the number of messages in a Kafka topic

Also check out this cool vid from @gAmUssA

riferrei · 18 February 2021 21:28

Another option is using the kafka.tools.GetOffsetShell command:

bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list <BROKER_LIST> --topic <TOPIC_NAME> --time -1 --offsets 1 | awk -F ":" '{sum += $3} END {print sum}'

The source-code of this command is not very intuitive (Scala ) but it may provide an idea about how it works:

github.com

apache/kafka/blob/b58b944e70780029806a6004114bba43ce5b9483/core/src/main/scala/kafka/tools/GetOffsetShell.scala

/*
 * Licensed to the Apache Software Foundation (ASF) under one
 * or more contributor license agreements.  See the NOTICE file
 * distributed with this work for additional information
 * regarding copyright ownership.  The ASF licenses this file
 * to you under the Apache License, Version 2.0 (the
 * "License"); you may not use this file except in compliance
 * with the License.  You may obtain a copy of the License at
 *
 *   http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing,
 * software distributed under the License is distributed on an
 * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
 * KIND, either express or implied.  See the License for the
 * specific language governing permissions and limitations
 * under the License.
 */
package kafka.tools

This file has been truncated. show original

rmoff · 25 February 2021 21:29

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
How to get offering wise topics, partitions, offsets, disk etc.. using kafka default commands? Ops	6	3185	27 April 2021
How do I get the the offset of last message of a Kafka topic using confluent-kafka-python? Non-Java Clients	1	7609	27 May 2022
How can I reset the topic offset while consuming from Kafka topic Non-Java Clients	2	272	21 March 2024
How can I read the last message from a Kafka topic that has three partitions? Kafka Connect	1	1005	9 December 2023
Consume messages in last N days using Python client Non-Java Clients	6	5343	22 February 2022

How do I count the number of messages in a Kafka topic?

Related Topics