How do I count the number of messages in a Kafka topic?

rmoff · 3 December 2020 14:53

I want to know how many messages are in the topic, how can I do this?

shree · 16 February 2021 08:46

@rmoff don’t we all want to know the answer to this
But I am certainly interested to know what others are doing to handle this.

rmoff · 16 February 2021 09:04

Here are a few options: Counting the number of messages in a Kafka topic

Also check out this cool vid from @gAmUssA

riferrei · 18 February 2021 21:28

Another option is using the kafka.tools.GetOffsetShell command:

bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list <BROKER_LIST> --topic <TOPIC_NAME> --time -1 --offsets 1 | awk -F ":" '{sum += $3} END {print sum}'

The source-code of this command is not very intuitive (Scala ) but it may provide an idea about how it works:

github.com

apache/kafka/blob/b58b944e70780029806a6004114bba43ce5b9483/core/src/main/scala/kafka/tools/GetOffsetShell.scala

/*
 * Licensed to the Apache Software Foundation (ASF) under one
 * or more contributor license agreements.  See the NOTICE file
 * distributed with this work for additional information
 * regarding copyright ownership.  The ASF licenses this file
 * to you under the Apache License, Version 2.0 (the
 * "License"); you may not use this file except in compliance
 * with the License.  You may obtain a copy of the License at
 *
 *   http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing,
 * software distributed under the License is distributed on an
 * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
 * KIND, either express or implied.  See the License for the
 * specific language governing permissions and limitations
 * under the License.
 */
package kafka.tools

This file has been truncated. show original

Topic		Replies	Views
How to get offering wise topics, partitions, offsets, disk etc.. using kafka default commands? Ops	6	3568	27 April 2021
Read Kafka Messages from a Specific Offset & Partition in the Console Consumer Command Line tools	0	3879	16 November 2020
How do I get the the offset of last message of a Kafka topic using confluent-kafka-python? Non-Java Clients	1	11730	27 May 2022
Topic disk size utilised Confluent Cloud	3	539	21 June 2024
How does Kafka Stream retain older msgs that retention policy? ksqlDB	1	3555	7 April 2021

How do I count the number of messages in a Kafka topic?

Related topics