3rd party command line tools for Apache Kafka

Tools Command Line tools
1

(See also 3rd party GUI tools for Apache Kafka)

In recent times there’s been a proliferation in CLIs for Apache Kafka, and instead of trawling through Reddit to find the announcements I thought I’d collect them here for reference :slight_smile:

I’ve copied notes from the project’s github pages below to give an idea of the kind of functionality that they offer.

GitHub - edenhill/kcat: Generic command line non-JVM Apache Kafka producer and consumer

Think of it as netcat for Kafka.

The grandaddy of them all :smiley: Still going strong, and my personal favourite.

Produce and Consume messages, and run Metadata queries.

Integrates beautifully with Linux pipes (stdin / stdout)

GitHub - kattlo/kattlo-cli: Kattlo CLI Project

Apache Kafka® Configuration Made Easy

Use an approach like Database Migrations to manage your evolutionary configurations for:

  • Topics
  • Schemas
  • ACLs
  • ksqlDB
  • Connect
  • Cluster

GitHub - adevinta/zoe: The Kafka CLI for humans

The Kafka CLI for humans

  • Consume kafka topics from a specific point in time
  • Filter data based on content, record metadata, and record headers
  • Supports offloading consumption of data to multiple lambda functions, kubernetes pods, etc. for parallelism
  • Monitor consumer groups’ offsets.
  • Upload avro schemas from a .avsc or .avdl file using different naming strategies.

demo

GitHub - jbvmio/kafkactl: CLI for Apache Kafka Management -

CLI for Apache Kafka, Zookeeper and Burrow Management.

  • Search / Filter for Groups and Topics
  • Show Topic Partition Details - Offsets, Replicas, Leaders, etc.
  • Show Group Details - Lag, Offsets, Members, etc.
  • Display / Modify Topics & Topic Configs
  • Delete Consumer Groups
  • Reset Partition Offsets for a Group
  • Tail Topics in realtime
  • Produce to a Specific Topic Partition or all Partitions at once
  • Perform a Preferred Replica Election for a target topic or for all topics
  • Increase / Decrease Replicas
  • Migrate Topics between Brokers
  • Query burrow details, Monitor burrow lag
  • Explore Zookeeper Paths, Create and Delete Zookeeper Values
  • Pass stdin to create Kafka messages or Zookeeper Values

GitHub - deviceinsight/kafkactl: Command Line Tool for managing Apache Kafka

A command-line interface for interaction with Apache Kafka

  • command auto-completion for bash, zsh, fish shell including dynamic completion for e.g. topics or consumer groups.
  • support for avro schemas
  • Configuration of different contexts
  • directly access kafka clusters inside your kubernetes cluster

asciicast

GitHub - devshawn/kafka-shell: ⚡A supercharged, interactive Kafka shell built on top of the existing Kafka CLI tools.

A supercharged, interactive Kafka shell built on top of the existing Kafka CLI tools.

1616×1000 215 KB

GitHub - birdayz/kaf: Modern CLI for Apache Kafka, written in Go.

Kafka CLI inspired by kubectl & docker

GitHub - kdrakon/topiks: An interactive CLI tool for managing Kafka topics

An interactive CLI tool for managing Kafka topics.

  • list topics, configurations, and offsets
  • interactively create & delete topics
  • modify a topics configuration

GitHub - mmolimar/kukulcan: A REPL for Apache Kafka

Kukulcan provides an API and different sort of REPLs to interact with your streams or administer your deployment.

1770×866 141 KB

GitHub - fgeller/kt: Kafka command line tool that likes JSON

a Kafka tool that likes JSON

  • Consume messages
  • Display topic information (e.g., with partition offset and leader info).
  • Modify consumer group offsets
  • JSON output for easy consumption with tools like kp or jq.
  • JSON input to facilitate automation via tools like jsonify.
  • No buffering of output.
  • Binary keys and payloads can be passed and presented in base64 or hex encoding.
  • Support for TLS authentication.
  • Basic cluster admin functions: Create & delete topics.

GitHub - twmb/kcl: Your one stop shop to do anything with Kafka. Producing, consuming, transacting, administrating; 0.8.0 through 3.2+

kcl is a complete, pure Go command line Kafka client. Think of it as your one stop shop to do anything you want to do with Kafka. Producing, consuming, transacting, administrating, and so on.

Unlike the small size of kafkacat, this binary is ~12M compiled. It is, however, still fast, has rich consuming and producing formatting options, and a complete Kafka administration interface.

GitHub - streamdal/plumber: A swiss army knife CLI tool for interacting with Kafka, RabbitMQ and other messaging systems.

plumber is a CLI devtool for inspecting, piping, massaging and redirecting data in message systems like Kafka, RabbitMQ , GCP PubSub and many more.

It’s like curl for messaging systems.

  • See what’s passing through your message systems
  • Pipe data from one place to another
  • Decode protobuf data in real-time
4 Likes
2

Another one! :slight_smile:

1 Like
3

Hey @mmolimar, good to see you here on the forum :slight_smile: Thanks for the link, I’ll add it to the original post.

1 Like
4

(different from the other kafkactl above) but has nice configs for different authentication types.

1 Like
5

Thanks @sarwar-confluent , I’ve added it to the original post

6

One more which is the only one that covers the whole Kafka API as far as I know: GitHub - twmb/kcl: Your one stop shop to do anything with Kafka. Producing, consuming, transacting, administrating; 0.8.0 through 3.2+

1 Like
7

Thanks @weeco - I thought I had that one already but evidently not :smiley: I’ll add it now

1 Like
8

Hi @rmoff within the project KNet there is the tool KNetCLI. It is available in NuGet as a dotnet tool package ( NuGet Gallery | MASES.KNetCLI 1.3.1): it replicates all shell commands available in Apache Kafka (see KNetCLI for a simple explanation).

9

Nice overview. Thanks :slight_smile:

I’d like to add Kafka-Utils. Features are listed in the toc of their docs. I once used this for renaming consumer groups which is a feature I didn’t find in any other tool.

10

Thank you…!
Loading my CLI gun with all these tools… :innocent:

11

Hi @rmoff,
we added a PowerShell command-line tool in the Apache Kafka ecosystem. It is available on PowerShell Gallery | MASES.KNetPS 1.4.4.
The usage, with all cmdlets available in the actual version, can be found in KNet/usagePS.md at 21a030bbae8e40e30e28be80ac9526d2766ef9da · masesgroup/KNet · GitHub or KNet: PowerShell Module | KNet website.
Hope it can be useful.

12

If anybody would want to use deviceinsight’s kafkactl, but wonders how to connect it to a Confluent Kafka cluster, we’ve just published a short article about it.

For example, it became useful to us to increase partition count of existing topics. Doing it through Confluent UI is currently impossible, and doing it through Kafka .sh scripts looked too difficult. So we made a short shell script that does it by running kafkactl in Docker.