Export topics, ACLs and other configuration as JSON/text

Tools Command Line tools
1

I’m surprised I’m unable to find more posts like this, frankly.

We have a cluster, it’s running fine, but we want to move from making configuration changes from a git repository (YAML files and Ansible playbooks) to making configuration changes directly in the cluster itself, and instead reading out the running configuration in some text format that we can then automatically commit into git. This will then serve as both backup/disaster recovery and documentation, while freeing us up to onboard new users with topics and ACLs more quickly and easily.

The problem is, I can’t find any way to easily take the running configuration in Kafka and export it in a serialized or text-based format, nor is there seemingly any way to take a known text format and write it back into Kafka. How can that be?

Are there really no tools for this?

My basic idea is that we would read all the topics and topic configurations, ACLs, quotas and the like, and output it to prettified JSON that we can then auto-commit to a git repo. If ever needed, for example as disaster recovery or just to resync the cluster, we could also do the reverse, read the JSON files into topics, ACLs, etc. in the running cluster.

Note that I am NOT asking to backup any topic data, that’s NOT the scope. All I need are tools to export and import the cluster’s running configuration. Static config files, filesystem backups, all of that are already taken care of.

2

hey @maek48

did you consider a small python snippet?

make use of confluent-kafka library:

from confluent_kafka.admin import AdminClient

admin = AdminClient({'bootstrap.servers': 'localhost:9092'})
topic = 'my-topic'

configs = admin.describe_configs([('topic', topic)])
for res in configs.values():
    config = res.result()
    config_json = {k: v.value for k, v in config.items()}
    print(config_json)

not sure if this what you’re looking for though it might be something to start with

best,

michael

3

or use kafka-configs

kafka-configs --bootstrap-server localhost:9092 --entity-type topics --entity-name demo --describe 
| awk -F’=’ ‘{printf “"%s": "%s",\n”, $1, $2}’ 
| sed ‘$s/,$//’ 
| awk ‘BEGIN{print “{”} {print} END{print “}”}’

{
“Dynamic configs for topic demo are:”: “”,
"  retention.ms": “600000 sensitive”
}
4

Yeah, I’ve taken to writing my own Ansible modules using confluent-kafka to export topics and ACLs into JSON, but the problem is that for quota configurations, there is no method to read those objects at all. librdkafka, which confluent-kafka uses to communicate with the Admin API, doesn’t yet support the features in KIP-546, which is what would allow me to export those as well.

My question comes from the surprisingly barebones and anemic set of tools around Kafka, especially the Confluent packaged version. How is it that in 2025, there isn’t even a single “–json” output option for the CLI tools or any other easy way to get the configuration out of the cluster in anything other than terminal shell output?

Anyway, for quotas, I figured out this oneliner to export the current configuration:

kafka-configs --describe --all --entity-type users | sed -n -e "s/^Quota configs for user-principal '\(.*\)' are \(.*\)=\(.*\)/'\1';\2;\3/p"

At the very least, this gives me easily machine-readable output (again, why do NONE of the tools have options for machine-readable output, and why are they ALL DIFFERENT types of output?) that I can then parse into something more usable, like JSON.

But that still leaves the problem of writing quota configurations unsolved. Currently, I have to do that one at a time, using ugly CLI command calls in Ansible, where every single call takes multiple seconds and I have literally thousands of users to configure quotas for…

5

hey,

fully understand your issues.

afaik there is nothing right here (as you already discovered)

but maybe someone of Confluent could take this with them and check whether something is planned or if there’s a tool or so.

@Joe are you aware of a tool @maek48 is looking for?

best,

michael

6

Thanks for asking for me.

To add some context, no one in our teams is a Java programmer, nor do we even work with developers at all. We operate the cluster solely as the input for an Elasticsearch and Logstash cluster, meaning all the Java-based API opportunities are entirely lost to us. We rely on things like CLI tools or APIs in more ops-centric scripting languages, like Python or REST.

Might also be worth noting that we’re still on Kafka 7.4 and Zookeeper, in the community edition. We do have plans to upgrade, but those are the options available to us right now.