Properties with and without "consumer." prefix

[Dawid]

Hello!

In our application, we use the Quarkus extension to configure Kafka Streams. This extension applies properties with the kafka-streams. prefix as Kafka Streams configuration.

I’m wondering about the following settings that were added at some point:

kafka-streams.metadata.max.age.ms=500
kafka-streams.max.poll.records=250
kafka-streams.max.poll.interval.ms=1200000
kafka-streams.consumer.max.poll.records=10
kafka-streams.consumer.max.poll.interval.ms=480000

Do the properties kafka-streams.max.poll.records and kafka-streams.max.poll.interval.ms have any effect if kafka-streams.consumer.max.poll.records and kafka-streams.consumer.max.poll.interval.ms are also defined (or vice versa)? For example, I couldn’t find in the documentation whether there’s any difference between max.poll.records with or without the consumer. prefix.

We also have metadata.max.age.ms without consumer. prefix, but we don’t define its consumer. equivalent. Does it do anything meaningful or is it ignored because it does not have consumer. prefix?

Thank you in advance for answers

[mjsax]

Yes, there is relationship. It’s basically a “hierarchy”.

I don’t know about the kafka-streams. prefix – seems to be a Quarkus thing, prefixing all configs passed into Kafka Streams.

Putting kafka-streams. prefix aside, you can pass in KS config as well as client configs into Kafka Streams. If you don’t prefix a client config (eg, consumer.) the config would be applied to all internal clients (ie, consumer(s), producer. admin) – there is some configs (eg, auth or metadata.max.age.ms from you example config) that apply to all clients. So metadata.max.age.ms in your example will be applied to all clients, as it’s not prefixed.

If you would want to set different value, eg, for consumer vs producer, you would specify two value one prefix with consumer. and one prefixed with producer..

If you have a consumer specific config, you might want to prefix it with consumer.. From your example. max.poll.interval.ms is a consumer config. If it’s not prefixed with consumer. it will also be passed into the producer and admin – both producer and admin don’t understand this configs and would ignore it (but they would log a WARN about it, what can be annoying – using the prefix avoids such WARN logs).

If you specify a config twice, with and w/o prefix, the prefixed one wins and overwrites the non-prefixed one. So in your example, max.poll.records would be set to 10, and max.poll.interval.ms would be set to 480000.

Btw: as there are three different consumer inside KS, there is actually also main.consumer, restore.consumer. and global.consumer prefix, allowing you to even configure the different consumers differently if there is a need for it.

Effective configs are also logged by each client at startup, so it’s a good way to double check how each client is really configured.

Does this help?

Docs Configure a Kafka Streams Application for Confluent Platform | Confluent Documentation actually cover it… Are the docs not clear?

[Dawid]

Thanks for such a detailed answer! The fact that in our application we have properties both with and without the .consumer prefix, along with some confusing answers from AI-based chats, made me want to double-check. Especially regarding max.poll.records, since some AI chats claimed that without the prefix, this property also affects other aspects of Kafka Streams behavior The part of the documentation about the hierarchy is clear.