Restore consumer keeps subscribing to the same partition

Hello,
I have noticed an unexpected behaviour in one of steram processors based on Kafka Streams. The app is reading some huge topic (hundreds of millions of events) as a table, and whenever it gets into rebalancing state, it starts to log same event intensively (hundreds a second):
[Consumer clientId=XXX.internal-StreamThread-1-restore-consumer, groupId=null] Subscribed to partition(s): APP_ID-SATE_STORE-changelog-PARTITION_NUMBER
Most of time it is only one partition being reported, but sometimes it’s more. Logs are emitted by only one stream processor’s instance, from a single thread. When rebalancing is finished, everything gets back to normal. How to interpret this situation? We are using Kafka 2.6.

Hi @yagiel,

That log message originates from the restore consumer that is used to restore the changelogs. The restore consumer gets partitions manually assigned. Manual assignment (i.e. KafkaConsumer#assign()) does not support incremental assignments. So each time a new changelog partition is added to or removed from the restore consumer during a rebalancing, we need to manually assign to the restore consumer the complete set of currently restoring changelog partitions plus or minus the added or removed changelog partitions. Thus, all this repetitive log messages.

In conclusion, the log messages are expected and harmless.

Best,
Bruno

2 Likes

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.