Hello Confluent Community,
I am looking for help, in my company we have kafka stream application. but last week we deleted all topics for data clean up and kafka consumer created new topics, but since then it is rebalancing, now application is not stable. we tried many ways to solve this issue but we have no luck so far. so looking for help.
We have apache kafka 2.7.0 version with below configuration
Cloud: AWS
Brokers: 4 AWS MSK
Schema Registry : 2 (Load Balancer)
Kafka Connect : 1
Kafka Rest Proxy 1
Consumer: 8
Cluser level configuration
auto.create.topics.enable=false
delete.topic.enable=true
default.replication.factor=3
min.insync.replicas=1
num.io.threads=8
num.network.threads=5
num.partitions=1
num.replica.fetchers=2
replica.lag.time.max.ms=30000
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
socket.send.buffer.bytes=102400
unclean.leader.election.enable=true
zookeeper.session.timeout.ms=18000
log.retention.hours=24
log.segment.bytes=104857600
log.cleaner.min.cleanable.ratio=0.30
message.max.bytes=10485760
Error we get:
[iq-stream-core-017e969c-1aac-4444-8d0d-0ccfdf18b3c1-StreamThread-1] INFO org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor - stream-thread [iq-stream-core-017e969c-1aac-4444-8d0d-0ccfdf18b3c1-StreamThread-1-consumer] All members participating in this rebalance:
[iq-stream-core-017e969c-1aac-4444-8d0d-0ccfdf18b3c1-StreamThread-1] INFO org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor - stream-thread [iq-stream-core-017e969c-1aac-4444-8d0d-0ccfdf18b3c1-StreamThread-1-consumer] No followup rebalance was requested, resetting the rebalance schedule.