Hello, I have problem with schema-registry, I run it on clean kubernetes cluster where installed is only Kafka Cluster, zookeeper and schema-registry. Brokers are running, I can connect to them and send/read messages from topics.
When schema-registry is starting I can see info in logs as below:
[2023-08-03 15:00:32,868] INFO [Schema registry clientId=sr-1, groupId=schema-registry] (Re-)joining group (io.confluent.kafka.schemaregistry.leaderelector.kafka.SchemaRegistryCoordinator)
[2023-08-03 15:00:32,975] INFO [Schema registry clientId=sr-1, groupId=schema-registry] Request joining group due to: need to re-join with the given member-id: sr-1-8592d5fc-2b0c-47a4-bc49-dda8d8496bf8 (io.confluent.kafka.schemaregistry.leaderelector.kafka.SchemaRegistryCoordinator)
[2023-08-03 15:00:32,976] INFO [Schema registry clientId=sr-1, groupId=schema-registry] Request joining group due to: rebalance failed due to 'The group member needs to have a valid member id before actually entering a consumer group.' (MemberIdRequiredException) (io.confluent.kafka.schemaregistry.leaderelector.kafka.SchemaRegistryCoordinator)
[2023-08-03 15:00:32,976] INFO [Schema registry clientId=sr-1, groupId=schema-registry] (Re-)joining group (io.confluent.kafka.schemaregistry.leaderelector.kafka.SchemaRegistryCoordinator)
Schema-registry after few restarts starts working but I don’t understand why it happens on clean clusters and I have to wait about 30-60 minutes when it will be ready. I’m using confluentinc/cp-schema-registry:7.3.2 . Also in brokers logs I don’t see any useful informations.
Broker logs:
[2023-08-03 14:38:31,688] INFO [ReplicaFetcherManager on broker 1] Removed fetcher for partitions Set(_schemas-0) (kafka.server.ReplicaFetcherManager)
[2023-08-03 14:38:31,776] INFO [LogLoader partition=_schemas-0, dir=/kafka-logs/kafka] Loading producer state till offset 0 with message format version 2 (kafka.log.UnifiedLog$)
[2023-08-03 14:38:31,777] INFO Created log for partition _schemas-0 in /kafka-logs/kafka/_schemas-0 with properties {cleanup.policy=compact} (kafka.log.LogManager)
[2023-08-03 14:38:31,780] INFO [Partition _schemas-0 broker=1] No checkpointed highwatermark is found for partition _schemas-0 (kafka.cluster.Partition)
[2023-08-03 14:38:31,780] INFO [Partition _schemas-0 broker=1] Log loaded for partition _schemas-0 with initial high watermark 0 (kafka.cluster.Partition)
[2023-08-03 14:38:47,095] INFO [GroupCoordinator 1]: Dynamic member with unknown member id joins group schema-registry in Empty state. Created a new member id sr-1-5ac668c4-8e2a-4006-b200-478853d1cfb6 and request the member to rejoin with this id. (kafka.coordinator.group.GroupCoordinator)
[2023-08-03 14:38:47,193] INFO [GroupCoordinator 1]: Preparing to rebalance group schema-registry in state PreparingRebalance with old generation 0 (__consumer_offsets-29) (reason: Adding new member sr-1-5ac668c4-8e2a-4006-b200-478853d1cfb6 with group instance id None) (kafka.coordinator.group.GroupCoordinator)
[2023-08-03 14:43:47,227] INFO [GroupCoordinator 1]: Member sr-1-5ac668c4-8e2a-4006-b200-478853d1cfb6 in group schema-registry has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2023-08-03 14:43:47,272] INFO [GroupCoordinator 1]: Group schema-registry with generation 1 is now empty (__consumer_offsets-29) (kafka.coordinator.group.GroupCoordinator)
[2023-08-03 14:43:47,282] INFO [GroupCoordinator 1]: Dynamic member with unknown member id joins group schema-registry in Empty state. Created a new member id sr-1-86fbc1af-d01d-4d30-99bf-36af91b5e475 and request the member to rejoin with this id. (kafka.coordinator.group.GroupCoordinator)
[2023-08-03 14:43:47,285] INFO [GroupCoordinator 1]: Preparing to rebalance group schema-registry in state PreparingRebalance with old generation 1 (__consumer_offsets-29) (reason: Adding new member sr-1-86fbc1af-d01d-4d30-99bf-36af91b5e475 with group instance id None) (kafka.coordinator.group.GroupCoordinator)
[2023-08-03 14:48:47,286] INFO [GroupCoordinator 1]: Member sr-1-86fbc1af-d01d-4d30-99bf-36af91b5e475 in group schema-registry has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2023-08-03 14:48:47,287] INFO [GroupCoordinator 1]: Group schema-registry with generation 2 is now empty (__consumer_offsets-29) (kafka.coordinator.group.GroupCoordinator)
[2023-08-03 14:49:38,680] INFO [GroupCoordinator 1]: Dynamic member with unknown member id joins group schema-registry in Empty state. Created a new member id sr-1-aa21eccc-a542-452b-b34e-6a4db2ac3d7c and request the member to rejoin with this id. (kafka.coordinator.group.GroupCoordinator)
[2023-08-03 14:49:38,767] INFO [GroupCoordinator 1]: Preparing to rebalance group schema-registry in state PreparingRebalance with old generation 2 (__consumer_offsets-29) (reason: Adding new member sr-1-aa21eccc-a542-452b-b34e-6a4db2ac3d7c with group instance id None) (kafka.coordinator.group.GroupCoordinator)
[2023-08-03 14:54:38,771] INFO [GroupCoordinator 1]: Stabilized group schema-registry generation 3 (__consumer_offsets-29) with 1 members (kafka.coordinator.group.GroupCoordinator)
[2023-08-03 14:54:38,775] INFO [GroupCoordinator 1]: Member sr-1-aa21eccc-a542-452b-b34e-6a4db2ac3d7c in group schema-registry has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2023-08-03 14:54:38,776] INFO [GroupCoordinator 1]: Preparing to rebalance group schema-registry in state PreparingRebalance with old generation 3 (__consumer_offsets-29) (reason: removing member sr-1-aa21eccc-a542-452b-b34e-6a4db2ac3d7c on heartbeat expiration) (kafka.coordinator.group.GroupCoordinator)
[2023-08-03 14:54:38,777] INFO [GroupCoordinator 1]: Group schema-registry with generation 4 is now empty (__consumer_offsets-29) (kafka.coordinator.group.GroupCoordinator)
[2023-08-03 14:54:38,881] INFO [GroupCoordinator 1]: Dynamic member with unknown member id joins group schema-registry in Empty state. Created a new member id sr-1-31aae003-2906-4373-b271-d5d15c64e303 and request the member to rejoin with this id. (kafka.coordinator.group.GroupCoordinator)
[2023-08-03 14:54:38,967] INFO [GroupCoordinator 1]: Preparing to rebalance group schema-registry in state PreparingRebalance with old generation 4 (__consumer_offsets-29) (reason: Adding new member sr-1-31aae003-2906-4373-b271-d5d15c64e303 with group instance id None) (kafka.coordinator.group.GroupCoordinator)
[2023-08-03 14:59:38,968] INFO [GroupCoordinator 1]: Member sr-1-31aae003-2906-4373-b271-d5d15c64e303 in group schema-registry has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2023-08-03 14:59:38,968] INFO [GroupCoordinator 1]: Group schema-registry with generation 5 is now empty (__consumer_offsets-29) (kafka.coordinator.group.GroupCoordinator)
[2023-08-03 15:00:32,972] INFO [GroupCoordinator 1]: Dynamic member with unknown member id joins group schema-registry in Empty state. Created a new member id sr-1-8592d5fc-2b0c-47a4-bc49-dda8d8496bf8 and request the member to rejoin with this id. (kafka.coordinator.group.GroupCoordinator)
[2023-08-03 15:00:33,065] INFO [GroupCoordinator 1]: Preparing to rebalance group schema-registry in state PreparingRebalance with old generation 5 (__consumer_offsets-29) (reason: Adding new member sr-1-8592d5fc-2b0c-47a4-bc49-dda8d8496bf8 with group instance id None) (kafka.coordinator.group.GroupCoordinator)
[2023-08-03 15:05:33,065] INFO [GroupCoordinator 1]: Member sr-1-8592d5fc-2b0c-47a4-bc49-dda8d8496bf8 in group schema-registry has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2023-08-03 15:05:33,066] INFO [GroupCoordinator 1]: Group schema-registry with generation 6 is now empty (__consumer_offsets-29) (kafka.coordinator.group.GroupCoordinator)
[2023-08-03 15:05:33,069] INFO [GroupCoordinator 1]: Dynamic member with unknown member id joins group schema-registry in Empty state. Created a new member id sr-1-0d5de3f6-1cd6-4fd5-a4c5-bdfd5f762680 and request the member to rejoin with this id. (kafka.coordinator.group.GroupCoordinator)
[2023-08-03 15:05:33,071] INFO [GroupCoordinator 1]: Preparing to rebalance group schema-registry in state PreparingRebalance with old generation 6 (__consumer_offsets-29) (reason: Adding new member sr-1-0d5de3f6-1cd6-4fd5-a4c5-bdfd5f762680 with group instance id None) (kafka.coordinator.group.GroupCoordinator)
My schema-registry configuration:
env:
- name: "SCHEMA_REGISTRY_HOST_NAME"
value: "localhost"
- name: "SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS"
value: "PLAINTEXT://kafka-all-broker:29092"
- name: "SCHEMA_REGISTRY_LISTENERS"
value: "http://0.0.0.0:18081"
- name: "SCHEMA_REGISTRY_KAFKASTORE_INIT_TIMEOUT_MS"
value: "600000"
- name: "SCHEMA_REGISTRY_KAFKASTORE_TOPIC"
value: "_schemas"
- name: "SCHEMA_REGISTRY_DEBUG"
value: "false"
- name: "JAVA_TOOL_OPTIONS"
value: |
-Dlog4j.configuration=file:/etc/app/resources/log4j.xml
Also I noticed when cluster is upgrading and brokers are down schema-registry is also in infinite rebalancing and it ends when I manually delete pod. How can I resolve this issue?