Hello!
We’ve encountered an interesting situation in Kafka Broker/Connect/Schema registry TEST cluster. We have a 3-node Broker cluster and 1 node of Connect and 1 node of Schema registry. Connect have a URLs of the Brokers and Schema registry in it’s configuration. For the testing purposes we’ve started additional schema registry on a different node connected to the Broker cluster. Both Schema registries had a same group.id, but in Connect configuration only the URL of the first Schema node was specified.
So, to the summary, for understanding the further situation: 3-node Broker cluster, 1 node Connect looking on the 1st Schema node, and 2 separated Schema regestries connected to the Kafka broker cluster, but with the same group.id.
We’ve added new Kafka source connector, when connector grab’s the messages from the database and writes it to the topic consumer throws the following:
org.apache.nifi.schema.access.SchemaNotFoundException: Could not find Schema with id 102396 from the Confluent Schema Registry located at http://<schema-registry-1st-node>:8081"
Messages catched by default Conductor’s consumer got the following ERR: Error retrieving Avro unknown schema for id 102396. After checking we’ve found that all related schemas was created on the 2nd Schema registry node.
We’ve fixed this issue in our TEST environment by changing the Schema regestries configs, but want to understand - how exactly Broker, Connect and Schema registry transfering data and interacting with each other? Tell me if You need more information, I’ll provide it.
Thank you!
Are both the Schema Registry instances within same DC? and what about the _schemas topic? Do you have 2 separate backend topics or same topic shared between both the Schema Registry instances? and I assume master=true for both the instances?