Confluent schema registry 7.6.0 issue with multiple instances while publishing schemas

Schema Registry
1

I have installed confluent schema registry based on “packages.confluent.io/archive/7.6/confluent-7.6.0.zip” on my on-premise
Kubernetes cluster,and pointing it to _schemas topic in Confluent Cloud Kafka.

When I run a single pod instance of schema registry (SR), I am able to publish new schemas to the SR consistently.
But when I run more than one pod instance, and the request hits the follower instance,the request fails with the error message like below,complaining about the follower instance unable to forward the request to leader due to "Unknown host " exception.

Error logs:
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)\n\tat java.base/java.lang.Thread.run(Thread.java:840)\nCaused by: io.confluent.kafka.schemaregistry.exceptions.SchemaRegistryRequestForwardingException: Unexpected error while forwarding the registering schema request to https://dp-schema-registry-7dc6cbdb5-kr2rk:1443\n\tat io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry.forwardRegisterRequestToLeader(KafkaSchemaRegistry.java:1248)\n\tat io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry.registerOrForward(KafkaSchemaRegistry.java:880)\n\tat io.confluent.kafka.schemaregistry.rest.resources.SubjectVersionsResource.register(SubjectVersionsResource.java:412)\n\t… 73 more\nCaused by: java.net.UnknownHostException: dp-schema-registry-7dc6cbdb5-kr2rk

Basically the pods corresponding to the different SR instances in the same cluster are trying to reach each other using pod name and hence unable to resolve the target pod.If I manually try to connect to one pod from the other using the pod IP,they are able to connect.

My question is how do I let the pods interact with each other using pod Ip instead of pod name? I tried setting SCHEMA_REGISTRY_HOST_NAME to status.podIP in env variable section of pod,and it is able to set the variable to pod IP,but looks like the SR is unable to understand this property and assign it to the pod host names.

Below are some of my settings in the schema registry properties:
listeners–> I have set it to “0.0.0.0:1443”
host.name → I have not set this

Please suggest how do I let the follower pod connect/forward to leader pod of SR in above scenario.

2

hey @bijoy.computers

I assume you’re not using Confluent for Kubernetes?
how did you create the pods?

in general I think you have to set host.name to the pods IP.

best,
michael

3

Hi @mmuehlbeyer

Thanks for your response.

I created a docker image using the confluent schema registry code available under https://packages.confluent.io/archive/7.6/confluent-7.6.0.zip and used it to instantiate pods on my onpremise K8 cluster.

By setting host.name to podIP,I assume you are referring to host.name property in schema-registry.properties file.That will not work for me because podIP may change if the existing pod gets killed and a new one is spinned up. So, I would ideally need a way to dynamically fetch the podIP and assign as host.name during the pod start up.

4

I see
why not using CfK or a similar operator which would take care such things?

https://bitnami.com/stack/schema-registry/helm

best,
michael

5

Hi,

We did not explore that option yet.However,currently it may be bit late for us to explore alternative options as we are in user testing phase.

Thanks

6

Hi,

I see.
I assume you’re looking for something like this

Best,
Michael

7

Hi,

Yes,I used code like below to set an env variable SCHEMA_REGISTRY_HOST_NAME to the pod IP dynamically:

- name: SCHEMA_REGISTRY_HOST_NAME
          valueFrom:
            fieldRef:
              fieldPath: status.podIP

But looks like though the above variable is being set to pod IP,but the pods are still using the pod names to communicate with one another and failing with error I mentioned in the logs snippet.

So,I am looking for either:

  1. a way for pods to communicate with each other using IP addresses instead of pod names.
    or
  2. A way for pods to be able to reach each other using pod names (via some DNS settings,etc.).

Thanks,
Bijoy

8

Hi @bijoy.computers

thanks for letting us know.

host.name in schema. registry.properties is not set, right?

9

I think your looking for something similar to this

though I would highly recommend to use an operator or similar to
run such things.
it will ease your pain to get this things up and running.

there are some archived helm charts available as well

disclaimer: not recommend for production usage

nevertheless I won’t recommend your current setup for prod usage :wink: