Hi all
Does anyone have a working example of checking/integrating schema registry via Golang and with msgs’s in a Protobuf’s packaging.
G
Hello,
@riferrei created a Golang SR client, and has some examples in the repo. GitHub - riferrei/srclient: Golang Client for Schema Registry
Hope that helps,
Dave
1 Like
hi there
thanks for this, it’s Avro example though,
will try and figure out how much of it “swops” over to Protobuf as is.
G
Could not find a blog post about Golang and the binary format. One of the things different from using Avro is that after the ‘magic byte’ and the 4 bytes for the encoded ID, there are one or more bytes to encode the message used within the proto file. If the first (or only) message is used, this will be a singe byte with value ‘0’.
1 Like
while back, can’t find the thread now… Robin shared a example also… that was JSON based.
would be nice to figure it out, then post the working example here.
let me see what I can figure out.
G
… working through it, contacted Ricardo for example of the right file format to read in if the schema does not exist, would be great if say control centre can dump a saved registry to this file format. will go ask in the control centre dev corner
G
… seems simply changing 
schema, err = schemaRegistryClient.CreateSchema(topic, string(schemaBytes), srclient.Avro, false)
TO
schema, err = schemaRegistryClient.CreateSchema(topic, string(schemaBytes), srclient.Protobuf, false)
This still works, up to:
native, _, err := schema.Codec().NativeFromTextual(value)
This only seems to be working for Avro, similar if you follow the package path:
GOPATH:/pkg/mod/github.com/linkedin/goavro/v2@v2.9.7/codec.go:388 +0x26
Lets wait for Ricardo.
G
so Ricardo is not responding.
Getting a null pointer on the schema.Codec() code…
I got the code up to this point to do a schema registration into CC, and if I click on validate CC thumbs up on it… which is great.
(figthting fires on multiple fronts at the moment so getting to this in between the other platform stuff being discussed.)
G
Hey, sorry for the delay in responding. I actually did create an example of the Schema Registry client for Go (thanks @daveklein for pointing this out) with Protobuf:
3 Likes
thanks for the updated example, will have a look.
G
Ricardo.
When your source ,proto file also contains a 2nd structure and a service definition, as to be used in a response, (see below), whats your advise, as you can’t now simply reuse this proto file for the schema registry check/register of new schema, as the 2nd structure and service definition is not part of the schema registry you’re registering.
G
//
// cd <app_root>
// protoc ./person.proto --go_out=plugins=grpc:person
//
syntax = "proto3";
package person;
option go_package = "/;person";
message Message {
string dest = 1;
string uuid = 2;
string path = 3;
string seq = 4;
string alpha = 5;
string last = 6;
string first = 7;
string birthday = 8;
string gender = 9;
string email = 10;
string street = 11;
string state = 12;
string city = 13;
string zip = 14;
string ccnumber = 15;
string date = 16;
string latitude = 17;
string longitude = 18;
string dollar = 19;
string note = 20;
}
message Response {
string uuid = 1;
string path = 2;
string note = 3;
}
service DataSaver {
rpc PostData(Message) returns (Response) {}
}
what’s your advise, as you can’t now reuse this proto file for the schema registry check/register of new schema, as the 2nd structure and service definition is not part of the schema registry you’re registering.
Could you keep them in separated .proto files? IMO this practice applies to pretty much everything in programming: if something has a broader usage throughout your code, they need to be treated as individual entities. I’d actually go further and keep your messages and services separated.
1 Like
Actually not… as the service definition is directly part of the protobuf message, it’s directly tied to each other.
For my use I’ve copied the protobuf message struct out to a file called person.pb which I then reference in the schema registry check/register, which implies I just have to make sure I copy the structure out should it ever be changed…
G
I see. Well, presumably, Schema Registry supports the notion of schema references, as you can read here. So I’m not sure about the actual problem you’re describing regarding “check/register”. If this is throwing you an error, then perhaps might be a problem with the Schema Registry version?
1 Like
the check/register I referred to was where in your code you checked if the topic has a recorded schema, and if it does not then you create/register one, using the proto file. I simply don’t use my proto file I use a file where I copied the structure into, so that I can exclude that Response message and the service definition.
Guess if I want to be pedantic, the Response is also a message going onto a topic so it also needs to be copied to it’s own .pb file to have an associated schema registry entry.
G
Got it. I understand now your problem. I think it has more to do with how you have broken down your .proto file to use only the parts interested to the specific use case right? In any case, the piece of code that you’ve referred from my example:
schema, err := schemaRegistryClient.GetLatestSchema(topic, false)
if schema == nil {
schemaBytes, _ := ioutil.ReadFile(schemaFile)
schema, err = schemaRegistryClient.CreateSchema(topic, string(schemaBytes), "PROTOBUF", false)
if err != nil {
panic(fmt.Sprintf("Error creating the schema %s", err))
}
}
Here, the design intention is to concurrently update the last schema on the registry based on the record written in Kafka that originated from the local schema. This mimics what the serializer found in most of Confluent’s implementations, making sure that the schema is ever updated based on what the client applications are sending to Kafka. It’s a form of keeping the consistency so other teams can read the new records and knowing what the schema used to serialize them was. That said, you don’t actually need to follow this design if keeping the schema updated in the registry is not a requirement for your use case.
1 Like
This is me exploring the technology and how it all glues together, as more experience is consolidated different methods/designs will be explored. For now, it’s just getting 4 wheels on the car, engine in and down the road.
Of course if we follow GiGo… we need to make sure the data is correct/structured so once I got the flows working I now decided, ok lets play with this schema registry modules, and of course I made it complicated for myself, I’m using Protobuf’s and not simple JSON or Kafka native/standard (Avro).
(of course to make the world even more interesting, some bits are in docker - cp-app-in-one (if I can get that working the way I want, otherwise it will be native on OS as the local zip file/drop service start), the producer is outside/native on OS, and the kafka topic consumer is inside minikube, this consumer then pushes the messages via a gRPC/Protobuf call to a 2nd app which played with various messages and then persists them in to one of a couple of “new” database options (all nonSQL/unstructured JSON stores), aka not Oracle (thats my long term background, although, for sh$t and giggles might just drop the messages into a Oracle DB also into their new JSON data type).
G
1 Like
Thanks for the updated PB example. it’s on schedule to be played with tonight, while dinner is consumed and some relaxing in front of the TV.
G
PS: you can ignore the direct email, as it’s this example I was asking for 
1 Like
… I don’t see code in your producer where you check the compatibility of the compiled proto message (value) compiled against the registered schema for the topic.
My logic says this checking should be done at producer (producers responsibility to make sure msg standards are adhered to) side, ensuring all messages on topic adhere to the schema registry, leaving the consumer to simply read whats there.
G