Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ProtoBuf & JSON Schema support #41

Open
tPl0ch opened this issue Oct 1, 2020 · 2 comments · May be fixed by #50
Open

ProtoBuf & JSON Schema support #41

tPl0ch opened this issue Oct 1, 2020 · 2 comments · May be fixed by #50

Comments

@tPl0ch
Copy link
Collaborator

tPl0ch commented Oct 1, 2020

No description provided.

@nickpoulos
Copy link

nickpoulos commented Oct 16, 2020

Looking forward to this PR, and would love to help implement. Is there a slack/discord/forum for discussion anywhere?

I was in a bit of a time crunch on a current project and did not realize this library did not support Protobuf formatted messages. In order to get around it for now we ignore SchemaRegistry, precompiled the protobufs, and then had to strip out the Magic Byte and other SchemaRegistry headers, but it does decode now. Obviously this is not ideal and kills the purpose of SchemaRegistry, so the sooner I can fix the better.

I am curious how you would implement this, I am not 100% sure how the Confluent deserializers operate behind the scenes. Are they compiling the Protobufs on the fly, caching, dynamically load the classes after with reflection?

@tPl0ch
Copy link
Collaborator Author

tPl0ch commented Oct 16, 2020

I am planning on separating these concerns. On the HTTP side the only thing we need is a schema type and a string representation of a given schema.

The hard part is that I would need to change the Registry interface to not return an AvroSchema, but a more general Schema that contains the type and the string representation so that downstream clients can make decisions on what to do with it.

There are multiple options that are currently in my head:

One Regisitry interface and a generic Schema representation

This is currently my preferred option. The HTTP side is very easily changed, and internally it would just build a generic Schema with a SchemaType and a string schema. I don't think that validation is necessary here, this concern should be handled further downstream.

Multiple Registry interfaces

We would have AvroRegistry, ProtoBufRegistry and JsonRegistry interfaces and downstream clients can inject there relevant implementations. This library would provide default implementations. While this is possible, it makes the general usage of this library more complicated.

Others?

I am open for suggestions, I started the SchemaType implementation on the issues/41 branch:
https://github.com/flix-tech/schema-registry-php-client/tree/issues/41/src/Schemas

@tPl0ch tPl0ch linked a pull request Oct 16, 2020 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants