Improve kafka adapter

[KhudaDad414]

Reason/Context

The current implementation of kafka adapter provides the bare minimum functionality and I think we can improve upon that by using bindings.

What are bindings

AsyncAPI is protocol agnostic, it means that it supports a wide range of protocols and doesn't "assume" the protocol when you are writing your AsyncAPI file.
but if it doesn't assume protocols, how are we supposed to write our file accustomed to an specific protocols?
Behold, the answer is bindings.
If you look at kafka bindings you will see a wide range of options that can be used to customize every part of your kafka client.

The Problem

Currently kafka adapter ignore lots of this options. for example. instead of using clientId from bindings for clinetId it just sets it to glee-client.

The Solution

Kafka bindings should be examined and we should support as much features as the bindings support.

[KhudaDad414]

/gfi ts

[shubham-y]

Hey @KhudaDad414 , I would like to work on this issue

[KhudaDad414]

@shubham-y sure. Just let me know if you have any questions 🙂

[shubham-y]

I see there is a similar draft PR #381 opened 2 months ago. So is there any discussion that I can refer to from that time?

[KhudaDad414]

Not that I know of. this issue is also kinda related to #27

[shubham-y]

Hey @KhudaDad414 , I went through the current implementation of kafka adapter and kafka bindings document.
I have a doubt
In Server Binding Object there is a field schemaRegistryVendor, which vendors will we be supporting?
Because I could only find this package for confluent @kafkajs/confluent-schema-registry which uses the avsc library for Avro encoding/decoding.
Also, will we be also supporting other serializers than Avro?

For further doubts in the future, please let me know which platform is preferred for such doubts Slack or GitHub comments

[KhudaDad414]

Hi @shubham-y,

In Server Binding Object there is a field schemaRegistryVendor, which vendors will we be supporting?

Based on a recent discussion that I had with @fmvilas, We think it is better to ignore the schemaRegistryVendor for now. This is something that needs to be done in parser-js. we basically get the parsed schema in glee. so we do not need to worry about it.

[KhudaDad414]

Hey @shubham-y, Have you started working on this issue? 😄 @Ruchip16 can continue to work on it or give a hand if you need some help.

[shubham-y]

Hey @shubham-y, Have you started working on this issue? 😄 @Ruchip16 can continue to work on it or give a hand if you need some help.

Hey @KhudaDad414, I had started working on it but had to take a break due to unavoidable personal circumstances. Sorry, I did not communicate earlier. I have resumed working on it and it will great if @Ruchip16 can help me with this as I am struggling a bit due to limited docs/tutorials for Glee.

[Ruchip16]

Hey @shubham-y, Have you started working on this issue? smile @Ruchip16 can continue to work on it or give a hand if you need some help.

Hey @KhudaDad414, I had started working on it but had to take a break due to unavoidable personal circumstances. Sorry, I did not communicate earlier. I have resumed working on it and it will great if @Ruchip16 can help me with this as I am struggling a bit due to limited docs/tutorials for Glee.

hey I can initiate a PR for you and you can take it from there, if that sounds good?

[shubham-y]

Hey @shubham-y, Have you started working on this issue? smile @Ruchip16 can continue to work on it or give a hand if you need some help.

Hey @KhudaDad414, I had started working on it but had to take a break due to unavoidable personal circumstances. Sorry, I did not communicate earlier. I have resumed working on it and it will great if @Ruchip16 can help me with this as I am struggling a bit due to limited docs/tutorials for Glee.

hey I can initiate a PR for you and you can take it from there, if that sounds good?

Sounds good. Thank you for that.

[kaushik-rishi]

Me and @shubham-y have agreed to work on it together 🙂

UPDATE (May 26)
Ruchi, said she'll work on it and we are not working on it.

[sudoshreyansh]

Hi, is this issue available to be taken?

[kaushik-rishi]

@Ruchip16 is working on it @sudoshreyansh 🙂

I wanted to take it up but she said she was working on it.

[sudoshreyansh]

@kaushik-rishi 👍

[oviecodes]

/progress 10 I'm starting to work on this.

[Souvikns]

any updates on this @oviecodes?

[oviecodes]

@Souvikns I'm sorry I got so busy, please can you take this, I'll join you once I'm free. Thanks

[Amzani]

@KhudaDad414 will take a look

[KhudaDad414]

/progress 10
started looking into the Kafka bindings.

[oviecodes]

@KhudaDad414 any updates on this?

[KhudaDad414]

@oviecodes I am busy with spec v3 support in glee. might do it later. I don't have any progress in it. so feel free to pick it up.

[oviecodes]

Alright man 👍...

[oviecodes]

@fmvilas @Souvikns @KhudaDad414 could anyone please explain this issue further?

[KhudaDad414]

@oviecodes to summarise, you have to look through the bindings here: https://github.com/asyncapi/bindings/tree/next-major-spec/kafka, parse them from the asyncapi document and apply them whenever necessary. we are using https://kafka.js.org/ in Glee, so all you have to do is parse the bindings and pass them to the kafkajs. that's it.

[Ruchip16]

Hey @KhudaDad414 can i start working on this again, I promise to complete it this time 😅?

[oviecodes]

@Ruchip16 wanna work on this one together? 😃

[Ruchip16]

yes yes suree 😊

[oviecodes]

that's great, we'll open a pull request to update our progress on this issue 🙂

[oviecodes]

/progress 20 looking into kafka bindings and kafka.js

[oviecodes]

The Kafka Adapter has to be updated to use spec v3. I'll open an issues as appropriate @Souvikns @KhudaDad414

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity 😴

It will be closed in 120 days if no further activity occurs. To unstale this issue, add a comment with a detailed explanation.

There can be many reasons why some specific issue has no activity. The most probable cause is lack of time, not lack of interest. AsyncAPI Initiative is a Linux Foundation project not owned by a single for-profit company. It is a community-driven initiative ruled under open governance model.

Let us figure out together how to push this issue forward. Connect with us through one of many communication channels we established here.

Thank you for your patience ❤️

[KhudaDad414]

still relevant.