Prepare for DAG based workflows

[lalitpagaria]

Working towards full DAG base workflows #8

Following things are part of these Epic -

• Introduce generic node structure
• Introduce different types of node structure (Observer, Informer, Analyzer, Preprocessor, Router etc)
• Constraint Each Node should be stateless, hence each incoming request will bring Node config with it (be it either via incoming payload, file, DB, Cache etc)
• Constraint Each Node should have reproducible properties ie avoid procedure call. Hence anyone can create setup locally or remotely.
• Unify input/output payload (first part if done in this ticket Unify input/output payload for each nodes #144)
• Optional Payload should able to accept text, image, vectors, audio, file etc.
• Introduce payload convertor
• Introduce channel structure
• Introduce different types of channel structure (Local, Rest, Messaging, Binary Protocol, etc)
• Each channel will receive/emit payload and can have a payload convertor. Payload convertor stage will be based on where channel is used ie input or output. For example if listening to Kafka events then payload convertor stage will sit after consumer and if sending payload via Rest then convertor will sit before Rest call.
• Each Node can have multiple incoming and outgoing channels. For example analyzer node can receive payload from http request, events or locally via procedure call and similarly it can sent payload to multiple http endpoints.

@GirishPatel @shahrukhx01 @sanjaybharkatiya
It is not compressive list, but let's list out the points.

[lalitpagaria]

Let's plan first version for release v0.1. This release will have many breaking changes hence better to increase minor version.

If we can leverage UI of https://github.com/orchest/orchest then it will be nice addition to Obsei.

[shahrukhx01]

@lalitpagaria https://github.com/orchest/orchest are using typescript, we need to make a decision about how we want to proceed with the UI. We may have to do some changes to adapt Orchest, that may avert our focus on Obsei itself. Second option is either continue with streamlit or use Justpy. Drag and drop interfaces are possible with Justpy and tailwind, for instance this example can easily be replicated with Justpy https://github.com/ananikets18/Drag-and-Drop-card-using-HTML-TailwindCSS-

[lalitpagaria]

@shahrukhx01 I agree that UI is not top priority for platform point of view. But I feel it will be good for user adoption point of view (I my view, but I might be wrong).

In past I experiment with streamlit but it did not clicked. Yes one point is to host demo on our own server (Now it is possible using Oracle cloud), with proper rate limiting to prevent users from abusing demo.

I also tried to build demo using Justpy but I felt it too complex. It could be due to that fact that front end is not my strong area. If you have experience, is it possible to prepare drag and drop based workflow creator? Main idea to generate workflow yaml for user. For first version we can support 1 time execution.

[shahrukhx01]

@lalitpagaria I can try creating a simple UI demo using Justpy once I get time. Meanwhile, we can decide how we want to design our backend codebase since that's more important.

[lalitpagaria]

Above issue for backend codebase only. For UI let's create separate issue :)

[shahrukhx01]

@lalitpagaria sure you can assign me that.

[GirishPatel]

For creating architecture diagram we can use Archimate.
We can use PlantUML for create diagram.