A new repo for shared libraries mon-deploy-infra

[mocsharp]

I suggest setting up a new GitHub repo to store shared libraries used across MONAI Deploy sub-systems. E.g., we need a library that provides an adapter pattern to publish/subscribe to messages (RabbitMQ) and a library for reading/writing files on virtual storage (mapped volumes, cloud storage, etc.).

💗 Option 1: single repo to store all share libs

• monai-deploy-infra

😄 Option 2: multiple repos, each repo contains a single lib

• monai-deploy-message-broker
• monai-deploy-virtual-storage

👍 Option 3: use main monai-deploy repo and create sub dirs

• monai-deploy

[whoisj]

are you suggesting cloning an existing repo as-is, or creating a repo for the abstraction layer, deployment tools, etc?

[mocsharp]

new repo(s) to store the shared libraries, which includes the abstraction layer and actual implementations.

[MMelQin]

Have we settled on RabbitMQ vs Kafka? It has been awhile since I compared and used both, though both can satisfy our use cases, e.g. topics, pub/sub, performance, and scalability.

Both RabbitMQ and Kafka provide client library, in the languages that MONAI Deploy may choose to use, e.g. Python and C# (I even used Scala client of Kafka in my past life). Accessor layer business logic for MQ should and is super simple, with the already provided MQ specific and language specific client lib, what is the need for a MONAI Deploy lib, e,g what more does it do, and in what language?

[mocsharp]

Either one is fine; we will have an adapter that allows users to easily plug in their own or to support native cloud services.

I remember @joshliberty mentioned using RabbitMQ so we can use that as the default. I have also started a draft of the message broker design in #44 .

[MMelQin]

both RabbitMQ and Kafka were on the table.

[MMelQin]

People can plug in their MQ, and all the accessors etc, so it is simple, then why the need to have a separate repo to host a simple lib?
Also, key platform components, e.g. MQ, should be determined, not left wide open.

[mocsharp]

We need an adapter for the sub-systems to use; the adapter defines the abstraction layer to enable plug-ins to the MQ clients. The library would also be used across sub-systems, assuming we use C# for both IG &WM.

Also, have to consider deployments for both on-prem & cloud. For on-prem, one can use RabbitMQ, Kafka, or even MSMQ if all on one box. For the cloud, one may use Azure Service Bus, Google PubSub, etc...

[MMelQin]

MSMQ? on Linux? It is easier link MSMQ to RabbitMQ if one wants to make that happen.
In any case, it is the message definition and the message management that is important, not the mechanism of accessing the Q. A message accessor layer is needed as the glue between app internal data structs and the message content.

[MMelQin]

Storage is yet another concern that has not gotten clear definition yet, e.g. provided as a storage service or open to client for direct access? The later may be simple, with multiple protocols supported, e.g. CIFS/HTTP/File, but it does not provide good separation of concern. With the storage service, performance or latency could be a good concern even though security and data compartmentalization can be maintained easier.

Before the architecture or high level design are settled, discussing new repo may seem lacking some facts.

[mocsharp]

At least we can start an adapter that read/write local FS/mounted volume.

[MMelQin]

What is really needed is the design of the file space, virtual file system, so multiple processes, distributed over different servers, can access the files consistently.

[mocsharp]

Yes and we need a repo to store that design too. 😁

[dbericat]

Things that are common to all the sub-systems should go to the monai-deploy main repo. You can create a folder with libs, or utils, or similar.

When we have something for storage and we move it to its own repo, we would move the design and reqs there.

@ericspod how does MONAI do these things?
@hshuaib90 organizationally, what is your take?

[mocsharp]

Same repo (monai-deploy) would work too. I have added option 3.

[dbericat]

I voted already! Thanks Victor!

[dbericat]

We should really think hard about how many repos we want to have. Too many can become problematic over time. So option 2 need to be justified. IMO is not today unless someone lays out clear benefits.

[MMelQin]

Both option 1 and 2 needs to have sufficient justification.

[ericspod]

We don't have a requirement right now to share libraries between repos, what we use is in the main repo and makes the most sense there. I vote to use the main deploy repo as this make sense as a common central place for shared things. The subdirectory should be descriptively named with a README.md file to make it clear what is stored there and specifically that it's not all there is to deploy.