Distributed executor?

[aadrian]

Hi,

Would it be possible to have a distributed executor (or independent of the GUI) so that one workflow can span more than one server?

Thank you.

[yohamta]

Hi 👋

Unfortunately it's currently designed for only one GUI & one scheduler (works as a worker as well).
We'd like to add the multiple worker functionality after some data schema refactoring.

Thanks!

[aadrian]

...after some data schema refactoring

Any details what this might involve?

I could not find any current "schema file" or DB Model - the "data model" seems distributed across packages (I'm not sure what really belongs to it, in order to see what might need refactoring)

[yohamta]

You're right! Dagu doesn't have database schema and writing everything to JSON files for now.

I was lightly thinking that we need to have database to have multiple workers, but yeah it's totally possible to have mulitple workers without having any database.

Dagu server has REST(ish) API endpoint so adding the worker addresses to the scheduler configuration file will make it possible to implement the functionality fairly easily I think.

[yohamta]

I think in the future having database is great but for now it's not necessary IMO.

I am totally fine with adding the multiple worker functionality while keeping the current JSON file database mechanism as is.

[yohamta]

I think this document would be interesting.
Airflow's Celery Executor

[yohamta]

Here's my initial idea about the worker distribution functionality.

[aadrian]

This looks great !

Would it be a little simpler if the GUI server would also dispatch the execution? Since the GUI still needs to interact with the workers anyway?
In the current implementation diagram, the scheduler also does not seem interact directly with the worker.

[yohamta]

Thanks! Yeah, you're right. GUI server and scheduler can be a single process.
I was thinking to add the functionality to the server command by adding --scheduler flag.

[yohamta]

In the current implementation diagram, the scheduler also does not seem interact directly with the worker.

I forgot to add the arrow and fixed it, thanks! =D

[yohamta]

Implementing the functionality would involve these tasks.

• Add Workers option to the server configuration
• Update Scheduler process to interact with Worker servers
• Implement functionality for collecting workflow history data from workers

[aadrian]

Would the GUI/API call the Worker Servers (A), or the Worker Servers call the central GUI/API only (B) , or Bidirectional (C) ?

Implementing the functionality would involve these tasks.
Add Workers option to the server configuration

In this mode, the GUI would not need to work or be rendered, right ?
(To have a "local sub-GUI" for each Worker Server might be an overkill.)

Implement functionality for collecting workflow history data from workers

This would depend on the direction of communication I suppose.
In case of A the Worker Servers would not even need an exposed port, since they would act as clients for the GUI/API ?

[yohamta]

Would the GUI/API call the Worker Servers (A), or the Worker Servers call the central GUI/API only (B) , or Bidirectional (C) ?

I think the GUI server, or maybe even a separate data collection process, will have to call the Worker Servers' API to gather log data.

In this mode, the GUI would not need to work or be rendered, right ? (To have a "local sub-GUI" for each Worker Server might be an overkill.)

You're right! The Worker servers don't need a GUI themselves, even though one's included by default right now. Not sure if we should take it out just yet. 🤔

This would depend on the direction of communication I suppose. In case of A the Worker Servers would not even need an exposed port, since they would act as clients for the GUI/API ?

I might be getting confused here. I was under the impression that in scenario A, the Worker Servers would still need to expose a port to the GUI server for API calls. Can you help me understand what you have in mind?

[aadrian]

I might be getting confused here. I was under the impression that in scenario A, the Worker Servers would still need to expose a port to the GUI server for API calls. Can you help me understand what you have in mind?

Sorry for the confusion - you are right: I was trying to point out that scenario "B" (not "A" - as I mistakenly wrote) seems like the simplest one to implement and also to secure, since it would be a simple "client/server" architecture.

[yohamta]

No worries! You're right, I see what you mean now. Scenario "B" - with the workers pushing the log to the server - does seem like the simplest and most secure way to go. Great catch! That actually sounds like a much better approach 😃

[yohamta]

Here's the revised version of the architecture.

[aadrian]

the revised version of the architecture.

This looks great !

[yohamta]

Thanks!!