Move it to Cloud !

[Arthavruksha]

Yo very nice work done .

I have a feature request that currently Dagu is an Self-hosted runner why don't u guys shift it to cloud and make virtual machine's to run a person's code just like Github Actions.

Thanks

[yohamta]

Thank you! That's a great idea. It is surely possible to develop a cloud service like that. I am thinking about use cases and what advantages it would offer compared to Github Actions.

[Arthavruksha]

Hi yohamta ,

The limitation with github actions is that it has limit of 2000 Minutes of each account per month.

You can remove runtime limit in the cloud version altogether?

I would definitely shift from github actions to that cloud service in ONE day if that's the case

[yohamta]

I see. That is a really interesting idea. With a virtual machine, there would be no limitation. But the CPU/memory capacity of the host would be one. I think it is better to make it a "serverless" service only sharing files between underlying servers. Also, the network connection between users' private networks might be an issue depending on their use cases. I will research further.

If that is the case you would be the first user :)

[Arthavruksha]

Great , I would be the first user in that case

Thanks yohamta

[Arthavruksha]

Any updates over this idea yohamta ?

[yohamta]

It is a very interesting idea, so I would like to move forward with it. Currently, I'm thinking about TODOs that need to be done to make the service.

• How to handle multiple users securely and separately without needing a long-running instance per user
• How to schedule DAGs for multiple users
• How to execute DAGs in separated environments
• What features are needed for the Front end of the cloud service (authentication, billing, etc)
• Landing Page

There will be a lot of tasks even for MVP, so I'm wondering what can be possible to minimize them and shorten the time.

[Arthavruksha]

Ahh got it. The runtime limit will be more than Github actions of 2000 Minutes right?

[yohamta]

Yeah, if I could make it, there would be no runtime limit. May I ask what your use case of Github Actions is? I guess 2000 minutes limit is more than enough for most use cases, so I wonder what is it.

[Arthavruksha1]

This is my other account yohamta . I use this for Git Scraping so the code runs in each hour of the day . Hence 2000 minutes isn't sufficient for me

[yohamta]

I see. By the way, have you considered using self-hosted runners? It seems there's no runtime limit for self-hosted runners.

[fishnux]

I think I forgot to comment on this topic early on. My idea for a Dagu SaaS application is a dashboard to monitor existing Dagu instances, as I have multiple servers with Dagu on it.

[yohamta]

Hey, thank you for the excellent idea! A dashboard SaaS application for Dagu seems both feasible and useful. I'm inclined to give it a try. Can you tell me how many servers you're running and what they're being used for?

[fishnux]

I have about 3 servers, each servers runs a different data processing resource-intensive task, and then the final results are uploaded to a remote location. To be honest, they're not running with Dagu yet, still have to figure out if it's the right choice for this. I've heard of Prefect lately, so I think I'll give that a try, but I just wanted to let you know of that SaaS idea :)

[yohamta]

Oh, I see! Thanks for sharing the interesting details. Prefect could be an excellent choice. It looks like I'll need to make Dagu more robust to compete with it. Haha.

[fishnux]

Edit: I think yohamta removed the question, but he basically asked what feature I'd be looking forward when choosing a workflow orchestration tool

@yohamta I think it's this one:
"Support multiple workers, allowing users to manage multiple servers in a single web server instance."

Instead of running 3 independent Dagu servers, I could run just one and have 3 workers. This is so I could build a DAG with the ability to define in which server a step should run. This differs from the SSH executor, because the SSH executor is prone to network failures, and these tasks take days to complete. Sure I could use something like Mosh (SSH alternative that tolerates network issues), but I'm thinking a worker model might be better long-term. I also would like the ability to move files around servers without writing rsync commands or stuff like that, something like "let Dagu server decide how to move the files".

However, this also means the Dagu SaaS as I initially envisioned wouldn't apply here (if you implement the worker model). Perhaps you'd be looking making something like Prefect Cloud, where basically you bring your own workers, but configuration and DAGs are on Dagu Cloud (analogy to Prefect Cloud).

[yohamta]

Oh, sorry, I thought I may have asked something too intrusive and deleted it. Thank you! That's a really interesting idea. We're currently exploring worker models here.
#463

If we put the DAG server in the cloud and have the workers interact with the cloud, we could potentially build a service similar to Prefect Cloud.