[Draft Proposal, Seeking Feedback] Implement a streamlined or automated pipeline on Cloudmos for deploying applications directly from GitHub onto Akash

[HoomanHQ]

Problem Statement:

Deploying and updating deployments on the Akash Network can be challenging for individuals unfamiliar with Docker images, containerization, and how Akash deployment works through SDL. Additionally, setting up continuous integration and continuous deployment (CI/CD) pipelines using tools such as GitHub Actions and DockerHub can be difficult for beginners and requires manual effort even for updating the deployment. In contrast, competitors like Vercel and Render have a significant advantage in deploying applications directly from GitHub.

Executive Summary:

The proposed solution involves creating a new feature on cloudmos that can build, deploy, and establish CI/CD pipelines for major JavaScript frameworks and other prominent frameworks such as React, Hugo, Astro, Vue, Next.js, Svelte, Flask, Django, Rails, etc. Users would only need to specify the repository URL during deployment or use OAuth login with GitHub. This feature will be natively built and implemented into Cloudmos, enhancing its accessibility and functionality for newcomers to the platform. Overall, the goal of this feature is to enable Cloudmos to deploy applications directly from GitHub onto Akash without the need for creating a Dockerfile or writing SDL. Instead, we will provide a GUI in Cloudmos for performing these tasks effortlessly.

Major Problems Addressed:

• Lack of familiarity with cloud technologies among beginners.
• Absence of automated deployment features comparable to competitors like Vercel, render etc...

Benefits The Product Offers to the Entire Ecosystem:

• Simplified Deployment: Streamlines the deployment process on Akash Network, regardless of users' cloud infrastructure expertise.
• Time Efficiency: Automates deployment tasks, reducing the time required to launch applications.
• Accessibility: Empowers developers of all levels to utilize the Akash network, fostering innovation and decentralization.
• Error Reduction: Automation minimizes the risk of human error during deployment, enhancing reliability.
• Community Growth and Engagement: Streamlining the deployment process can significantly contribute to the growth of the Akash Network community and reduce friction during onboarding.

Implementation Overview:

Phase 1: Development of Automated Deployment System

Tech Stack Needed:

• Docker
• Python Scripting
• TypeScript(React or any other framework)
• GitHub and GitLab REST API and workflows
• UI/UX for GUI and web development for implementation on cloudmos

Implementation Steps:

1. Create Docker Image Template:

   • Develop Docker images for each supported framework (React, Hugo, Astro, Vue, Next.js, Svelte, Flask, Django etc).
   • Configure images to accept repository URLs as environment variables.
   • Include necessary dependencies and build scripts for each framework.

2. Repository Monitoring Logic:

   • Implement Python scripts within the Docker containers to monitor specified repositories.
   • Utilize Git commands via Python to fetch, compare, and pull changes from repositories.
   • Continuously monitor changes and trigger deployments based on predefined conditions.

3. Deployment Trigger:

   • Implement Python logic to automatically trigger deployments upon detecting changes in repositories.
   • Utilize webhooks/sockets or periodic checks to monitor repository activity.
   • Automatically trigger deployments based on new commits or branch updates.

4. Redeployment Logic:

   • Ensure seamless handling of updates to hosted websites using Python scripts.
   • Automate steps such as rebuilding sites and restarting servers as needed.
   • Implement caching mechanisms for speed during re-builds.
   • Implement mechanisms to track changes and ensure consistency in deployments.

Phase 2: Testing and Optimization

Implementation Steps:

5. Frontend:

   • Develop the frontend for Cloudmos, where users can submit the repository link they want to deploy or log in using OAuth with GitHub.

6. Testing with Multiple Frameworks:

   • Test deployment and redeployment processes with various frameworks (React, Hugo, Astro, Vue, Next.js, Svelte etc).
   • Ensure compatibility and reliability across different frameworks comparable to competitors like vercel does.
   • Optimize deployment scripts for each framework to maximize efficiency and performance.
   • Actively, add support for frameworks of all major programming languages like Rust, Go etc.

User Access:

Implementation on Cloudmos:

• This feature will be implemented on Cloudmos, providing users with a straightforward deployment process.
• A dedicated GUI within Cloudmos could prompt users to enter their repository URL from platforms like GitHub or GitLab.
• Users can select the best bid for deployment, and their site will update automatically upon pushing changes to their repository.
• Recognizing the intricate nature of this undertaking, it is evident that constant support and insights from the Cloudmos team will be invaluable in realizing our objectives. With their collaboration and receptiveness to reasonable updates, I am confident that together, we can navigate any challenges and forge a path towards success.

Team:

• @HoomanHQ & Team:

  • Demonstrated a robust track record of contributions in the Akash ecosystem.
  • Recent notable contribution includes the complete development of the Akash Network website, accessible at the official link.

• Advisor:

  • Alani Kuye (Former core-team member of 'Akash Network' / 'Overclock Labs')
  • Envy (A NetaDAO member, https://github.com/envy-cosmos)

More details will be added soon.

Thank you
HoomanHq

[HoomanHQ]

Here is the quick POC video (in a week we will share more solid prototype, this video is just to get around the concept): https://youtu.be/0k9V6mSNRZQ

[anilmurty]

Hey @HoomanHQ,

Thanks for another well written proposal (+clear demo!) and for coming to sig-clients to demo and talk about it as well. Overall I agree that this functionality built into console.akash.network would be very valuable and will help in onboarding new users, so appreciate you taking the initiative here.

Here are some questions and comments I can come up with right now (and may have follow ups + others as I think about this more):

1. My understanding is that this solution would consist of building and maintaining one container image per JS framework, so there's a need for ongoing maintenance of those containers to make sure they are updated to support new versions of them, their dependencies etc, correct? Is there a way to avoid having to build/ maintain one per framework (I can understand needing to maintain runtimes for different programming languages like Go, Rust etc in the future) but would be great if there's a way to avoid having to have a separate container per JS framework.

2. The base container image essentially clones the entire repo into the provider and builds it there? Wouldn't this bloat the resources to be much higher than what's necessary if the artifacts were built on an external build server (like github actions)? Not sure of what the best solution is but I think Vercel has fixed limits (for cpus, mem and disk space) based on the tier ("hobby" vs "pro") and lets "enterprise" customers request custom limits and charges people. Also for the hobby and pro tiers, the default is to have one single build server and queue requests as they come in. So if you trigger multiple builds, you still use the same resources, but the builds don't run concurrently (unless you pay for concurrent builds). Overall is the plan to essentially mirror the compute resources to what you see in Vercel and not support queuing?

3. How would the SDL that contains the base (per framework) container image know what resources to request? Is this something you'll request of the user? Will the user know what the size of the repo + built artifacts is? Or will you just fix this to some max limits and pass back to user if you run into OOM issues and stuff (I'm guessing this is the way but curious :) )?

4. The demo shows the simple case of a public repo and passes the repo url as an ENV variable (likely received through the webhook that you establish with GH/ Gitlab), which makes sense. You had mentioned (on the call) that you think you can also support private repositories. I'm curious how you will be passing Auth info to the base container (will that be through ENV variables as well)? As a side note, the core team has been working on supporting private container registries as well, which is somewhat related to this, though not entirely (Private Registry Support #321). cc @troian on this.

5. Assuming this moves forward, it'll be great to sit down and discuss the actual workflow design within cloudmos (with @baktun14 @aktdenis ) and code changes/ contributions to cloudmos/ console (with @baktun14 @Redm4x)

Thanks!
Anil

[HoomanHQ]

Thank you, @anilmurty, for reviewing our proposal and taking the time to provide feedback and ask questions. Below are our responses to your concerns and questions:

1. We have indeed considered an approach to minimize the necessity for routine maintenance by utilizing a singular container for all JavaScript frameworks (Yes, there will be only one base container for all JavaScript frameworks.). Our strategy involves the development of robust Python scripts capable of identifying the specific framework in use and deploying it efficiently. The script will manage typical framework-related issues and analyze the repository's package.json to ensure the accuracy of build commands, thereby reducing the likelihood of failures.

To accommodate customizations, updates, and enhance flexibility, users will have the option to specify essential parameters such as the required Node.js version, install/build command, and the build directory. This approach streamlines the deployment process and offers a level of customization to meet individual needs.

Furthermore, we are committed to continuously improving this project through open-sourcing the base scripts. This initiative will allow us to collaborate closely with the community, ensuring that any necessary adjustments to the scripts are made promptly. Our goal is to cultivate a supportive community around this product, empowering users to contribute and assist each other, thereby enhancing the overall effectiveness and reliability of the solution.

---

2. We are aware of the potential resource inefficiencies associated with cloning and building entire repositories on the provider's infrastructure. To address this, we have two strategies in mind to optimize resource use and align our service with efficient models like Vercel, catering to various user needs.

   • Using the provider's VM: This method will result in a slight increase in computational resource usage. However, the impact is expected to be minimal, as the process of building JavaScript frameworks is not excessively resource-intensive. Moreover, users will always have the flexibility to enhance their resource allocation, such as CPU and storage, according to their project's specific requirements.

   • GitHub Actions Integration: As an alternative, we can use GitHub Actions for the build process. By utilizing users' GitHub tokens or login, we can orchestrate builds externally and push the artifacts to a common DockerHub repository. This method minimizes the resource footprint on the user's VM, making use of GitHub's infrastructure for builds.

---

3. In our streamlined deployment, we will have multiple predefined computational resources with options to customize, eliminating the need for users to specify these details. This ensures efficient and streamlined builds without manual intervention.

However, for projects involving server-side technologies that might need more power such as Next.js, Nest.js, Go, etc., users will need to specify the necessary computational resources. This requirement allows us to tailor the deployment environment to the specific needs of more complex applications, ensuring optimal performance. Through this strategy, we aim to offer a simplified deployment experience while providing the necessary customization options.

---

4. Primarily, we aim to develop a 'Login with GitHub' through OAuth option on Cloudmos, which will let users select private repos by giving required access through an access token. However, if this is not a viable option, we will ask users to input their GitHub token as an ENV variable. In this case, we will use HTTPS to clone the private repo via the token.

Yes, we are open to hosting a meet with the Cloudmos team to discuss the plan and proper feasibility of the solution. Please let us know a suitable time for the meeting.

Thank you
HoomanHq

[anilmurty]

We have a time set for 4 (tomorrow) where we can chat more but why not use https://buildpacks.io/ for building images?

[anilmurty]

@jigar-arc10 - are you guys still pursuing this? have you built anything? https://twitter.com/praetor_app/status/1560397178199236611 - seems very similar to the proposal here.

[jigar-arc10]

We are not pursuing the deployment project at this time. We have created a POC and demoed it in different meetings, and we have posted a video of the demo at https://twitter.com/praetor_app/status/1560397178199236611. If you want our feedback on what kind of tech to use and help with architecture, we are open to meeting with the Cloudmos team and @HoomanHQ team.

[anilmurty]

Good to know - would be great if you can recommend suggestions for how were thinking about the build problem (not bloating the resource requirements in order to clone and build the whole repo in the deployed container) and how you were thinking of dealing with the open env variable problem (if you were planning to support private registries)

[anilmurty]

@HoomanHQ - to follow up from the conversation we just had - we agreed that a good first step is to scope this down to just supporting public GH repos and validate by building an SDL so that

1. We're not conflicting with the ongoing code changes that @baktun14 and MaxC
2. Making sure that the solution works before we introduce it into the main user deployment path in Console
3. Allowing time to discuss our Console 2.0 plans that include changes to the main deployment workflow

[HoomanHQ]

Current Progress:

• We have built a template SDL file which has environment variables for repo link, node version(optional), build command(optional), build directory(optional) and port.
• It is capable of deploying React, vue, Astro and others..
• The sdl has an automatic CI/CD pipeline setup which pulls changes to the deployment as soon as changes are made to the repo.
• Presented the live demo in yesterday's meet.

Things to be done:

• extended support to almost all major js frameworks.
• more robust error handling.
• reducing the hardware overhead for building the projects even further.

Next major step:

• Propose a formal & final proposal.
• push proposal onchain and request a funding from community pool.

Thank you
HoomanHq