[EngAut] CommandBot: Create GHA version

[crystalin]

Background:

• current command bot is fully dependent on GitLab, because it was designed to manage tasks queue via GitLab CI pipelines.
• if any parachain team would want to re-use this bot for their benchmarks and other tasks - the GHA+self-hosted runner is much simpler to set up than deploying a runtime NodeJS app with DB + self-hosted runners

Stakeholders and Goals:

• [Parity Internal]: unblock gitlab deprecation, as current implementation is locked-in for GitLab
• [Ecosystem] setup and run benchmarks easier

Original request:
We've been using the benchbot and are thinking to move to the command-bot, however we don't rely on gitlab for our CI.
Our current setup is using Github Action but with self-hosted instances (dedicated to building/testing).

I like the idea of the command-bot allowing commands from different sources (matrix/github/...) but it seems to be quite customized to how parity work (using gitlab, helm,...).

Do you have any plan on making it more generic to allow non-gitlab service ?

---

GitHub action milestone

Tasks

Beta Give feedback

1. Remove GitLab code cmd-action#4
   action +1
   
2. Command schema cmd-action#7
   action +1
   
3. Status check - schema cmd-action#10
   action +1
   
4. Add json schema for yaml files cmd-action#14
   action +1
   
5. Generate commands documentation cmd-action#12
   action +1
   
6. Make app get information from a comment cmd-action#2
   action +1
   
7. Handle arguments in commands cmd-action#8
   action +1
   
8. Add allowlist cmd-action#19
   +0
   
9. Make app work as a one time executable cmd-action#1
   action +1
   
10. Actionize the app cmd-action#3
    action +1
    
11. Set auto-publish cmd-action#5
    action +1
    
12. Rewrite bench to be generic for any kind of substrate-based projects #274
    +0
13. Report error on failure cmd-action#20
    action +1
    
14. Use file name instead of name field cmd-action#22
    action +1
    
15. Produce link to comment cmd-action#23
    action +1
    

[mutantcornholio]

Could you elaborate on what currently doesn't work, or how command-bot's way of spawning jobs contradicts your CI configuration?

[crystalin]

I didn't go as far a trying to setup everything yet. The documentation contains many element but it is a bit hard to know how all of the elements are orchestrated overall.
But in the setup part it mentions:

Create a repository in GitLab and set up GITLAB_PUSH_NAMESPACE along with GITLAB_ACCESS_TOKEN to run the commands for the GitHub repository (Step 3).

That is why I supposed it required to do so, I'll try to set the bot up and see where I get stuck

[crystalin]

Also, the first line of readme also mention:

command-bot executes arbitrary commands on GitLab CI

[mutantcornholio]

You're going through deployment/development setup. Are you trying to host your own instance?

[crystalin]

Ideally yes, but if you offer a service where we can plug our repo and our CI servers, I'm happy to try it (I don't think that would work in term of security however so we would end up hosting it)

[mutantcornholio]

Okay, I think I've misread the situation. Let us think on it.
Currently, it'll basically mean replacing at least half of the code, and that raises a big question on the vision of the project.

[mordamax]

@crystalin Currently, we defer the tasks to run and queue for the CI, in our case Gitlab. We are going to make it less opinionated and instead of creating a Gitlab CI job - create Kubernetes jobs.

What would be a desired infrastructure set up so you could deploy and use it on your side?

[crystalin]

Thank you, I think plugin it with github and using their GHA (self-hosted runner) is preferred but we could probably combine it with our kubernetes cluster too

[mordamax]

Current plan is to move away from gitlab completely, so will be porting the exiting command bot to separate repo, the idea is to set this up as Github action, with decoupled repo for command scripts and interface.
https://github.com/paritytech/command-bot-scripts/tree/main/commands
This allows to avoid re-deploying/releasing the bot business logic from the commands implementation.

[Bullrich]

I will be taking up this task.

I have created a milestone to track this:

GitHub action milestone

[Bullrich]

The repository cmd-action has been created. The action will be developed there.

I ran two experiments to see if this is feasible, both of them were succesfull:

• added prototype ci script cmd-action#6
• added cmd-action prototype paritytech-stg/polkadot-sdk#26

[Bullrich]

Currently working on paritytech/cmd-action#1.

I'm making the system parse content from the comments and validating it with the commands.

[Bullrich]

Preparing schema handling.

I've been in contact with Oliver, our main stakeholder.

• We have decided that the repository will hold the required scripts (unless that a project evolves too much)
• Also, a similar action was commented by Basti: Exclude pallet_example_* from bench-all-substrate command-bot-scripts#69 (comment)

The logic of configuration handling will be similar to review-bot's: Easy to read and hosted in the target repository.

cmd-action will be mainly a tool to "parse" commands and pass it to a github action with a repo script.

[Bullrich]

Currently working on paritytech/cmd-action#7

Configuration files will be moved to be in yaml (can also be toml) so it's easier to read.

I'm preparing this ticket in combination with paritytech/cmd-action#10 so we have a schema validator out of the box

[Bullrich]

Completed the following tasks

• Command schema cmd-action#7
• Status check - schema cmd-action#10
  • Added schema validation cmd-action#11
• Generate commands documentation cmd-action#12
  • added generation of command documentation cmd-action#15
• Add json schema for yaml files cmd-action#14
  • Added json schema validation cmd-action#17

[Bullrich]

22-26 of April

• Generate commands documentation cmd-action#12
  • added generation of command documentation cmd-action#15
• Currently working on Make app get information from a comment cmd-action#2
  • Extract parameters from a command

[Bullrich]

Currently working on the finishing touches for paritytech/cmd-action#18

This will close:

• Make app work as a one time executable cmd-action#1
• Make app get information from a comment cmd-action#2

[Bullrich]

Closed:

• Report error on failure cmd-action#20
  • Report run and failure with comment cmd-action#21