Automate the creation of an offline runner

[npalm]

The current approach requires we alwsys have an offline runner regirstered. Those offline runners are removed by github eacht 60 days. Therefore it would be convient to automate the process to keep 1 offline runner so we can scale back to 0.

Potential solution direction

Register via ED2 instance

Use the same mechanism we use to spin up runners with an extra lambda that only execute the user data till the config step. And also ensure the ec2 instance is take down

Reverse egineer github config process

Create lamda that is using github http calls based on reverse engineering, see actions/runner#558

Run the config in a lambda

Create a lambda that can exectue the config via a lambda layer

[miked63017]

@npalm I've figured out most of the logic, urls, etc involved with registering a runner. I plan on making a python module to handle this using python 3x and requests. This will probably be this weekend when I'm off work.

If you would prefer to implement it yourself, I can share my notes I've gathered about the process before then.

I honestly think I want to spend some more time tracing the entire process, so it could actually pull and perform workflows as well, but the new runner registration was a must have for me.

Basically the process I have found to work:
Get runner token via api using pat or app creds
Post token to secretish endpoint, receive back json data with new secretish endpoint and a jwt
Use jwt to auth via auth bearer header, using newly revealed endpoint, can query existing "agents"(as they are called in the api), add a new one, or update an existing one.

There's obviously some more details involved, including creating an RSA key, and a bunch of headers. I haven't looked further than registration yet...

[npalm]

@gertjanmaas I think you will like the comment above

[mcaulifn]

@miked63017 let me know if you don't get to it. We are looking for this as well so I would have some time to work on it.

Edit: Not sure if you saw, but this is newly released for Python: https://github.blog/2020-12-18-learn-about-ghapi-a-new-third-party-python-client-for-the-github-api/

[miked63017]

@npalm @mcaulifn here is a link, it's still pretty beta'ish and not well documented, but I guess we can say the same thing about the runners/actions API in general :-)

https://github.com/miked63017/pyghrunner

[miked63017]

@mcaulifn in RE to the ghapi module it looks cool, but most of these calls are undocumented pieces of the api, and probably subject to change.

[mcaulifn]

Overall looks like it should work. Are you planning on adding it to this repo?

[npalm]

@gertjanmaas any opinion?

[miked63017]

@npalm @mcaulifn not sure if I have the context to add it here to this repo, I am personally working in a GKE operator to do similar but figured it could help others to share some simple code to integrate with other projects. Seems to be a fairly common request for this functionality. If you'd like me to take a crack at adding here via a PR I can maybe spend some time this weekend.

[gertjanmaas]

I quickly skimmed through the python code and it seems to confirm what I saw when I tried to reverse engineer it a while ago. Would be great if this could be implemented here. Getting tired of adding offline runners by hand :P

[miked63017]

@gertjanmaas where I am running (the equivalent) of this code(in a private library), we just run the few methods periodically, or in response to an event, and overwrite the previous "virtual runner". We are basically just using it as a placeholder so jobs queue rather than fail because no runners with labels exist. Then we look at the jobs details, and spin up the appropriate runner, with appropriate labels as needed, and with the --once flag.

I still have plans of further investigation into creating a full custom runner, most likely written in python, that can then be embedded in other places. This just hasn't been high priority for me yet.

[dimisjim]

How about runner deregistration?

The offline runner basically needs to be recreated every 30 days, in order to never have 0 runners in the org.

This should be automated as well.

[boldandbusted]

FYI, I'm running into this issue, still. At first I thought it might be because the team who has the runners isn't using them often, but thinking through the workflow a bit more, I realized GitHub is deleting the offline runner purely because the offline Runner Agent isn't 'phoning home'. :( I posted this in the GitHub Actions Discussions: https://github.community/t/github-actions-offline-runner-removed/191044/5?u=boldandbusted - so I hope that perhaps GitHub Engineering may offer some alternative approach.

If there is an automation that doesn't involve a PAT I'd love to hear it! :) We can limp along by manually bringing up the offline runner (or re-adding it every 29 days, which is kind of easier), but this isn't really a scalable path, sadly. I don't think this is the fault of this repo's code, BTW, to be clear, but an architecture deficit in the GitHub Runners approach.

[npalm]

Any help on automating this is still very welcome

[miked63017]

If it helps, I have additional code that parses and decrypts all the communications, only thing missing is the ability to send logs back and forking off subprocceses. I haven't had time to do anything with it, but essentially I wanted to rewrite the runner itself in pure python which could make it a little more flexible.

[axel3rd]

I confirm Offline runner(s) is useless too since GHES v3.3(.1) series.