[$500 bounty] Speedup CI setup to <20s

[adeebshihadeh]

The best case time of the setup-with-retry stage that runs in most of our CI jobs is ~1m4s. All it does is setup the openpilot environment, and most of that time is pulling an already built docker image. This puts a hard limit on how fast our jobs can finish; a job that finishes in 1m is 10x better than one that finishes in 2-3m.

Some possible strategies:

• make the docker image smaller
• move the docker cache somewhere faster
• ditch docker, install directly on the 20.04 runner, and cache the apt and pip packages

Requirements for the bounty:

• all setup-with-retry on the final PR commit must finish in less than <20s

Sub-bounty of $100 for <40s if you can't get to <20s. $500 is for <20s. Bounties don't stack.

https://github.com/commaai/openpilot/blob/master/.github/workflows/setup-with-retry/action.yaml

[nelsonjchen]

wild idea, compatibility unknown, gain (or loss) unknown:

use containerd or similar drop-in runtime instead of stock GHA moby with lazy loading of compatible estargz docker image "pages".

https://github.com/crazy-max/ghaction-setup-containerd

[jimbrend]

@lukechilds what do you think?
just out of curiosity on first glance, either way

[jimbrend]

• ditch docker, install directly on the 20.04 runner, and cache the apt and pip packages

what are the cons here?

[mbiernat42]

That sleep 30 seems questionably long

[profknow]

If it's just loading previously configured env, then why not just operate on Flash Drive, leaving the state where it was when last active.

Even on my own computer, I often keep copies of working system/environments that I simply dump into active memory without "booting up". It saves SO MUCH time when you already know the final state anyway.

--Loren Grayson