[WHAT IF] We had automated testing of deployment!

[hulto]

Automated deployment testing will improve the speed of development while aligning to our north star goal of improving educational outcomes.

[KCarretto]

We could write unit tests for the tomes used for deployment 👀

[hulto]

That'd be poggers.
We could even define supported OSes for each tome and verify with CI.

[Milkshak3s]

Just thinking out loud here re: CI

CircleCI has "orbs" which allow teams to define a CI yaml that matches best practices, environment, etc.

Idea

What if we provided "orbs" for deployment testing that matched what we have available in our testing infra?

• We could set up VMs in infra that match competition machines, then define them as orbs for tomes to reference for testing
• CI pipeline handles running tests on orb VMs, tome can reference list of test machines in metadata yaml
• Added benefit of tying tomes to specific machines

Cons

• We'd need to figure out an ops solution to managing the VMs & roll our own test runners
• Would be tied to our specific infra, increases setup complexity (unless very generic runner solution)
• Possibly expensive unless running on private equipment

[hulto]

That'd be pretty neat-o SE and MA mostly just use base AMIs from AWS can we take advantage of that? Maybe run once a month to keep costs lower.

[Milkshak3s]

Maybe run once a month to keep costs lower.

True! We could make that a GH actions button and just understand "1 run = $2". We'd lose some of the CI aspects but it would be better than being completely cost-limited.