Design for partial automation of release process

[irbekrm]

At the moment our release process is largely manual and requires individuals to look at periodic test results in TestGrid to detemine CI health and then perform steps to stage an publish a release.

This does not work that well in practice - lots of toil and risk of human error when looking at TestGrid.

It would be good if it was possible to automate the gating of release on CI health as well as parts of release process

See also https://kubernetes.slack.com/archives/CDEQJ0Q8M/p1655221978315829

[munnerz]

For reference to the Kubernetes upstream project & how they automate checking tests/ensuring quality during releases, they call out to/check testgrid from within the release tooling that runs on release manager machines/GCB.

This allows them to 1) see the health of a release ahead of time (i.e. visualise the past N test runs on testgrid in case they need to begin fixing things ASAP) and 2) have confidence that the release tooling won't allow them to push an insufficiently tested release: https://github.com/kubernetes/release/tree/master/pkg/testgrid