Create cert-manager specific testing infrastructure

[jakexks]

To prepare for the cert-manager release automation drive and the separation of cert-manager prow from jetstack prow, we should have automated cert-manager test / release infrastructure in place.

Progress

• Investigate what we currently have
• Try using official GKE terraform modules - dislike the opinionated setup and resulting cost
• Infrastructure as code - k8s cluster
  • Investigate best practice for GKE
  • Deploy / destroy test cluster with tf
  • Investigate OIDC on GKE - kube-oidc-proxy + gangway + dex + github oauth
  • integrate OIDC into build.
• deployments as code - identity provider for k8s api access (kubectl) deployed
• integrate identity provider with Github - org membership grants read access

Next steps

• Create the repository cert-manager-test-infra
• Push all config for infra in to the cert-manager-test-infra repo
• Port prow jobs to new testing repo under cert-manager org
• Create github bot account
• Run the new infra
• Decommission old prow setup

[munnerz]

Awesome to see 😄 when you say being an org member grants read access to the cluster - is that for all org members? Could we use a GitHub team within the org instead, so we can have more granular access controls?

I'm nervous about exposing access to the cluster running prow (this contains credentials like the bot user, which is an org admin) to anyone in the org (and I think this will implicitly 'raise the bar' of being an org member too, which may make us less likely to have people as part of it)

[jakexks]

A Github team makes much more sense 👍

[SgtCoDFish]

I've renamed this since I think it addresses the more challenging technical issue of moving the testing across. Release infra should be much simpler; I've created #50 to track that.

[maelvls]

Seems like we are getting close!