proposal: Use TUF-on-CI to maintain root-signing

[jku]

Hi root-signing contributors and interested bystanders,

@kommendorkapten and I have been working on TUF-on-CI for the past few months -- it's a more "productized" TUF signing system that works on top of git and GitHub like root-signing. I'd like to propose gradually moving the maintenance of Sigstore root-signing to happen with TUF-on-CI.

Summary

• I believe moving root-signing to TUF-on-CI is good not just for Sigstore but also for other projects looking for similar artifact delivery solutions (see Thoughts around if this is generalizable for other root/key signing ceremonies? #762)
• I'd like to start by creating a separate (but almost identically operating) TUF repository (that I'll refer to as root-signing-staging here but I'll take better names). The purpose is to have a low-risk test environment not just for this change but any future root-signing design changes. This new repository would be maintained with TUF-on-CI and would replace "staging" directory in this git repository.
• root-signing-staging would be where missing features and bugs are ironed out. Based on experience from root-signing-staging we can then decide whether to maintain production root-signing with TUF-on-CI as well.

More details

• My initial root-signing plan with details and open questions
• TUF-on-CI documentation
• An example of what the finished root-signing-staging would look like

Next steps

• arrange a meeting where folks who are interested can ask questions and state opinions on the open design questions
• find a couple of volunteers to help out with the final root-signing-staging repository design details and to be root-signing-staging signers willing to test drive things

I'll start a thread on slack (sigstore-keyholders maybe?) about this: comments are welcome here or in slack

[joshuagl]

This is great! I fully support the proposal to use TUF-on-CI to maintain this repository.

For the keyholder audience, it would likely be useful to be able to compare the current ceremony to how the ceremony should work after the change. That would be a good demo and an even better diagram/document.

[jku]

I forgot to update this issue:

• I've proposed 6pm EEST (11am EDT) on Wed Sept 6 as meeting time about this
  • feel free to suggest another time, especially if you're interested in being a guinea pig (a signer for the staging repository)
  • Just a couple of interested keyholders should be enough to give us the feedback needed, but everyone interested is welcome
• I've written some more about how the different processes work: https://docs.google.com/document/d/1Omnym92RcEBijauDiwobsMuO2alUavKAWA62rGuMSPw/edit?usp=sharing

[jku]

I'll need to find the right place to file an issue for this but:

• I'd like to get a new project under the sigstore org: root-signing-staging is the best name I have so far (but I'll take other options): the purpose as documented above is a separate TUF repository that operates like root-signing but allows testing new processes and tools without affecting production service level
• Project maintainers could include @kommendorkapten and me (@jku) to start with
• the project needs access to one online (GCP KMS) key: preferably a new key only used for this purpose, but a generic testing key works too. Preferably I'd also like roles/cloudkms.publicKeyViewer for myself to make the setup easier but if that is difficult I'll think of another solution
• Plan is to start publishing the repository in GitHub Pages (so sigstore.github.io/root-signing-staging/) -- maybe change this to match production root-signing later. This means we'll need to be able to modify settings->pages and settings->environments (or I can document what needs changing here)

[jku]

for reference: sigstore/root-signing-staging#1 is ongoing but the TUF repository is not functional yet (KMS is not working). Next steps for sigstore/root-signing-staging are:

• get online signing working in the new project
• test the result with various clients, ensure smooth upgrade from current root-signing-staging TUF metadata
• setup publishing from the new project to the root-signing-staging GCS bucket
• remove staging metadata (and publishing) from this repository

[jku]

sigstore/root-signing-staging is now technically operational (provisional, not published to the official GCS bucket yet).
Next steps:

• client smoke tests are ongoing
• some small improvements to tuf-on-ci would make sense before adding more signers
• Arrange a show-and-tell for interested signers, add a few more signers
• Implement publishing to GCS

[jku]

Update:

• sigstore/root-signing-staging is working OK with four signers
• sigstore-probers has a PR to make it compatible with future changes
• root-signing-staging tests are now IMO more comprehensive than what we have here

My next step is a dry-run of importing the production repo into tuf-on-ci:

• there's an open issue with npmjs.org delegation handling but other than that I'm not aware of any blockers.
• the deprecated roles in snapshot.json should not be an issue but I'd like to be sure

[jku]

Update:

• there is a test repo https://github.com/jku/root-signing-test that is based on production root-signing (contains same roles, delegations and artifacts) and seems to work: it is compatible with cosign and the tuf-on-ci tests. The repo does have several hacks obviously since I had to make myself the signer but should be fairly representative of production root-signing
• As a result of the tests, two minor issue were found: investigate build-repository failure theupdateframework/tuf-on-ci#342, actions/test-repository fails if both "compare_source" and initial root are set theupdateframework/tuf-on-ci#345
• A more significant issue was fixed earlier where keyids were not spec compliant: keyids for new keys are not spec compliant theupdateframework/tuf-on-ci#292
• npmjs.org delegation handling is still an open question but fredrik is working on it

Next:

• write a migration plan, get it reviewed