Make release process simpler

[mossheim]

Motivation

the release process is very complicated and involves lots of manual actions. automating most or all of it would help a lot.

Description of Proposed Feature

a set of scripts (in python) to automate every step of the release process, and use cues for the manual steps.

Plan for Implementation

at the dev call yesterday @joshpar , @jrsurge , @dyfer and i all expressed interest in working on this together. others are of course welcome to help too!

the current plan is to have a python script. i think @claremacrae 's suggestion here is really great (quoting from a Slack conversation):

Did I describe the steps we do in Approval Tests, for bits that require manual intervention - the script does:

1. Write out a prompt to ask user to do “Some manual step X…”
2. If relevant, open in a browser any URLs that might save the developer time
3. Prompt the dev to say Y (if step is done) or any other character if not - which halts the release…

It meant we could start with a script that had no automation at all - just a load of “do step x manually, then y, then z”
And over time we could automate more and more of the intermediate steps…
The code ended up in Python with tests…

https://github.com/approvals/ApprovalTests.cpp/tree/master/build

[dyfer]

I have some comments for the git/github part of the process. I wonder how should we handle merging in terms of approval(s).

AFAIU the current process is:

• (make sure all planned cherry-picks are done)
• merge release branch (e.g. 3.xx) into master
  • does this ideally need to be approved by a second person, or could the release manager do it themselves?
• do actual release off of master
  • make a tag
  • wait for artifacts (?)
  • upload artifacts to GH (or does that happen automatically?)
  • perform additional steps
    • sign macOS release
    • ...?
  • update website
  • publicize release through various channels (mailing lists, facebook etc)
• merge master into develop (AFAIU could happen anytime after 3.xx -> master merge)

Usually there are at least 2 ppl involved in every merge. Would we want to (could we even?) perform the 3.xx -> master merge automatically/without second person's approval?

We could still have master -> develop merge as a PR approved by a second person, since no other step of the release depends on that IIUC. In that case the automated step would be to create a PR for that merge, not perform actual merge.

[mossheim]

if we have more automation, i'd prefer having fewer merges. in fact i think we can get away with just one to check the changelog, and the other git-flow merges can be done without going through pull requests, or at least without going through approval & CI for pull requests.

i've added a link to our current process documentation to the top of my original post.

upload artifacts to GH (or does that happen automatically?)

all artifacts except the source tarball are uploaded automatically.

[dyfer]

Thanks for pointing to the current writeup! I obviously missed some steps. If you think that release branch into master can be automatic that's fine with me (though I'm not a git guru).

What are the proposed next steps? one person writes a first version of the python script or we do it collectively somehow?

[claremacrae]

I would suggest pairing or mobbing to get going on the first version.

I would love to be involved in this, if anyone would like a hand - UK time zone.

There are some things around making the release process itself automatically testable - which we found were quite important - in order to make the Python code refactorable, for example.

[mossheim]

@claremacrae that sounds good, and thanks for offering to help! i am probably free this weekend to pair or mob, if that works for you. happy if other people want to join.

[claremacrae]

@brianlheim thanks - this weekend’s fine for me too!

Shall we talk via Slack to sort it out - If you could @ me there, I’ll see it on my phone.

[claremacrae]

I’m also happy for others to join too, of course.

[claremacrae]

@brianlheim and I talked yesterday about the tagging process. As I understand it, the SuperCollider release process has the release manager add the tag manually and then upload the sources.

if that’s correct, then I just wanted to note that github may take care of some of that...

With ApprovalTests.cpp, we open a new release page, with the tag included in the URL, and then we drag in one artefact (the single header).

When we save the release page, github automatically adds the source tarball and zip.

For an example, see https://github.com/approvals/ApprovalTests.cpp/releases/tag/v.10.2.1 - the .hpp was the thing we added manually.

[dvzrv]

@brianlheim and I talked yesterday about the tagging process. As I understand it, the SuperCollider release process has the release manager add the tag manually and then upload the sources.

if that’s correct, then I just wanted to note that github may take care of some of that...

FTR: The tagging should always be done manually and ideally even be signed by the release manager so that downstreams (e.g. Linux distributions) can verify the signature and check the authenticity of the release.

I'm not entirely sure, but assume you are referring to the source tarball (in regards to "upload the sources") in this case.
It might be worth exploring whether it can be a solution to integrate a tool such as gh with the script generating the source tarball (to upload the source tarball as an asset via an API call after creation, if possible), or alternatively running the script generating the tarball in a github action, directly creating assets upon tagging and automatically starting a release pipeline.

That being said: As travis is in use for CI, I'm not entirely sure how happy the maintainers would be about switching to github actions (if that even makes sense).

[mossheim]

It might be worth exploring whether it can be a solution to integrate a tool such as gh with the script generating the source tarball (to upload the source tarball as an asset via an API call after creation, if possible)

this is possible with github.com/github/hub. hub release edit <tag> -a <asset>. it looks like gh has more active support, but for now i would probably use hub to automate github tasks since it has a more fully featured command set, and i think we only need one github-interacting command line tool.

alternatively running the script generating the tarball in a github action, directly creating assets upon tagging and automatically starting a release pipeline.

i'm OK with this script being run locally since it's relatively fast, reliable, and easy to automate; fiddling with CI is relatively time-intensive by comparison.

As travis is in use for CI, I'm not entirely sure how happy the maintainers would be about switching to github actions (if that even makes sense).

it's a no from me :) i would like to avoid investing further in microsoft-specific technology if it can be avoided as there's a big risk of vendor lock-in, and i know some community members are already unhappy that we've remained on GitHub after the acquisition.

The tagging should always be done manually and ideally even be signed by the release manager so that downstreams (e.g. Linux distributions) can verify the signature and check the authenticity of the release.

wasn't aware of this, but it makes sense. we can definitely do that. would you still consider the tagging to be manual if it's created by a script, run manually by the release manager, which performs validation checks and asks for confirmation?

[dvzrv]

wasn't aware of this, but it makes sense. we can definitely do that. would you still consider the tagging to be manual if it's created by a script, run manually by the release manager, which performs validation checks and asks for confirmation?

Yes, as long as it is done on the release manager's machine and not on a random remote CI machine (some projects actually do that by just auto-tagging every push to master, which IMHO is a really really bad security and release practice).
The validity and usefulness of a release manager's PGP key is only as good as the environment that it is used in (and again some upstreams actually PGP sign their releases in CI, using a keypair that is available to the public, basically rendering the entire practice useless). To that end, github's PGP key is useless, too.

That all being said: Please be aware, that as most downstreams follow TOFU in regards to the usage of trusted upstream PGP keys, it is essential, that a chain of trust is established from then on. This means:

• if the release manager changes his/her key pair, the new key should be signed by the old key and the old key by the new key and this change should be announced before doing a new release
• if the release manager role changes from one person to another, this should be announced before doing a new release and the previous release manager should sign the key of the new release manager
• if the key pair of the release manager has been compromised it should be announced and the key pair be revoked. if a chain of trust can not be established between new and old key, this needs to be communicated
• it is totally fine to have more than one release manager key-pair (as in: more than one person) as long as the first person releasing with a signature ensures the chain of trust for the other(s) and this is communicated
• announcements to changes of the release manager and/or the trusted keys in use can be done e.g. in the README of a project and should be done by the current release manager with a signed commit before doing a new release that uses a new key pair

As a sidenote: Whatever you do, don't use Thunderbird's enigmail without making sure, that your key pairs are alright (double check your signatures!). It has a track record of messing up your keys in use by auto-generating new one's due to the p=p stuff in there. I'm not entirely sure whether this is finally removed/fixed, but has proven an issue for many users and projects alike.

Another sidenote: If you are interested in securing your key using a hardware token, look into nitrokey or alternatively yubikey.

[dyfer]

On the dev meeting on May 2, 2021 we decided to push it to 3.13 milestone and divide this into smaller actionable items

[claremacrae]

I’m still really happy to help with this, when you get to it...

[dyfer]

I'm pinning the issue for now, since we haven't fully addressed it for the 3.12 release