Infra: Create a GitHub Action to simplify release process

[tomschr]

Situation

The current release process is described in the file release-procedure.md. It's a bit of cumbersome as a lot of things needs to be done manually.

Some ideas were already described in #186, this is specific for releases.

Proposed Solution

A GitHub Action seems a good method for this task. Some ideas what our Action could do:

• start the release process when a) a tag is created with adheres to the semantic version, or, b) a branch release/VERSION is merged into master.
  (not sure which one of this will be better or more feasible)
• Verify if a branch release/VERSION matches with the __version__ variable in semver.py?
• Verify if the CHANGELOG.rst contains the correct version string?
• Maybe use the branch release/VERSION to publish it to the TestPyPI first.
• When the branch release/VERSION is merged into master (and only then!), do:
  • deploy it to PyPI
  • create a tag (and use the content of the CHANGELOG.rst?)
  • update the "Releases" tab on our GitHub repo page
  • ...

It's just a collection of rough ideas. I need to try it what's possible and what is more difficult.

@scls19fr If you have more ideas or disagree with the above, just let me know so I can tailor this Action for our specific use case. :-)

Hint:
I will try it out in my fork first so this repo will be kept clean.

See also

Possible GH Actions to investigate further:

• create-release action
• pypi-publish
• deploy-pypi
• Publishing package distribution releases using GitHub Actions CI/CD workflows
• GitHub Actions: Manual triggers with workflow_dispatch (added on 2020-11-15, found by @tlaferriere, thanks!)

[scls19fr]

Be aware @tomschr that our release procedure should be fixed see #173

[tomschr]

Thanks for the reminder @scls19fr. I will take that into account. 👍
Any further wishes? 😉

[scls19fr]

Great idea to upload first to TestPyPI!

[tomschr]

Great idea to upload first to TestPyPI!

Better safe than sorry. 😺

[ppkt]

@tomschr is it possible to trigger Action manually? It would be really nice to have this in one place.

BTW. I have a related idea - maybe we could add also publishing RC releases? For example when someone needs bugfix asap (and does not want to install package directly from git) he could use this version until stable will be released?

[tomschr]

@ppkt Sorry for the delay.

is it possible to trigger Action manually?

Theoretically yes, but I'm not sure. What's your understanding of "manually": Pushing a button? Running a command?

have a related idea - maybe we could add also publishing RC releases?

Yes, that's what I have planned. 😉 My basic idea is to trigger the release process when you open/close a release/<RELEASE> branch. The <RELEASE> placeholder can be anything, even a RC release, for example release/2.9.1-rc1.

Of course, I need to figure out the details, but that's the plan. I don't know if this works in the end, but we will see.

For example when someone needs bugfix asap (and does not want to install package directly from git) he could use this version until stable will be released?

Who is "he": a contributor, a developer, or this team?

If the GitHub Action has been implemented and is stable, the whole process (creating a new release, publishing it to TestPyPI and PyPI, pushing it to ReadTheDocs etc.) can be done automatically. In principle, everybody can start a release (by opening a release branch, see above), but only when you merge it to master, it will be really done.

Does that make sense? 😄

[ppkt]

@tomschr no problem, I'm quite busy too :)

What's your understanding of "manually": Pushing a button? Running a command?

Something like this - maybe editing certain file? I've never used Actions before so I don't even know what's possible :)

My basic idea is to trigger the release process when you open/close a release/<RELEASE> branch.

Wonderful :)

Who is "he": a contributor, a developer, or this team?

I'm talking about situation like in ticket: #114 (comment) Clearly there is someone affected by bug in our code and it'll break their workflow until stable version is released. By releasing "unstable" version (with rc part) it should be possible to use that version until stable version of our library is released.

Overall - I'm very happy with your idea, CI/CD loop is perfect in this case.

[tomschr]

@ppkt

no problem, I'm quite busy too :)

Thanks, I was busy the last days with other unrelated things, but I hope I can work on this soon.

What's your understanding of "manually": Pushing a button? Running a command?

Something like this - maybe editing certain file? I've never used Actions before so I don't even know what's possible :)

Ok, I see. Basically, the GitHub Actions are centered around events: When you receive a push event, do this; when you receive a event for labeling, do that.

I've never done this, so I'm probably not qualified enough to answer this completely. However, from what I've read, it should be possible. It's just a question of the right Action.

My basic idea is to trigger the release process when you open/close a release/<RELEASE> branch.

Wonderful :)

I hope it will work. :-)

Who is "he": a contributor, a developer, or this team?

I'm talking about situation like in ticket: #114 (comment) Clearly there is someone affected by bug in our code and it'll break their workflow until stable version is released. By releasing "unstable" version (with rc part) it should be possible to use that version until stable version of our library is released.

Maybe if we whould release some "release candidates" more often, would that help?

With my above idea, that would be possible.

Overall - I'm very happy with your idea, CI/CD loop is perfect in this case.

Thanks! 👍

[tomschr]

@python-semver/reviewers Ok, I can report some progress. 😉 (also trying out the new reviewers team). You can find a first example in my fork, see the GH Action 'prepare-release'.

I've created a GH Action now which contains two jobs: verify-release and prepare-release. The first job checks if the release branch is equal to the semver version. If successful (and only then!), the second job (prepare-release) is called. This job will build the wheel & sdist and upload it to TestPyPI.
This seems to work now. Maybe I need to change some details here or there, but it looks promising.

The next step would be to create another Action which would be triggered when the pull request is closed. This is work in progress and not finished yet.

@scls19fr To prepare the first Action, could you do me a favor and create a new token on/for TestPyPI? Do we have an account on TestPyPI already?

At the moment, I'm using my own token for my fork, but this needs to be adapted of course. 😉

I would suggest the following procedure (taken from the Packaging Guide, but slightly modified):

1. Go to https://test.pypi.org/manage/account/#api-tokens and create a new API token.
   If you have the project on TestPyPI already, limit the token scope to just our project. You can call it something like GitHub Actions CI/CD — project-org/project-repo in order for it to be easily distinguishable in the token list. Don’t close the page just yet — you won’t see that token again.
2. In a separate browser tab or window, open the Secrets tab.
3. Create a new secret called TESTPYPI_TOKEN and copy-paste the token from the first step.

For the time being, this should be enough. I only need TestPyPI. If the second Action is ready to be published, we need to do the same procedure for PyPI.

---

During the work on this issue, I've came more and more to the conclusion that we probably need a dedicated repository for our own GH Actions. As we have our own domain now, this should be fairly easy. Separating source code from infra structure looks like a good idea (as it is done with other projects as well).

At the moment, it is not really needed, but maybe I need to refactor that in the future.

[scls19fr]

I tried to create a token on TestPypi
So I went to https://test.pypi.org/manage/account/token/

I noticed that I can currently only create a token with "Entire account" scope... which is not something I want (for security reason).

So I think I first need to register semver on TestPyPI... to create token just for this project.

So I did (according to https://packaging.python.org/tutorials/packaging-projects/ )

$ python3 -m twine upload --repository-url https://test.pypi.org/legacy/ dist/*
Enter your username: scls
Enter your password:
Uploading distributions to https://test.pypi.org/legacy/
Uploading semver-2.9.0-py2.py3-none-any.whl
100%|██████████████████████████████████████| 16.8k/16.8k [00:01<00:00, 13.4kB/s]
NOTE: Try --verbose to see response content.
HTTPError: 403 Client Error: The credential associated with user 'scls' isn't allowed to upload to project 'semver'. See https://test.pypi.org/help/#project-name for more information. for url: https://test.pypi.org/legacy/

So I went to https://test.pypi.org/help/#project-name

Why isn't my desired project name available?
Your publishing tool may return an error that your new project can't be created with your desired name, despite no evidence of a project or release of the same name on PyPI. Currently, there are three primary reasons this may occur:

The project name conflicts with a Python Standard Library module from any major version from 2.5 to present.
The project name has been explicitly prohibited by the PyPI administrators. For example, pip install requirements.txt is a common typo for pip install -r requirements.txt, and should not surprise the user with a malicious package.
The project name has been registered by another user, but no releases have been created.

I tried with a bad password

$ python3 -m twine upload --repository-url https://test.pypi.org/legacy/ dist/*
Enter your username: scls
Enter your password: badpass
Uploading distributions to https://test.pypi.org/legacy/
Uploading semver-2.9.0-py2.py3-none-any.whl
100%|██████████████████████████████████████| 16.8k/16.8k [00:00<00:00, 22.6kB/s]
NOTE: Try --verbose to see response content.
HTTPError: 403 Client Error: Invalid or non-existent authentication information. for url: https://test.pypi.org/legacy/

so that's not a user issue... but a project name issue on TestPyPi.

[scls19fr]

I noticed what is wrong...

@tomschr is cybersquatting this project name on TestPyPi 😆

https://test.pypi.org/project/semver/

@tomschr could you go to
https://test.pypi.org/manage/project/semver/collaboration/
and grant me owner role there?

[tomschr]

@scls19fr Ohh, I'm sorry! 😲 Argghh, I was under the impression that each name is bound to a user name. Right, I've added that to test with the GH Actions (which kind of worked), but with good intentions. Again sorry for the confusion.

Which is your account name on TestPyPI? Of course, I can add you as another owner. The name scls19fr doesn't work.

[scls19fr]

No problem @tomschr
My TestPyPi account name is same as my PyPi account name : scls

[tomschr]

@scls19fr Ok, you are now added. 😉

[scls19fr]

Ok API token named semver_upload for project semver on TestPyPi created.
I can't find your email so I sent you a PM on LinkedIn. Could you have a look?

[tomschr]

Ok API token named semver_upload for project semver on TestPyPi created.

Thanks!
As you know, I'm a big friend of consistency. 😉 As we will need other tokens (for pypi.org) probably, we should make them distinguish easily. 😉

Therefor, I would suggest to better name the token TESTPYPI_TOKEN (or TESTPYPI_UPLOAD_TOKEN or TESTPYPI_SEMVER_UPLOAD_TOKEN choose what you prefer) on test.pypi.org and github.com (under the security settings). Does that make sense? 😉

[scls19fr]

Ok new API token named TESTPYPI_SEMVER_UPLOAD_TOKEN

I tried to also add k_bx on TestPyPi (@k-bx on Github / k_bx on PyPi) but he doesn't seems to have an account there.

[tomschr]

Thank you very much Sébastien! 👍

@ppkt Karol, do you want to be also on the TestPyPI party? 😄 🎉 If yes, just let us know.

[tomschr]

You can find the token TESTPYPI_SEMVER_UPLOAD_TOKEN also in the GitHub Secrets tab.

[tomschr]

On example can be found at https://github.com/escaped/cookiecutter-pypackage/blob/master/.github/workflows/release.yml or https://github.com/escaped/cookiecutter-pypackage/blob/master/%7B%7B%20cookiecutter.project_slug%20%7D%7D/.github/workflows/release.yml