More frequent patch releases

[hackermd]

During the development of highdicom, we having been regularly contributing bug fixes to pydicom, but recently found ourselves waiting for critical fixes to be included in a new release and for the next version of the package to become available on pypi. This currently prevents us from releasing highdicom more frequently.

Would it be possible to create and publish patch releases for bug fixes more frequently? Can we somehow help in expediting the release life cycle?

[darcymason]

Would it be possible to create and publish patch releases for bug fixes more frequently?

Super-quick answer: Sure.

Longer answer - to me the workload around patch releases is mainly:

• ensuring release notes are updated to include all fixes (we're pretty good now at updating the notes in each PR, but sometimes a few slip through without that).
• cherry-picking only bug fixes from 'master' into the release branch (e.g. 2.2.X)

The actual release is now done through a github actions workflow and is not too much work, especially for patch releases.

Another question though - what kind of frequency would you be looking for? I wouldn't be against doing minor releases more often (I slipped on this latest one, but could try to do better in future 😉)

Can we somehow help in expediting the release life cycle?

Given what was said above, if you wanted to update any missing notes in a master PR, then make a PR against the release branch (e.g. 2.2.X) of cherry-picked bug fix commits, that would be a great help. In some cases, I suppose there might need to be some prior discussion if there are multiple inter-related fixes/issues on the go.

[mrbean-bremen]

To follow up on this: I have had the same experience (waiting for the release of bug fixes) with other projects several times, so I fully support this.
Given that releases are now relatively cheap, I would also go for releases with new features more often. I know I haven't been very active in pydicom for quite some time now, but I'm happy to help to get out releases more often. In the (small) projects I maintain I tend to release as soon as a new feature or a relevant bug fix is added, basically never doing any cherrypicking - the only exception being fixes backported to a major version (for example for Python 2 support).

I have been a fan of "Release early, release often" since I read The Cathedral and the Bazaar, and now this is much easier than at the time that was written - I just had to add my 2 eurocent here :)

[hackermd]

Another question though - what kind of frequency would you be looking for? I wouldn't be against doing minor releases more often (I slipped on this latest one, but could try to do better in future 😉)

I tend agree with @mrbean-bremen's response and would say as often as possible :) If the process is more or less automated, a release after every accepted PR would be great.

Given what was said above, if you wanted to update any missing notes in a master PR, then make a PR against the release branch (e.g. 2.2.X) of cherry-picked bug fix commits, that would be a great help. In some cases, I suppose there might need to be some prior discussion if there are multiple inter-related fixes/issues on the go.

I would be happy to do that if that is the preferred workflow.
An alternative approach would be to keep larger features in separate development branches (regularly rebasing them onto the master branch if necessary) and draft new releases directly from the master branch after pull requests have been merged. This would have the advantage that preparing a release would become less cumbersome, which may result in an increased release frequency. What do you think?

[pieper]

a release after every accepted PR would be great.

We've been using semantic-release for this in dcmjs and I have found it very handy to have a link between the commit messages and the release types and then automatic packaging. It makes the intent of the commits/PRs clear and then handles the details. There's a python option but I have never tried it.

[darcymason]

If the process is more or less automated, a release after every accepted PR would be great

We've been using semantic-release for this in dcmjs

I'm not sure I'm ready for that kind of granularity on releases - and besides, by my understanding of semantic versioning, patch releases should only be bug fixes, not new features.

There is also always the option of people pip installing from master if they wish, or working with a local clone and pip install -e pydicom in it and git pull whenever they like.

An alternative approach would be to keep larger features in separate development branches (regularly rebasing them onto the master branch if necessary) and draft new releases directly from the master branch after pull requests have been merged.

Hmmm, again I know this is a common workflow, but I'm reluctant to change what we have now, at least at this point. I think they are pretty equivalent, just comes down to what is labelled what. 'Master' for us is basically 'dev' - but since most of the action happens there, and not generally over long periods with conflicts emerging, I think it works fine.

There is also a worry in my mind that fixes sometimes inadvertently create new bugs - and at least if that occurs in master, such a new error can be confined to a small number of users rather than everyone getting hit.

But I am hearing the message from several key users here, that more releases would be much appreciated. I'd suggest we try the increased patch release path discussed above first and see how that goes.

[hackermd]

There is also a worry in my mind that fixes sometimes inadvertently create new bugs - and at least if that occurs in master, such a new error can be confined to a small number of users rather than everyone getting hit.

I think if we release more frequently this will become less of a concern, since we can just release a new patch version that fixes any introduced bugs.

[hackermd]

An alternative approach would be to keep larger features in separate development branches (regularly rebasing them onto the master branch if necessary) and draft new releases directly from the master branch after pull requests have been merged.

Hmmm, again I know this is a common workflow, but I'm reluctant to change what we have now, at least at this point. I think they are pretty equivalent, just comes down to what is labelled what. 'Master' for us is basically 'dev' - but since most of the action happens there, and not generally over long periods with conflicts emerging, I think it works fine.

There is also a worry in my mind that fixes sometimes inadvertently create new bugs - and at least if that occurs in master, such a new error can be confined to a small number of users rather than everyone getting hit.

But I am hearing the message from several key users here, that more releases would be much appreciated. I'd suggest we try the increased patch release path discussed above first and see how that goes.

Ok! Let's stick to the existing workflow. I will work on a patch release branch (bugfixes only) as you suggested.

[hackermd]

There is also always the option of people pip installing from master if they wish, or working with a local clone and pip install -e pydicom in it and git pull whenever they like.

That's true and works fine for applications. However, for a library (e.g., highdicom) that is itself distributed via pypi, this is not a great option.

[mrbean-bremen]

by my understanding of semantic versioning, patch releases should only be bug fixes, not new features.

That's true, but that just means that you increase the minor version instead of the patch version for a new release. It would just mean that the minor version increaes faster than before.

There is also always the option of people pip installing from master if they wish, or working with a local clone and pip install -e pydicom in it and git pull whenever they like.

As already mentioned, this is not an option for libraries, and in my experience, hardly anyone does it, or wants to do this, because it feels like an unstable version.

An alternative approach would be to keep larger features in separate development branches

This is basically what we do in large PRs. I'm generally not a friend of large PRs or long-running feature branches (they tend to introduce regressions, especially with several such branches existing at once), but try to go for incremental changes as much as possible, and holding the main branch stable.

There is also a worry in my mind that fixes sometimes inadvertently create new bugs - and at least if that occurs in master, such a new error can be confined to a small number of users rather than everyone getting hit.

That is true, and the most valid concern IMO, but there are a couple of things that would mitigate that.
First, for major changes you already have the habit of making pre-releases. This increases the number of users that will see the release compared to users working on the master branch (which very few IMO, as I already mentioned). Second, if pydicom is used in a product, it certainly will have a fixed version, and version changes will be monitored and tested. Also - if indeed a regression happens, we can always react very fast to it, but getting out a patch release that fixes the problem, or, in the worst case, reverses the change. And last - with our current unit test coverage, regressions are less likely to appear than before.

Ok, I think I made my point - will stop preaching now 😏

[scaramallion]

I'm happy either way. I do think it gets increasingly difficult to do cherry-picked bugfix releases as more and more regular PRs get merged, though.

[hackermd]

That has been my experience as well. My preferred approach is to just release as soon as bug fixes or new features are available, being mindful about releasing bug fixes before releasing new features.

[hackermd]

if you wanted to update any missing notes in a master PR, then make a PR against the release branch (e.g. 2.2.X) of cherry-picked bug fix commits, that would be a great help. In some cases, I suppose there might need to be some prior discussion if there are multiple inter-related fixes/issues on the go.

@darcymason I created a pull request for the patch release targeting the 2.2.X branch: #1489

[hackermd]

I just saw the 2.2.1 release. Thanks for the super-fast turnaround @darcymason!

[darcymason]

I just saw the 2.2.1 release. Thanks for the super-fast turnaround @darcymason!

No problem. With you prepping the PR, and the GitHub actions script to publish to PyPi, it was quite fast to do.

[hackermd]

@darcymason @scaramallion the last patch release (v2.2.1) went well in my opinion. However, I still think that releases are not frequent enough and release planning is too strict. While I generally think it's good practice to plan for the next release (e.g., v2.3), I also think that this can create problems. For example, I would now like to update the dictionary to make new attributes available (#1514) and it would be desirable for those changes to become available via a package on pypi as soon as possible. This should probably not just be a patch release, but I also don't want to wait for all the other features planned for v2.3 to be finalized beforehand.

In conclusion, I suggest moving away from strict release planning (for minor versions) and instead just release a new version as soon as critical changes have been introduced (and tested).

Thoughts?

[darcymason]

I suggest moving away from strict release planning (for minor versions)

I disagree with the sentiment that our current system is overly strict - in fact our next release checklists are often only partially completed, and items are constantly pushed forward or changed. They are just a guide of what kinds of things we might want to work on next. The only thing I feel strict about is thinking ahead to breaking changes in major versions and having deprecation warnings some time ahead; I don't think that is affected by this, it's just that perhaps more minor versions might occur than in the past.

instead just release a new version as soon as critical changes have been introduced (and tested).

I can agree with this (of course what defines 'critical' is always open to interpretation).

For example, I would now like to update the dictionary to make new attributes available (#1514) and it would be desirable for those changes to become available via a package on pypi as soon as possible.

I can support this - if you don't mind doing the PR similar to last time I'm happy to get out a release.

Bottom line, with only a couple of data points, I still don't see evidence for any big change in how we do things. And I believe there is a whole other category of less heavy users, less experienced users, perhaps, that do benefit from releases being somewhat curated rather than following all commits. I could comment further on that if you wish, but hard data is non-existent so it is more extrapolation from my own experience and interactions.

But I encourage you to keep asking for updates as needed - sorry if I have seemed reluctant in the past, but I agree that the last one worked well, and I'd like to keep going with that for now.

[hackermd]

instead just release a new version as soon as critical changes have been introduced (and tested).

I can agree with this (of course what defines 'critical' is always open to interpretation).

To me, "critical" in this context means either new features, bug fixes, or new versions of the standard as compared to improvements to documentation or internal refactoring - essentially any changes that would require applications or other libraries that depend on pydicom to increase the package version in setup.py or pyproject.toml.

Bottom line, with only a couple of data points, I still don't see evidence for any big change in how we do things. And I believe there is a whole other category of less heavy users, less experienced users, perhaps, that do benefit from releases being somewhat curated rather than following all commits.

I don't see how more frequent releases would negatively impact less heavy or less experienced users. To the contrary, more frequent, incremental changes are probably less risky. There is in my opinion no good argument for letting changes linger in the master branch for a long time without being released. I tend argue that the hurdle should be merging into master, not releasing.

But I encourage you to keep asking for updates as needed - sorry if I have seemed reluctant in the past, but I agree that the last one worked well, and I'd like to keep going with that for now.

Ok. I'll then draft a v2.3.0 release via a separate PR and branch as soon as #1514 will be merged.

My concern with this approach is that it will get increasingly messy. For example, this release will "invalidate" #1464. Therefore, I thought it would be easier to formulate milestones for features rather than concrete releases/package versions and just draft a new release as soon as something "critical" (see definition above) has been merged into master.

[darcymason]

I respect your expertise and opinions, and I have lots of rebuttals and expansions of arguments I've already made, but it's not worth my time to make them, so for now I'm going to pulll the BDFL card, and move on. But I have considered all the arguments, and they may eventually affect my thinking as we go forward.

I'll head over to #1514 and we should be able to get that merged.

[hackermd]

Thanks for reviewing and merging #1514. I've created a PR for another patch release 2.2.2 (#1518).

[mrbean-bremen]

What I usually do after a fix that is not critical is asking if the reporter needs a released version, or is ok with the master and a later release, and make a patch release depending on the response.