Numpy 2.0 release thoughts

[betatim]

Some notes from the EuroSciPy maintainer track about Numpy 2.0

You can track the progress Numpy is making towards 2.0 and what changes are coming in numpy/numpy#24300.

Some things we should consider doing:

• pin to numpy<2 in the next releases to reduce the number of people who will install a version of scikit-learn that is incompatible with the numpy 2 version
• add some code in scikit-learn that detects that numpy 2 is installed and output a good error message for users
  • Olivier: do we really need this? Maybe the upper-bound pinning is enough.
• numpy 2 might come out at the end of 2023 or early 2024
• modify existing conda-forge packages to include a numpy<2 constraint
  • unclear if we can do this on PyPi as well
  • How to deal with upcoming Numpy 2.0 release and existing scikit-learn releases conda-forge/scikit-learn-feedstock#227
• once numpy 2.0 has been released we should build against that instead of "oldest-supported-numpy"
  • already do this now in the nightlies?
  • make sure that this works while still having 1.x.quite-old as the minimum required numpy
  • We no longer build against oldest-supported-numpy for Python 3.9 thanks to BLD: Build scikit-learn nightlies using NumPy nightlies #27735 that was merged in main before the release of scikit-learn 1.4.

cc @ogrisel

[adrinjalali]

well, last we talked about putting upper bounds, we decided not to 😁 (xref: #26869)

[betatim]

Yeah, I'd still vote against upper limits though. It feels like this is a once in a life time situation. (I hope this comment will age well ;) ).

A thing that will make this tricky is that a lot of the changes for Numpy 2.0 aren't yet merged. This makes it hard to find out how much breakage there will be. I still hope that maybe older versions of scikit-learn can "just work" with Numpy 2.0 if we rebuild the wheels against Numpy 2 as part of a patch release? But yeah.

[lorentzenchr]

How about a CI that builds with nightly numpy?
Do we anticipate any breakage so far?

[betatim]

We have a nightly (not sure if it is strictly nightly or more like "every few nights") CI and so far I think it has been the normal amount of things breaking and then getting fixed.

Trying to think about it now, I realised that it might be useful for Numpy to make a list of "anticipated big changes for Numpy 2.0" where we can go and see if they've been merged already or not. As a way to get an overview of how much more there is (potentially) to come.

edit: (of course) they already have an issue like that: numpy/numpy#24300

[ogrisel]

In #27899 I enabled the scikit-learn test suite to run against numpy 2.0.0.dev0. There were only two estimators to fix to get 100% of the scikit-learn test suite to pass.

So maybe we actually do not need to upper bound the dependency on numpy 2.0?

@seberg do you still envision breaking changes for numpy 2 that have not yet been implemented in the numpy dev branch?

[seberg]

I do envision a couple, although there is a reasonable chance that you won't notice, by now, I guess. But if you do a release before we do an rc (hopefully end of the year/early next year), having the pin is probably better.

[ogrisel]

Thanks, we shall add the upper bound in our dependency to numpy in the scikit-learn 1.4.X release branch then.

[rgommers]

With numpy 2.0.0rc1 and scipy 1.13.0 out, I think you should be able to make a numpy 2.x-compatible release now. It'd be great to see that relatively soon, since a lot of other packages are going to depend on scikit-learn for doing/testing their own numpy 2.x-compatible releases.

[betatim]

I think the next release will be 1.5.x and the plan is to get it done before May.

As an aside, the nightly from https://anaconda.org/scientific-python-nightly-wheels/scikit-learn is being built with Numpy 2. (strictly speaking the Numpy nightly). So there is some amount of testing people can do with that.

[lesteve]

Just for clarity I chatted with @glemaitre to make sure my understanding was correct.

I think the plan is currently:

• do a bugfix release 1.4.3 with Numpy 2 compatibility soonish (not a new release, i.e. 1.5.x which would be a lot more work).
• start working on a 1.4.3 release candidate next week and do a release shortly after this if there are no issues

[rgommers]

This is now all done, with 1.4.2 built against numpy 2.0.0rc1, right?

[glemaitre]

@rgommers Yes, we release 1.4.2 on PyPI and build against NumPy 2.0.0.rc1.
We did not release on (conda-forge/scikit-learn-feedstock#251). We did not see a compatible release for SciPy 1.13. We were also not sure how the pin_compatible is going to work there. Do you have any advice @rgommers? Is there a migration that is going to happen on conda-forge side?

[rgommers]

You can release 1.4.2 on conda-forge with the same build configuration as 1.4.1 (so built against lowest-supported numpy 1.2x.

The conda-forge team will do a coordinated rebuild for all packages to add support for 2.0 before the final 2.0.0 release. The exact details of that are still under discussion (see conda-forge/numpy-feedstock#311 and issues linked from there); the pin_compatible mechanism needs some change. But that's the same for every package, so individual feedstock maintainers don't have to worry about figuring it out now.

[glemaitre]

Great thanks.

[adrinjalali]

So I guess we can close this one?