NEP: add NEP on a Python API cleanup for NumPy 2.0 #23537

rgommers · 2023-04-06T12:33:48Z

This is still draft, but close enough for now. It was discussed in the NumPy 2.0 developer meeting a few days ago, with mostly positive feedback.

mhvk

I like this very much. Some comments in-line, as well as suggestions that perhaps go too far given the wish to ensure that most code that runs on 2.0 would also run on 1.2*

mhvk · 2023-04-06T19:06:32Z

doc/neps/nep-0052-python-api-cleanup.rst

+    # Legacy (prefer not to use). Third grouping in the reference guide.
+    numpy.char
+    numpy.distutils
+    numpy.matrixlib  # or deprecate?


We put in a PendingDeprecationWarning for 1.15 (#10142) - I'd say it could go to a real deprecation soon and then be removed for 2.0...

I looked at this, it's np.matrix that has the PendingDeprecationWarning. That's a topic that may be unblocked soon, because there's work on shuffling around *_matrix and *_array in scipy.sparse. But it is unrelated to removing the matrixlib namespace, because that namespace is pretty pointless, its contents are already imported into the main namespace with from .matrixlib import *.

doc/neps/nep-0052-python-api-cleanup.rst

mhvk · 2023-04-06T19:19:38Z

doc/neps/nep-0052-python-api-cleanup.rst

+    numpy.exceptions
+    numpy.fft
+    numpy.linalg
+    numpy.ma


Numpy 2.0 might be the last chance to really think through what MaskedArray should be like... If unchanged, perhaps putting it under legacy makes more sense?

Agreed. I'd really like to see @ahaldane's rewrite included for 2.0. That needs a champion to move it forward though. The current numpy.ma isn't salvageable I'd say, and if we keep it then I agree with you that "legacy" makes sense. We're then effectively abandoning masked data in favor of proper missing data support as provided by dataframe libraries.

Maybe @ahaldane and I can have a joint look; my sense is that some of the API really has to be different. Note that I ended up rewriting masked data for astropy, as we really need to be able to have proper masked subclasses; see https://docs.astropy.org/en/latest/utils/masked/index.html. I think my scheme has a bit more hope of being useful also for different array implementations (say, masked dask), though I have not actually tested that.

That would be great!

I got some more feedback on legacy status from numpy.ma during the community meeting. After some more discussion on what legacy means (de-emphasize, move to a different place in the docs, suggest more appropriate alternatives - but no deprecation warnings) people were fine with that. Everyone agreed that numpy.ma is not in good shape, and it doesn't have a maintainer; a replacement is very much welcome (inside or outside NumPy).

Moved numpy.ma to the legacy section now.

doc/neps/nep-0052-python-api-cleanup.rst

mhvk · 2023-04-06T19:26:04Z

doc/neps/nep-0052-python-api-cleanup.rst

+
+.. note::
+
+    TBD: will we preserve ``np.lib`` or not? It only has a couple of unique


Agreed, but it would help here to know where would things like arraysetops and nanfunctions would live (I realize this is not so important from a user perspective, since the functions are in the top namespace, but it is probably also good to think from the developer perspective).

I'd say that they have to move somewhere, which is a single git mv plus use of git-blame-ignore-revs to not mess up the usefulness of git blame. Where it ends up exactly is less important to me, at least from the perspective of this NEP. It could be:

numpy/_lib/nanfunctions.py

numpy/lib/_nanfunctions.py

or something more different than adding a single underscore

doc/neps/nep-0052-python-api-cleanup.rst

mhvk · 2023-04-06T19:52:55Z

doc/neps/nep-0052-python-api-cleanup.rst

+.. code:: python
+
+   >>> # np.intp is different, but compares equal too
+   >>> np.int64 == np.int_ == np.dtype('i8') == np.sctypeDict['i8']


I've always disliked this mixing up of a data type with a scalar variable class of that data type. If we can rid of those... (and perhaps have definitions for the standard dtypes, i.e., f8 = np.dtype('f8')? see below too)

I'd like __eq__ on dtypes to do the expected thing for Python objects indeed.

Less sure about f8 = np.dtype('f8') - I dislike the character codes a lot, they're too obscure. Scalars will hopefully go away completely at some point (they're not necessary for anything, and everyone seems to be in favor of them being removed eventually, it's just too big a lift for 2.0 it looks like).

What I'd like is clear principles, like:

We have two sets of names for things:

the "bit-suffixed names" like int32, float64 - with canonical names like bool, void, object added to that

the "C names" like short, int, double, clongdouble, etc.

Other aliases that don't fall into these categories should either be removed, or de-emphasized in case we need them for backwards compatibility

Avoid creating or using array scalars by instantiating them with np.dtype directly as much as possible, it's typically an anti-pattern

Hmm, I'm super-used to using the string variants, but definitely can see your point. The annoyance is that type(int32) is not actually a dtype subclass. Anyway, that's for #23358; agreed here that a minimal set is a good idea.

h-vetinari

Exciting stuff! Not much to comment ATM, just a typo.

doc/neps/nep-0052-python-api-cleanup.rst

…n too

[skip ci]

[skip azp] [skip actions] [skip cirrus]

rgommers · 2023-05-18T13:50:52Z

I propose to merge this with Draft status. It's in pretty good shape, and it'd be nice to make it more visible and have the rendered html version to point at.

WarrenWeckesser · 2023-05-18T15:45:43Z

I propose to merge this with Draft status.

According to NEP 0, before merging, the next step is to announce the NEP on the mailing list.

rgommers · 2023-05-18T15:50:47Z

Thanks Warren, good point - I thought I did that already last month, but it appears that my memory is faulty. Will write an email now.

[skip ci]

doc/neps/nep-0052-python-api-cleanup.rst

shoyer · 2023-05-20T00:35:19Z

doc/neps/nep-0052-python-api-cleanup.rst

+    numpy.ctypeslib
+    numpy.emath
+    numpy.f2py  # only a couple of public functions, like `compile` and `get_include`
+    numpy.lib.stride_tricks


What about moving stride_tricks to numpy.stride_tricks, and considering the rest of numpy.lib legacy stuff?

That's a reasonable option; there's a few more public things in numpy.lib though:

lib.format (module with multiple functions)

lib.mixins.NDArrayOperatorsMixin (module with a single object)

lib.NumPyVersion

lib.Arrayterator

I think that's all, but hard to be 100% sure. It's hard to find good places for each of them, so I had in mind to keep np.lib as a grab bag of stuff that does not deserve its own namespace right below the main namespace.

We can probably get rid of Arrayterator; I'm not sure about the rest.

I've added the list above to the note. I'd like to resolve this later (post-merge of this NEP as Draft). Leaning towards keeping numpy.lib, but either way is not ideal.

[skip ci]

rgommers · 2023-05-29T09:22:10Z

I addressed the few comments that came in after posting the draft NEP to the mailing list two weeks ago; I think it'd be good to merge this with Draft status now.

mattip · 2023-05-29T10:58:01Z

Merged. It would be good to have a follow-on PR that

adds a link to the mailing list announcement/discussion
tighten up the language a bit in the "Cleaning up the main namespace" section, there is a mixture of "can be removed", "will be removed", and "New namespaces are introduced"

mattip · 2023-05-29T10:58:20Z

Thanks @rgommers

rgommers · 2023-06-14T18:20:27Z

Merged. It would be good to have a follow-on PR that

Thanks for the suggestions Matti. I'll take this along soon. It won't be the last update of this NEP.

leofang · 2023-06-14T19:00:15Z

Q: Is there a general announcement/milestone post/issue tracker that we can link to regarding NumPy 2.0? Would be useful to refer to it in this NEP and elsewhere 🙂

rgommers · 2023-06-15T09:04:22Z

Q: Is there a general announcement/milestone post/issue tracker that we can link to regarding NumPy 2.0? Would be useful to refer to it in this NEP and elsewhere slightly_smiling_face

Good question @leofang. There isn't at this point - only a project board (https://github.com/orgs/numpy/projects/9, not useful for this purpose) and a scattered number of proposals, mailing list threads, meeting notes, etc.

I think we could use a pinned issue for this, since there's bound to be more and more questions like these. A problem of course is that it may also attract a lot of random questions, at which point it becomes too noisy. We could perhaps do it as a locked pinned issue, purely meant for maintainers to unlock, post update, and lock again. Pointing to relevant updates, docs, etc. as they get posted. WDYT?

seberg · 2023-06-19T08:45:10Z

Might as well do that for now just to have something brief somewhere.

rgommers · 2023-07-31T11:02:54Z

Tracking issue created: #24300

bsipocz · 2023-08-30T17:30:26Z

I was recently linking to this NEP as a reference for the upcoming 2.0 changes, and noticed it's still in "Draft" state. I wonder whether it could be updated to accepted as the implementations are basically well underway?

(I suppose more NEPs in the "Draft" status could be updated, but that's a bigger scope than this comment, and I'm happy to open an issue for it).

rgommers · 2023-09-15T13:51:42Z

xref gh-24620 to mark this NEP as Accepted - still under review but should get merged soon.

rgommers added the component: NEP label Apr 6, 2023

mhvk reviewed Apr 6, 2023

View reviewed changes

This was referenced Apr 7, 2023

API: Add DType classes into new numpy.dtypes module #23358

Merged

What is Public API astropy/astropy#13910

Open

ngoldbaum mentioned this pull request Apr 7, 2023

DEP: deprecate np.math and np.lib.math #23555

Merged

h-vetinari reviewed Apr 23, 2023

View reviewed changes

doc/neps/nep-0052-python-api-cleanup.rst Outdated Show resolved Hide resolved

eendebakpt mentioned this pull request May 17, 2023

EHN: make real_if_close elementwise #11375

Open

rgommers and others added 9 commits May 18, 2023 13:44

DOC: add NEP on a Python API cleanup for NumPy 2.0

8949403

Edit and add questions to NEP52: numpy API refactoring

6727199

Adopt some of Stefan's comments in the text, plus a few more tweaks

71b2114

Adopt some of Stefan's comments on namespaces; put back Ralf's versio…

6ac2330

…n too

Fix typos; remove alternative module grouping

7bcf21a

Updates for some minor review comments/discussion

e63d6e5

Add in/out of scope for this NEP

409a864

minor style fix

b91a0a1

[skip ci]

DOC: address review comments on Python API cleanup NEP.

b032b05

[skip azp] [skip actions] [skip cirrus]

rgommers force-pushed the nep-python-api branch from dae48d6 to b032b05 Compare May 18, 2023 12:11

STY: reflow the text and minor typo fixes

0687bf6

[skip ci]

BvB93 reviewed May 18, 2023

View reviewed changes

doc/neps/nep-0052-python-api-cleanup.rst Show resolved Hide resolved

adeak reviewed May 19, 2023

View reviewed changes

doc/neps/nep-0052-python-api-cleanup.rst Outdated Show resolved Hide resolved

doc/neps/nep-0052-python-api-cleanup.rst Outdated Show resolved Hide resolved

shoyer reviewed May 20, 2023

View reviewed changes

mtsokol mentioned this pull request May 28, 2023

API: deprecate compat and selected lib utils #23830

Merged

DOC: address review comments on numpy.lib and lazy importing

02e6615

[skip ci]

mattip merged commit fa284a5 into numpy:main May 29, 2023
7 of 9 checks passed

rgommers deleted the nep-python-api branch June 13, 2023 13:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

NEP: add NEP on a Python API cleanup for NumPy 2.0 #23537

NEP: add NEP on a Python API cleanup for NumPy 2.0 #23537

rgommers commented Apr 6, 2023

mhvk left a comment

mhvk Apr 6, 2023

rgommers May 18, 2023

mhvk Apr 6, 2023

rgommers Apr 7, 2023

mhvk Apr 7, 2023

rgommers Apr 7, 2023

rgommers Apr 12, 2023

rgommers May 18, 2023

mhvk Apr 6, 2023

rgommers Apr 7, 2023

mhvk Apr 6, 2023

rgommers Apr 7, 2023

rgommers Apr 7, 2023

mhvk Apr 7, 2023

h-vetinari left a comment

rgommers commented May 18, 2023

WarrenWeckesser commented May 18, 2023

rgommers commented May 18, 2023

shoyer May 20, 2023

rgommers May 20, 2023

rgommers May 29, 2023

rgommers commented May 29, 2023

mattip commented May 29, 2023

mattip commented May 29, 2023

rgommers commented Jun 14, 2023

leofang commented Jun 14, 2023

rgommers commented Jun 15, 2023

seberg commented Jun 19, 2023

rgommers commented Jul 31, 2023

bsipocz commented Aug 30, 2023

rgommers commented Sep 15, 2023


		.. note::

		TBD: will we preserve ``np.lib`` or not? It only has a couple of unique

NEP: add NEP on a Python API cleanup for NumPy 2.0 #23537

NEP: add NEP on a Python API cleanup for NumPy 2.0 #23537

Conversation

rgommers commented Apr 6, 2023

mhvk left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

h-vetinari left a comment

Choose a reason for hiding this comment

rgommers commented May 18, 2023

WarrenWeckesser commented May 18, 2023

rgommers commented May 18, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rgommers commented May 29, 2023

mattip commented May 29, 2023

mattip commented May 29, 2023

rgommers commented Jun 14, 2023

leofang commented Jun 14, 2023

rgommers commented Jun 15, 2023

seberg commented Jun 19, 2023

rgommers commented Jul 31, 2023

bsipocz commented Aug 30, 2023

rgommers commented Sep 15, 2023