Upgrade numpy to 1.21.4 #1934

leopsidom · 2021-11-07T06:20:33Z

Description

Currently pyodide has numpy @1.17.5 but the newest numpy version is 1.21.4. This PR is to keep the numpy package in pyodide up to date. Some changes made in this PR include:

patches removed: add-emscripten-cpu.patch and rm-duplicate-symbols-link.patch. These patches got removed because the underlying issue has been fixed in the numpy package.
patches added:
1. fix-invalid-asm-instruction.patch to fix invalid asm instructions
2. limit-cpu-baseline.patch this is to limit cpu baseline to AVX so it doesn't try to build SIMD feature AVX512 that emscripten doesn't support;
3. not-build-lapack-lite-as-64-bit.patch this is to prevent building lapack-lite as 64 bit integer mode which emscripten doesn't support;
patches modified (both are needed because the original files got changed and the original patches can no longer apply):
1. fix-static-init-of-nditer-pywrap.patch
2. use-local-blas-lapack.patch

Notes: in numpy build file, I'm currently setting:

self.cpu_baseline = "avx"
self.cpu_dispatch = "avx"

I think this means there are still basic SIMD optimization built with this which is supported by emscripten. We might want to check if we want to enable it yet.

Issue: #1827

Checklists

Not all of these steps are necessary for every PR.

Add a CHANGELOG entry or explain why an entry is not needed
Add / update tests
Add new / update outdated documentation

leopsidom · 2021-11-07T06:27:43Z

packages/numpy/patches/disable_optimization.patch

@@ -0,0 +1,13 @@
+diff --git a/numpy/distutils/command/build.py b/numpy/distutils/command/build.py


Unintentionally included this file. This patch is not needed.

ryanking13 · 2021-11-07T09:23:11Z

This must've been a hard job. I appreciate your effort!

A minor comments:

patches added:

fix-invalid-asm-instruction.patch to fix invalid asm instructions

limit-cpu-baseline.patch this is to limit cpu baseline to AVX so it doesn't try to build SIMD feature AVX512 that emscripten doesn't support;

not-build-lapack-lite-as-64-bit.patch this is to prevent building lapack-lite as 64 bit integer mode which emscripten doesn't support;

I think adding these descriptions into the patch file would help other people who later see patch files.

It would be great if we add some tests for newly available features of numpy 1.21 (and if possible, 1.18, 1.19, 1.20 also). I'm not sure which are the core new functionalities of these numpy versions, but we could find out from their release notes.

rth · 2021-11-07T10:52:46Z

Thanks a lot for doing this work @leopsidom !

I think this means there are still basic SIMD optimization built with this which is supported by emscripten. We might want to check if we want to enable it yet.

Right now the CI fails with,

emcc: error: Passing any of -msse, -msse2, -msse3, -mssse3, -msse4.1, -msse4.2, -mavx, -mfpu=neon flags also requires passing -msimd128!

which seems to be expected as discussed in emscripten-core/emscripten#12714. I'm very in favor of investigating what we need to use SIMD a separate issue (and in particular check that it doesn't break Safari), for now maybe the easiest would be,

self.cpu_baseline = "min"
self.cpu_dispatch = "min"

to disable simd ?

Add / update tests

It would be great if we add some tests for newly available features of numpy 1.21 (and if possible, 1.18, 1.19, 1.20 also).

This is another subject we should improve on in Pyodide. Historically we have not run much tests for packages, because while we have the ability to run the package test suites in Pyodide #69 running them would be very slow, and there are also some known failure (cf related open issues about numpy). So we have just written some sanity tests and that's it. However now we have much more CI resources than before and I think we could run them but maybe triggered manually and not for each commit. Improving the package test situation is also somewhat a blocker for #1577

So there is a lot work that can be done there, for this PR for instance running the numpy test suite and report new issues would probably be sufficient. We can also do this after merging of this PR. The only thing maybe we should add a test for, is one of the operations that uses SIMD to make sure that calling it doesn't error after all the workaround that were done here.

leopsidom · 2021-11-07T18:33:11Z

I think adding these descriptions into the patch file would help other people who later see patch files.

Good idea. I have added the description to the corresponding patch files.

It would be great if we add some tests for newly available features of numpy 1.21

I think numpy has it's own test suite. Ideally we might want to see how we can run the tests in pyodide as commented by @rth.

leopsidom · 2021-11-07T18:38:23Z

Right now the CI fails with,
emcc: error: Passing any of -msse, -msse2, -msse3, -mssse3, -msse4.1, -msse4.2, -mavx, -mfpu=neon flags also requires passing -msimd128!

Yeah, that means we have to pass an extra cflag -msimd128 to emscripten when compiling SIMD features. It looks like an explicit restriction that emscripten tries to call out about what it actually supports.

for now maybe the easiest would be
self.cpu_baseline = "min"
self.cpu_dispatch = "min"

Sounds good. I've switched both to min. Though I don't know if this completely disables SIMD. We might have to use self.disable_optimization = True to completely disable it.

hoodmane · 2021-11-10T18:59:02Z

Scipy build is failing because it can't link npymath.

leopsidom · 2021-11-11T06:51:41Z

Hmm yeah, will check a little later this week.

leopsidom · 2021-11-13T06:18:47Z

The npymath library issue can be fixed by updating NUMPY_LIB here: https://github.com/pyodide/pyodide/blob/main/packages/Makefile#L6. But it seems there are some more compatibility issue between numpy and scipy.

ModuleNotFoundError: No module named 'numpy.testing.decorators'

Looks like numpy.testing.decorators has been removed in the newer version of numpy.

leopsidom · 2021-11-13T06:36:06Z

I think this will depend on success of this upgrade: #1293

hoodmane · 2021-11-13T07:06:24Z

Well it's still available as
from numpy.testing._private.decorators import setastest
Maybe we can patch numpy to copy numpy.testing._private.decorators to numpy.testing.decorators.

leopsidom · 2021-11-13T07:22:22Z

Ahh, good point. Let me try that.

rth

Thanks @leopsidom, great I see the CI passes!

A few minor comments below otherwise looks good. Could you also please merge upstream/main to sync? I merged #1708 recently which adds a few more patches to numpy, but you should be able to remove the fix-random-double-fill.patch with this update.

rth · 2021-11-13T11:35:23Z

pyodide-build/pyodide_build/pywasmcross.py

@@ -329,6 +329,8 @@ def handle_command(line, args, dryrun=False):
            continue
        if arg == "-lffi":
            continue
+        if arg == "-mpopcnt":


Maybe let's add a comment that this is a SSE 4.2 instruction, and we currently build without SIMD.

Just so it's easier to know what to re-enable once we want to build with SIMD.

Added a comment -

packages/numpy/patches/add-back-decorators-module.patch

… argument

leopsidom · 2021-11-13T17:26:11Z

It seems the fix-getters.patch is also not needed: https://github.com/numpy/numpy/blob/main/numpy/core/src/multiarray/getset.c#L29 ?

rth · 2021-11-13T17:28:10Z

It seems the fix-getters.patch is also not needed

It's possible, I wasn't 100% sure in what release it got included.

rth · 2021-11-13T17:39:20Z

No actually my initial comment was correct. That patch is not included in 1.21.4, so it's still necessary.

leopsidom · 2021-11-13T17:43:46Z

That patch is not included in 1.21.4, so it's still necessary.

Yeah, I was looking at a newer commit @1.22.0.dev0

leopsidom · 2021-11-13T18:12:27Z

Also I'm seeing two patches not applied in meta.yaml: 0008-fix-dict.patch and make-int-return-values.patch, wonder why we included them in the patches ?

hoodmane · 2021-11-13T18:14:58Z

Also I'm seeing two patches not applied in meta.yaml: 0008-fix-dict.patch and make-int-return-values.patch, wonder why we included them in the patches ?

Maybe they aren't needed any more, but we just forgot to delete them?

leopsidom · 2021-11-13T18:18:09Z

Okay, makes sense. Let me just remove them -

leopsidom · 2021-11-13T18:21:13Z

Oh wait I think they are correct. The first one is Cython patch not a numpy patch. The 2nd does get included in meta.yaml.

rth · 2021-11-14T14:03:45Z

Thanks @leopsidom !

hoodmane · 2021-11-14T16:17:28Z

Thanks @leopsidom! This is very helpful. Though I think we should have a changelog entry for this?

leopsidom · 2021-11-14T18:12:43Z

No problems. Thanks for accepting the PR @rth. @hoodmane sure, I can take a look and add a changelog for it a little later.

leopsidom · 2021-12-20T01:40:02Z

Thanks @leopsidom! This is very helpful. Though I think we should have a changelog entry for this?

Opened a PR to update the change log.

leopsidom mentioned this pull request Nov 7, 2021

Upgrade numpy version to 1.20+ #1827

Closed

leopsidom commented Nov 7, 2021

View reviewed changes

leopsidom closed this Nov 13, 2021

leopsidom reopened this Nov 13, 2021

rth reviewed Nov 13, 2021

View reviewed changes

Ubuntu and others added 7 commits November 13, 2021 16:58

Upgrade numpy to 1.21.4

1ce6f3b

Remove unused patches

121a76d

Add description to new patches; Change cpu-baseline optimization to min

8859ea0

Disable SIMD optimization

5c4fc97

Update NUMPY_LIB

f10ec57

Add the removed decorator module to be compatible with old scipy

ac9a4cf

import decorators from _private instead of copying; add comment to an…

c3f2b38

… argument

fix fix-getters.patch

65dc1a6

rth merged commit 4662024 into pyodide:main Nov 14, 2021

leopsidom mentioned this pull request Dec 20, 2021

DOC Update changelog for numpy upgrade #2054

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade numpy to 1.21.4 #1934

Upgrade numpy to 1.21.4 #1934

leopsidom commented Nov 7, 2021 •

edited

leopsidom Nov 7, 2021

ryanking13 commented Nov 7, 2021

rth commented Nov 7, 2021 •

edited

leopsidom commented Nov 7, 2021

leopsidom commented Nov 7, 2021

hoodmane commented Nov 10, 2021

leopsidom commented Nov 11, 2021

leopsidom commented Nov 13, 2021 •

edited

leopsidom commented Nov 13, 2021

hoodmane commented Nov 13, 2021

leopsidom commented Nov 13, 2021

rth left a comment

rth Nov 13, 2021

leopsidom Nov 13, 2021 •

edited

leopsidom commented Nov 13, 2021

rth commented Nov 13, 2021

rth commented Nov 13, 2021

leopsidom commented Nov 13, 2021

leopsidom commented Nov 13, 2021

hoodmane commented Nov 13, 2021

leopsidom commented Nov 13, 2021

leopsidom commented Nov 13, 2021

rth commented Nov 14, 2021

hoodmane commented Nov 14, 2021

leopsidom commented Nov 14, 2021

leopsidom commented Dec 20, 2021

		@@ -0,0 +1,13 @@
		diff --git a/numpy/distutils/command/build.py b/numpy/distutils/command/build.py

Upgrade numpy to 1.21.4 #1934

Upgrade numpy to 1.21.4 #1934

Conversation

leopsidom commented Nov 7, 2021 • edited

Description

Checklists

leopsidom Nov 7, 2021

Choose a reason for hiding this comment

ryanking13 commented Nov 7, 2021

rth commented Nov 7, 2021 • edited

leopsidom commented Nov 7, 2021

leopsidom commented Nov 7, 2021

hoodmane commented Nov 10, 2021

leopsidom commented Nov 11, 2021

leopsidom commented Nov 13, 2021 • edited

leopsidom commented Nov 13, 2021

hoodmane commented Nov 13, 2021

leopsidom commented Nov 13, 2021

rth left a comment

Choose a reason for hiding this comment

rth Nov 13, 2021

Choose a reason for hiding this comment

leopsidom Nov 13, 2021 • edited

Choose a reason for hiding this comment

leopsidom commented Nov 13, 2021

rth commented Nov 13, 2021

rth commented Nov 13, 2021

leopsidom commented Nov 13, 2021

leopsidom commented Nov 13, 2021

hoodmane commented Nov 13, 2021

leopsidom commented Nov 13, 2021

leopsidom commented Nov 13, 2021

rth commented Nov 14, 2021

hoodmane commented Nov 14, 2021

leopsidom commented Nov 14, 2021

leopsidom commented Dec 20, 2021

leopsidom commented Nov 7, 2021 •

edited

rth commented Nov 7, 2021 •

edited

leopsidom commented Nov 13, 2021 •

edited

leopsidom Nov 13, 2021 •

edited