BUG: BLAS setup can silently fail and not be reported in show_config()

[larsoner]

Describe the issue:

With 2.0.0dev0 wheel on scientific-python-nightly-wheels there is a big slowdown for @ / dot for us. First noticed a ~1.5x slowdown in MNE-Python CIs, then went through a bunch of stuff with @seberg on Discord (thanks for your patience!). Finally I think I created a minimal example:

$ python3 -m venv ~/python/virtualenvs/npbad
$ source ~/python/virtualenvs/npbad/bin/activate
$ pip install --default-timeout=60 --extra-index-url "https://pypi.anaconda.org/scientific-python-nightly-wheels/simple" "numpy==1.25.0rc1+218.g0e5a362fd"
$ pip list
Package    Version
---------- ------------------------
numpy      1.25.0rc1+218.g0e5a362fd
pip        23.0.1
setuptools 66.1.1
$ which python
/home/larsoner/python/virtualenvs/npbad/bin/python
$ python -m timeit -s "import numpy as np; x = np.random.RandomState(0).randn(300, 10000)" "x @ x.T"
20 loops, best of 5: 18.6 msec per loop
$ pip install --default-timeout=60 --upgrade --extra-index-url "https://pypi.anaconda.org/scientific-python-nightly-wheels/simple" "numpy==2.0.0dev0"
$ python -m timeit -s "import numpy as np; x = np.random.RandomState(0).randn(300, 10000)" "x @ x.T"
1 loop, best of 5: 911 msec per loop

TL;DR: 18.6ms on "1.25.0rc1" from just under a month ago, 911ms on latest 2.0.0dev0 on my machine.

I've tried to reproduce this on main on my machine as well by building myself by setting dispatches, using meson or not, etc. but have only ever managed to get the good/fast time.

Reproduce the code example:

Above

Error message:

N/A

Runtime information:

Above

Context for the issue:

Above

[rgommers]

Thanks for the report @larsoner. That is probably simply expected right now, because SIMD support is incomplete in the Meson build. We can leave this open for a while, but I would recommend not investigating anything because looking at anything performance-related for a Meson vs. a distutils build isn't all that useful right now.

[larsoner]

FWIW locally I built using the Meson and distutils backends and performance was the same. But maybe I didn't configure options properly to see the slowdown...

I also kind of expected this to just get sent over to OpenBLAS so didn't think the SIMD / dispatch stuff would matter. So I thought perhaps an OpenBLAS version changed between a month ago and now and that had performance problems.

[rgommers]

TL;DR: 18.6ms on "1.25.0rc1" from just under a month ago, 911ms on latest 2.0.0dev0 on my machine.

Oh wait, I missed that this was ~50x or so. If you rebuilt and it's both that much slower, maybe the build is silently using lapack_lite instead of OpenBLAS? This seems like a lot.

We do have a whole bunch of logic in numpy/core/src/umath/matmul.c.src, so it's not always straightforward to determine that one is really library an optimized BLAS library.

[larsoner]

If you rebuilt and it's both that much slower, maybe the build is silently using lapack_lite instead of OpenBLAS? This seems like a lot.

No it's the opposite rather -- locally I rebuilt using Meson and distutils with my own OpenBLAS that was the same each time (0.3.21 or something?), and it's just as fast as the 1.25.rc wheel from SPNW (no regression): the only problematic case seems to be the 2.0.0dev0 wheel.

Both the 1.25.rc (fast) and 2.0.0dev0 (slow) show that the OpenBLAS lib used is "OpenBLAS 0.3.23 with 1 thread" when I look using threadpoolctl on my machine (locally I limit it to 1 thread). But if something changed about how that was built or what version was actually used (despite being reported the same) in the last ~30 days that's a good candidate!

[rgommers]

But if something changed about how that was built or what version was actually used (despite being reported the same) in the last ~30 days that's a good candidate!

• The OpenBLAS builds we use saw some changes: https://github.com/MacPython/openblas-libs/pulls?q=is%3Apr+is%3Aclosed
• We're also pulling in a different version and there's some wheel build related changes: https://github.com/numpy/numpy/commits/main/tools/openblas_support.py

[mattip]

How are you locally limiting OpenBLAS to one thread?

[mattip]

How are you locally limiting OpenBLAS to one thread?

Nevermind, this reproduces even when not limiting OpenBLAS to one thread

[mattip]

As far as I can tell, both wheels use the same OpenBLAS site-packages/numpy.libs/libopenblas64_p-r0-7a851222.3.23.so

Each build in the scheduled wheel build log has artifacts. Perhaps someone could download them and find the point at which the slowdown started.

[larsoner]

I can do it tomorrow morning EDT unless someone beats me to it (feel free!)

[larsoner]

Okay clearly I must have meant tomorrow July 19th EDT :)

[larsoner]

Actually it only took a couple of minutes to bisect using cp311-manylinux_x86_64:

$ pip install --force-reinstall ~/Desktop/*5367488372/*.whl
...
$ python -m timeit -s "import numpy as np; x = np.random.RandomState(0).randn(300, 10000)" "x @ x.T"
20 loops, best of 5: 18.8 msec per loop
$ pip install --force-reinstall ~/Desktop/*5396768625/*.whl
...
larsoner@bunk:~/python/mne-installers$ python -m timeit -s "import numpy as np; x = np.random.RandomState(0).randn(300, 10000)" "x @ x.T"
1 loop, best of 5: 913 msec per loop

• The good 5367488372
  • commit message "0e2af84 Merge pull request BUG: Ensure __array_ufunc__ works without any kwargs passed #24031 from seberg/array-ufunc-no-kwargs"
  • 0e2af84 from 24 days ago
• The bad 5396768625
  • commit message "85385d6 Merge pull request BLD: use -ftrapping-math with Clang on macOS in Meson build #24060 from rgommers/clang-floatingpoint-flags"
  • 85385d6 from 21 days ago

Which I guess gets us this diff: https://github.com/numpy/numpy/compare/0e2af84..85385d6

[larsoner]

... I am a bit worried about this meson check EDIT: potentially failing in the wheel build:

https://github.com/numpy/numpy/compare/0e2af84..85385d6#diff-d4017f02419a9799bc2d59ae04399d2c45b9855443bb4a5ff7544ff76b040258

It passing on my machine and failing in the wheel build (for some reason!) would be a possible explanation for the slowdown because @ might revert to a non-BLAS version...

[charris]

I stuck a Backport-Candidate tag on this as the problem may have been backported.

[seberg]

Aha, must have been gh-23984.

[seberg]

One guess: The probing code for C-blas is probably missing the suffix, which we use in the wheels. That would explain why local testing doesn't pick it up.

Maybe it would be good if compilation would just fail if some of these blas variables are set but then blas isn't found?

[rgommers]

Maybe it would be good if compilation would just fail if some of these blas variables are set but then blas isn't found?

Yes, I agree. I've thought about it before, but somehow we've never done this. I suspect the reason is that we have/had a whole bunch of users who didn't care about performance and wanted numpy to build from source without much trouble. Making the build fail basically is going to increase our "numpy doesn't build" issue load by a lot I think, but it'll avoid accidentally getting lapack-lite builds.

The probing code for C-blas is probably missing the suffix, which we use in the wheels. That would explain why local testing doesn't pick it up.

That makes sense, I use non-suffixed OpenBLAS or MKL builds typically.

[larsoner]

So I think there are arguably three bugs / places for improvement here:

1. The one that gave rise to not using OpenBLAS even though it's available (something about a suffix / BLD: improve handling of CBLAS, add -Duse-ilp64 build option #23984)
2. The fact that a user can request/specify an accelerated BLAS but end up with a non-accelerated one as raised by @seberg above. One option would be to try to differentiate between users who as @rgommers says "just want it to build in any way" from those who "want it to build in this specific way" like the SPNW builders -- like if there's a setup.cfg or options passed to specify the bindings then it should error. (For me personally for example, I specify a [openblas] section in my site.cfg so it really seems like NumPy should error out if it fails to use it.) And even if "build in this specific way" is detected, there could in principle be an optional flag (disabled by default) to let it succeed even if it can't be built as requested.
3. The reported bindings seem incorrect. Here is show_config on the good/fast version:

   2.0.0.dev0+301.g0e2af8463
   Build Dependencies:
     blas:
       detection method: cmake
       found: true
       name: OpenBLAS
       version: 0.3.23
     lapack:
       detection method: cmake
       found: true
       name: OpenBLAS
       version: 0.3.23
   

   and from threadpoolctl:

   >>> threadpoolctl.threadpool_info()
   [{'user_api': 'blas', 'internal_api': 'openblas', 'prefix': 'libopenblas', 'filepath': '/home/larsoner/python/virtualenvs/base/lib/python3.11/site-packages/numpy.libs/libopenblas64_p-r0-7a851222.3.23.so', 'version': '0.3.23', 'threading_layer': 'pthreads', 'architecture': 'Haswell', 'num_threads': 1}]
   

   and here it is on the bad/slow one:

   2.0.0.dev0+330.g85385d6eb
   Build Dependencies:
     blas:
       detection method: cmake
       found: true
       name: OpenBLAS
       version: 0.3.23
     lapack:
       detection method: cmake
       found: true
       name: OpenBLAS
       version: 0.3.23
   ...
   >>> threadpoolctl.threadpool_info()
   [{'user_api': 'blas', 'internal_api': 'openblas', 'prefix': 'libopenblas', 'filepath': '/home/larsoner/python/virtualenvs/base/lib/python3.11/site-packages/numpy.libs/libopenblas64_p-r0-7a851222.3.23.so', 'version': '0.3.23', 'threading_layer': 'pthreads', 'architecture': 'Haswell', 'num_threads': 1}]
   

   it seems like a bug that it says OpenBLAS is being used in show_config (and secondarily threadpool_info) when it's not!

[rgommers]

2. One option would be to try to differentiate between users who as @rgommers says "just want it to build in any way" from those who "want it to build in this specific way" like the SPNW builders

We can make that differentiation indeed if we want. We've never done it before. There's a bunch of subtleties though. For example, it is possible to build OpenBLAS without LAPACK. In that case what do we do when you only say "give me openblas"?

Ignoring the details for now, I think we could go either way. The main two options are:

1. We default to blas=auto, lapack=auto, and if we find nothing for either component we use lapack_lite
2. We default to blas=auto, lapack=auto, and if we find nothing for either component we error out (letting the user explicitly say "lapack_lite is fine" with some switch)

In both cases, if the user say blas=specific_lib and then specific_lib is not found, we should error out.

Another change I think I already made is that blas and lapack are always separate. IIRC numpy.distutils will completely ignore OpenBLAS if it was not built with LAPACK support.

The one that gave rise to not using OpenBLAS even though it's available (something about a suffix

Let's make our own wheel builds more robust either way by explicitly selecting OpenBLAS so that the build always fails if there's a detection issue.

[seberg]

FWIW, I think gh-24222 should fix the wheels. But if there is a larger fix that isn't complicated that is better!

Erroring out generously seems fine as long the error is clear about how to work-around things.

Not sure weird compiles (mixing no cblas, but have full lapack with blas) is even worth the trouble in practice? (I would even be open to just drop lapack-lite if I was sure it's not a big burden on even niche platforms to get a proper lapack.)

[rgommers]

Not sure weird compiles (mixing no cblas, but have full lapack with blas) is even worth the trouble in practice?

It was until recently, because Arch Linux built OpenBLAS like that. Luckily they finally fixed it, maybe 2 months ago. So you're probably correct here, no longer worth the trouble.

I would even be open to just drop lapack-lite if I was sure it's not a big burden on even niche platforms to get a proper lapack

Me too, that code is more a burden than helpful at this point (e.g., gh-24198 from a few days ago, older ones like gh-20079, and other issues like build warnings with the low-quality generated C code), and the 50x slowdown type things are very rarely wanted.

Two thoughts:

1. It's easy to build at least reference BLAS/LAPACK on niche platforms. The one reason it would not be easy is that it requires a Fortran compiler. I can't think of any platform where gfortran nor a custom compiler runs but that would need numpy. If so, that platform could still build lapack-lite separately.

   • Even that is not entirely true perhaps, BLIS + LAPACKE should work, which is C-only.

2. Pyodide right now uses lapack-lite for NumPy builds. However, it also packages SciPy so it has a proper BLAS and LAPACK available, it's just not used for NumPy in their builds nor our Emscripten CI job. There's a number of issues about this topic on the Pyodide repo, e.g. Using a high performance BLAS pyodide/pyodide#227. Benchmark OpenBLAS and consider it for numpy pyodide/pyodide#3763 is about packaging OpenBLAS. I'll comment there in case they have concerns.

@charris @mattip can you think of other reasons?

[hoodmane]

We should look into this. I would hate for building numpy for Emscripten to become too much like building scipy. On the other hand, we don't want to be holding back the world, if it's a good idea to remove lapacklite for other reasons, we'll just have to deal with it. And numpy is extremely simple compared to scipy so maybe we won't need such complicated hacks.

[mattip]

#24222 was merged, which should close this issue. Or would you like to change the title and keep it open for further discussion and cleanup?

[larsoner]

#24222 was merged, which should close this issue.

It would seem like it based on the issue title, but in my mind there is more that came out of the discussion here.

I suggested above that there were three problems, and I think #24222 only takes care of the first one. The remaining two are points (2) and (3) above, namely 2) someone can say "I want fast BLAS" but get a build with slow BLAS (and @rgommers had some ideas for that), and 3) in the wheel build that was broken numpy.show_config still claimed that OpenBLAS was used when it wasn't, making the slowdown a mystery that took quite some time to debug rather than immediately obvious! (and so far I don't think anyone commented on this part).

So my vote would be to keep this open until those two other things are fixed, or create new issues that outline those two problems.

[seberg]

Just to note, OpenBLAS was used in those wheels, just only for lapack functions (which use blas functions also!). show_config isn't to blame, unless it showed more details (I think there is an issue about that).

[rgommers]

Agreed this is a show_config bug, our requirement is "BLAS with CBLAS", and if that's not met then we say "not found" and try the next solution.

[rgommers]

I'm taking along the fix for show_config in gh-24238.

I would hate for building numpy for Emscripten to become too much like building scipy. On the other hand, we don't want to be holding back the world, if it's a good idea to remove lapacklite for other reasons, we'll just have to deal with it.

Thanks @hoodmane. It won't become as complicated I'm sure. In case we decide we want to move forward with this and it's a problem for Pyodide, we can perhaps just split off lapack-lite as a separate thing; it doesn't really ever change so it can be built in Pyodide once and be used there until OpenBLAS or another proper BLAS library lands.

That said, let's not worry about it now. I think the first step would only be to fail by default, and let users opt in to a slow BLAS if they don't care about performance for linalg functionality. So for Pyodide the only change would be to pass a flag like -Duse-lapacklite.

[hoodmane]

Yeah I think in the medium term it will be better for us to also use better lapack/blas implementations. We'll get there eventually =)

[rgommers]

I experimented a bit with lapack-lite. When you have any BLAS library on your system, it's pretty difficult to tell the setup.py build to ignore it, but it can be done with the NPY_BLAS_ORDER and NPY_LAPACK_ORDER env vars:

$ NPY_BLAS_ORDER='^blas,openblas' NPY_LAPACK_ORDER='^lapack,openblas' python setup.py build_ext -i
INFO:   libraries armpl_lp64_mp not found in ['/home/rgommers/mambaforge/envs/numpy-dev-noblas/lib', '/usr/local/lib', '/usr/lib64', '/usr/lib', '/usr/lib/']
INFO:   NOT AVAILABLE
INFO: 
INFO: blas_mkl_info:
INFO:   libraries mkl_rt not found in ['/home/rgommers/mambaforge/envs/numpy-dev-noblas/lib', '/usr/local/lib', '/usr/lib64', '/usr/lib', '/usr/lib/']
INFO:   NOT AVAILABLE

...

### Warning:  Using unoptimized lapack ###

The build completes, and is not linked against a BLAS or LAPACK library:

$ ldd numpy/linalg/lapack_lite.cpython-39-x86_64-linux-gnu.so 
        linux-vdso.so.1 (0x00007ffd731fd000)
        libc.so.6 => /usr/lib/libc.so.6 (0x00007febdcc1d000)
        /usr/lib64/ld-linux-x86-64.so.2 (0x00007febdcf73000)

numpy imports and basic operations work, even dot/matmul:

>>> a = np.array([[1, 0], [0, 1]])
... b = np.array([[4, 1], [2, 2]])
... np.dot(a, b)
array([[4, 1],
       [2, 2]])
>>> a @ b
array([[4, 1],
       [2, 2]])
>>> a @ b.T
array([[4, 2],
       [1, 2]])
>>> A = np.array([[1,-2j],[2j,5]])
>>> np.linalg.cholesky(A)
array([[1.+0.j, 0.+0.j],
       [0.+2.j, 1.+0.j]])

However, trying to run the test suite fails, because lapack_lite isn't importable:

>>> np.test()
Traceback (most recent call last):
  Cell In[21], line 1
    np.test()
  File ~/code/numpy/numpy/_pytesttester.py:182 in __call__
    from numpy.testing import IS_PYPY
  File ~/code/numpy/numpy/testing/__init__.py:11
    from ._private.utils import *
  File ~/code/numpy/numpy/testing/_private/utils.py:25
    import numpy.linalg.lapack_lite
ImportError: /home/rgommers/code/numpy/numpy/linalg/lapack_lite.cpython-39-x86_64-linux-gnu.so: undefined symbol: dlapy3_

dlapy3 is indeed a LAPACK symbol and defined in f2c_z_lapack.c. It's not used though, and the only place lapack_lite is explicitly imported is in the test suite. However, no one noticing that the test suite isn't runnable when only lapack-lite is available is a decent indication that this config isn't used a lot.

The following diff fixes the test suite, and it passes then:

--- a/numpy/testing/_private/utils.py
+++ b/numpy/testing/_private/utils.py
@@ -22,7 +22,6 @@
 from numpy.core import (
      intp, float32, empty, arange, array_repr, ndarray, isnat, array)
 from numpy import isfinite, isnan, isinf
-import numpy.linalg.lapack_lite
 
 from io import StringIO
 
@@ -54,7 +53,12 @@ class KnownFailureException(Exception):
 IS_PYPY = sys.implementation.name == 'pypy'
 IS_PYSTON = hasattr(sys, "pyston_version_info")
 HAS_REFCOUNT = getattr(sys, 'getrefcount', None) is not None and not IS_PYSTON
-HAS_LAPACK64 = numpy.linalg.lapack_lite._ilp64
+
+try:
+    import numpy.linalg.lapack_lite
+    HAS_LAPACK64 = numpy.linalg.lapack_lite._ilp64
+except ImportError:
+    HAS_LAPACK64 = False
 
 _OLD_PROMOTION = lambda: np._get_promotion_state() == 'legacy'

So I think I'll fix up the test suite like that and implement a switch to opt into lapack-lite in main. That will help guard against regressions like in this issue, and we get to see if anyone actually runs into failing builds in a way that's not easy to resolve.

Of course, there is more to clean up here - if we're not importing numpy.linalg.lapack_lite at all except for one trivial test suite check, why do we build it at all? We may need the C sources, because they go into umath_linalg too - but we're building them twice now.

[rgommers]

Actually, I think I'll just clean it up straight away. I did a GitHub search and I don't find usage in the wild of the numpy.linalg.lapack_lite module. And since it's clearly broken and we're building the sources twice, let's just get rid of one of them. It's a 5 line diff to add the _ilp64 attribute that we need for testing to umath_linalg.cpp.

[rgommers]

Hmm, not entirely true unfortunately - numpy/linalg/tests/test_linalg.py contains a couple of test functions which use lapack_lite. And one of the two is for xerbla, which is a broken test - see #12472 (comment).

[rgommers]

gh-24279 adds an -Dallow-noblas=false build switch.

[rgommers]

gh-24279 adds an -Dallow-noblas=false build switch.

This caught some issues, but is also tripping folks up. The most annoying consequence is when a package has a transitive build-time dependency on numpy. That will start failing, it typically doesn't matter for building wheels, and it's very difficult to work around the problem because there's no good way to pass the -Dallow-noblas=false to a transitive dependency.

We'll be reverting to defaulting to true on the 1.26.x branch, and probably also in main later on.

[larsoner]

there's no good way to pass the -Dallow-noblas=false to a transitive dependency.

Could we make the default "True if env var NUMPY_ALLOW_NOBLAS is set and False otherwise"? Then these wheel builders or whoever can set this env var. Apologies if someone thought of this and discussed it already...

[rgommers]

That is possible, and we may end up doing that in main perhaps. I'm not thrilled about it though, because environment variables are really bad for diagnostics. I got rid of all of them (we had lots of partially documented and untested ones floating around), and right now all of the build options can be found in meson_options.txt, and the build log clearly shows any non-default values that were used. New environment variables bypass all that - it's like programming with globals.

[rgommers]

Update on this: for 1.26.1 we keep the value of -Dallow-noblas as false, in order to get bug reports rather than silent fallbacks to slow code in case anything changed with the updated BLAS/LAPACK detection (see gh-24893). We've had a bunch of complaints about this (see the tracking issue gh-24703), but not so many that it's super urgent to switch the default back.

[rgommers]

I just opened #25063 to revert the default to allow-noblas=true. That should land in 1.26.2 and 2.0.

I decided against the environment variable for now, I really don't like it. The situation after gh-25063 will just go back to how it always was - with the difference that the warning about the slow fallback being used is much more prominent in the build log.