Buggy Accelerate Backend when using numpy 1.19

[billlyzhaoyh]

Numpy 1.19 cannot import due to run time error in MacOS 10.14.6. Polyfit sanity test emitted a warning, most likely due to using a buggy Accelerate backend. If you compiled yourself, see site.cfg.example for information. Otherwise report this to the vendor that provided NumPy.

Reproducing code example:

import numpy as np

Error message:

RuntimeError: Polyfit sanity test emitted a warning, most likely due to using a buggy Accelerate backend. If you compiled yourself, see site.cfg.example for information. Otherwise report this to the vendor that provided NumPy.
RankWarning: Polyfit may be poorly conditioned

Traceback (most recent call last):
File "/Users/billyzhaoyh/anaconda/envs/simulation/lib/python3.7/runpy.py", line 183, in _run_module_as_main
mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
File "/Users/billyzhaoyh/anaconda/envs/simulation/lib/python3.7/runpy.py", line 109, in _get_module_details
import(pkg_name)
File "/Users/billyzhaoyh/Desktop/Simulation/reina-model/calc/init.py", line 1, in
from .utils import calcfunc
File "/Users/billyzhaoyh/Desktop/Simulation/reina-model/calc/utils.py", line 9, in
from utils.quilt import load_datasets
File "/Users/billyzhaoyh/Desktop/Simulation/reina-model/utils/quilt.py", line 4, in
import quilt
File "/Users/billyzhaoyh/anaconda/envs/simulation/lib/python3.7/site-packages/quilt/init.py", line 87, in
from .tools.command import (
File "/Users/billyzhaoyh/anaconda/envs/simulation/lib/python3.7/site-packages/quilt/tools/command.py", line 24, in
import numpy as np
File "/Users/billyzhaoyh/Desktop/Simulation/reina-model/src/numpy/numpy/init.py", line 286, in
raise RuntimeError(msg)
RuntimeError: Polyfit sanity test emitted a warning, most likely due to using a buggy Accelerate backend. If you compiled yourself, see site.cfg.example for information. Otherwise report this to the vendor that provided NumPy.
RankWarning: Polyfit may be poorly conditioned

Numpy/Python version information:

Python 3.7.6
Numpy -e git+https://github.com/numpy/numpy.git@078ac01a85c4db46e7f148829c2e0d0e0f30c36f#egg=numpy

[mattip]

This is by design. Did you read the error message? Is it unclear?

[billlyzhaoyh]

so how can I fix this error, by updating the Accelerate backend? I had a search and don't think I can update that for my MacOS.

[vrakesh]

@billlyzhaoyh Requesting to use OpenBLAS while building on MacOS instead. Use site.cfg in the root repo directory to build using the same.

[mattip]

Closing. We strongly recommend not using Accelerate.

[xbeta]

Closing. We strongly recommend not using Accelerate.

Sure, how do we not use it? I got the same error when just run simple import numpy in my macOS

[mattip]

If you are building your own NumPy, get OpenBlas. You can use python3 tools/openblas_support.py to download the one used to build the official wheel. If you are using a provider's NumPy, ask them why they are using Accelerate.

[xbeta]

If you are building your own NumPy, get OpenBlas. You can use python3 tools/openblas_support.py to download the one used to build the official wheel. If you are using a provider's NumPy, ask them why they are using Accelerate.

I am definitely not building on my own. I am using macOS, and run pip3 install -U numpy , so I am not sure what "provider" do you mean by that .

[mattip]

Pip can build from source behind your back. If it takes more than a few seconds to install, it may be compiling. To force pip to only use binary wheels and never compile, use

pip3 install --upgrade --only-binary :all: <package>

Can you try to uninstall your numpy and install with the --only-binary :all: argument?

[slw07g]

This fixed the problem for me. (I replaced pip with pip3 ... and the numpy version as of today will be 1.19.1)
https://gist.github.com/yatsu/47bdde35e8abbe7d14bbe730342aa9e0#file-numpy-openblas-macos-pip-sh

EDIT: Pasting complete solution in case gist gets deleted :)

# Setup HomeBrew if not already installed: https://brew.sh/
brew install openblas
mkdir /tmp/numpy_local
cd /tmp/numpy_local
pip3 download --no-binary :all: --no-deps numpy
unzip numpy-*.zip  # (assuming there's only one version in this folder)
cd numpy-1.19.1 # the version may be a later version than this

cat > site.cfg <<EOF
[openblas]
libraries = openblas
library_dirs = $(brew --prefix openblas)/lib
include_dirs = $(brew --prefix openblas)/include
runtime_library_dirs = $(brew --prefix openblas)/lib
EOF

pip3 install .
# cleanup
cd /tmp
rm -rf numpy_local

[glyph]

A slightly simpler solution:

$ rm -v ~/Library/Caches/pip/wheels/////numpy # clear the pip wheel cache of any built numpy wheels

Per https://twitter.com/pradyunsg/status/1317081239526936576?s=20 this can now be done with a built-in pip command:

$ pip cache remove numpy
$ brew install openblas # make sure OpenBLAS is installed
$ # activate your pypy3 virtualenv
$ OPENBLAS="$(brew --prefix openblas)" pip install numpy # let numpy's setup.py know where OpenBLAS is installed

This has the added benefit that the wheel will not need to be rebuilt, and any future pip install numpy's on pypy3 will work.

[glyph]

Workarounds aside though, why is the default build behavior to produce a broken wheel that can't even import? It seems like it being "by design" to build such a broken artifact is a bug either way; a self-test at the end of the build that aborts if it finds itself in this state would be a welcome addition. (Especially if it could provide a helpful error message like "try brew install openblas" or somesuch.)

[mattip]

Trying to import after the build sounds like a good idea.

[xrisk]

Why not simply refuse to build on macOS if openblas is not found?

[mattip]

There are various valid blas backends, besides OpenBLAS. The only invalid one is Accelerate. Unfortunately, macOS does not provide a way to tell us that they are providing Accelerate, we can only detect it at runtime by triggering a known bad calculation. If you can figure out how to detect Accelerate earlier that would be great. The best we had so far is to detect a symlink, but that is very fragile.

[mikkelam]

If any python-poetry users come across this thread, I would just like to add that @glyph's solution works as well if you just replace the last line with OPENBLAS="$(brew --prefix openblas)" poetry add numpy

[stephenyang21]

I've tried a previous version of numpy and it worked for me
pip3 install numpy==1.18.0

Hope this help

[SteveG34]

uninstall and install 1.18.0 -- worked for me! kuddos to the above ^^

[fny]

I'm here to report that the suggestion to use openblas doesn't work. This crashes:

OPENBLAS="$(brew --prefix openblas)" pip install numpy
python -c 'import numpy'

# =>  RuntimeError: Polyfit sanity test emitted a warning, most likely due to using a buggy Accelerate backend. If you compiled yourself, see site.cfg.example for information. Otherwise report this to the vendor that provided NumPy.

while this works without any issues:

pip install numpy==1.18.0
python -c 'import numpy'

[xrisk]

@fny check the output to see if pip is using a cached wheel. You might have to clear the cache in that case.

[glyph]

@fny see #15947 (comment) for updated instructions

[poke1024]

Same here. Strange this is that this seems like a random condition. On some interpreter starts it works, on some it doesn't. :-(

macOS 10.15.7 
python 3.8.5
numpy 1.20.1

Downgrading to numpy-1.18.0 works.

[xkortex]

Mac Catalina 10.15.7 / Python 3.10.0 / Numpy 1.21.1 fails for me. The same works fine on 3.9.5 and lower.

Running pip install --upgrade --only-binary :all: numpy fails:

ERROR: Could not find a version that satisfies the requirement numpy (from versions: none)
ERROR: No matching distribution found for numpy

so I presume there are no prebuilt wheels. pip install --upgrade --prefer-binary numpy triggers a build, confirming my suspicion. I have openblas installed (/usr/local/opt/openblas -> ../Cellar/openblas/0.3.17) so I tried

pip uninstall numpy
export OPENBLAS=$(brew --prefix openblas)
pip install --no-cache-dir  numpy # avoid the cached .whl!

and that worked, so many thanks @fny !