Numpy conda package broken inside tox on Windows, Python 3.7/3.8

[goerz]

The conda-package for numpy does not work on Windows inside a tox environment (using tox-conda):

Steps to reproduce are:

• Set up a tox.ini that uses tox-conda
• Define a tox environment that has numpy in conda_deps, either from the default or the conda-forge channel
• Invoke tox such that numpy is imported inside the environment

The result in an error such as

Traceback (most recent call last):
  File "C:\projects\2020-02-06-numpy-on-appveyor\.tox\envdir\lib\site-packages\numpy\core\__init__.py", line 24, in <module>
    from . import multiarray
  File "C:\projects\2020-02-06-numpy-on-appveyor\.tox\envdir\lib\site-packages\numpy\core\multiarray.py", line 14, in <module>
    from . import overrides
  File "C:\projects\2020-02-06-numpy-on-appveyor\.tox\envdir\lib\site-packages\numpy\core\overrides.py", line 7, in <module>
    from numpy.core._multiarray_umath import (
ImportError: DLL load failed while importing _multiarray_umath: The specified module could not be found.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\projects\2020-02-06-numpy-on-appveyor\.tox\envdir\lib\site-packages\numpy\__init__.py", line 142, in <module>
    from . import core
  File "C:\projects\2020-02-06-numpy-on-appveyor\.tox\envdir\lib\site-packages\numpy\core\__init__.py", line 54, in <module>
    raise ImportError(msg)
ImportError:

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy c-extensions failed.
- Try uninstalling and reinstalling numpy.
- If you have already done that, then:
  1. Check that you expected to use Python3.8 from "C:\projects\2020-02-06-numpy-on-appveyor\.tox\envdir\python.exe",
     and that you have no directories in your PATH or PYTHONPATH that can
     interfere with the Python and numpy version "1.18.1" you're trying to use.
  2. If (1) looks fine, you can open a new issue at
     https://github.com/numpy/numpy/issues.  Please include details on:
     - how you installed Python
     - how you installed numpy
     - your operating system
     - whether or not you have multiple versions of Python installed
     - if you built from source, your compiler versions and ideally a build log

- If you're working with a numpy git repository, try `git clean -xdf`
  (removes all files not under version control) and rebuild numpy.

Note: this error has many possible causes, so please don't comment on
an existing issue about this - open a new one instead.

Original error was: DLL load failed while importing _multiarray_umath: The specified module could not be found.

ERROR: InvocationError for command 'C:\projects\2020-02-06-numpy-on-appveyor\.tox\envdir\python.exe' -c 'import numpy; print(numpy.__version__)' (exited with code 1)

The error occurs for Python versions 3.7 and 3.8, but not 3.6. It also only occurs when installing the numpy package inside of tox (as opposed to just the default conda environment directly)

I've set up a repository at https://github.com/goerz-testing/2020-02-06_numpy_on_appveyor that illustrates several variations of the problem. In that repository, the tox.ini defines a run-cmd environment to execute arbitrary commands inside the environment. In .appveyor.yml I then invoke tox to run

python.exe -c "import numpy; print(numpy.__version__)"

inside the environment.

The repository reproduces the error on AppVeyor, but I've observed the same behavior "manually" on a standard Windows 10 installation.

Variations of things that do and don't work are branches in the repository:

• master branch: Python 3.8, numpy from conda-forge channel ❌ (AppVeyor log)
• 37 branch: Python 3.7, numpy from conda-forge channel ❌ (AppVeyor log)
• defaultchannel branch: Python 3.8, numpy from default channel ❌ (AppVeyor log) → Problem isn't just with the conda-forge numpy package
• systemtox branch: Python 3.8, numpy from conda-forge channel, use system-tox instead of installing tox into conda ❌ (AppVeyor log) → Problem isn't with the tox installed via conda install tox)
• 36 branch: Python 3.6, numpy from conda-forge channe ✅ (AppVeyor log)
• notox branch: Python 3.8, numpy from conda-forge channel, install into system-conda(no tox) ✅ (AppVeyor log)

It's not clear whether numpy or tox-conda or a weird interaction between the two is responsible for the problem.

[goerz]

I further find the same problem if I just install numpy in any conda (non-base) environment, without tox/tox-conda:

• condaenv branch: Python 3.8 in conda env, no tox ❌ (AppVeyor log)

It seemed like this should prove that the problem is not related to tox, but with the new approach, I'm still having a problem if I switch to Python 3.6 (AppVeyor log)

Thus, not being able to install numpy into a conda env may actually be a different bug than the original not being able to install numpy into a conda-based tox environment 🤷‍♂️

Possible duplicates:

• Numpy wont import in Python 3 in anaconda env #15523
• "module not found error" in VS Code using Conda #15343
• Importing the numpy c-extensions failed. #15447

[mattip]

Do you need to activate your conda environment?

[goerz]

Indeed! Using an .appveyor.yml that activates the testenv explicitly works: https://ci.appveyor.com/project/goerz/2020-02-06-numpy-on-appveyor

Thanks!

I had been under the assumption that when invoking Python with its full path (C:\\Miniconda3-x64\\envs\\testenv\\python.exe) it would not be necessary to activate testenv. Alas, it appears that activate does more than just setting the PATH for python.exe, and numpy won't load without activation.

That's a rather non-obvious detail of conda. There should probably be a big bold warning somewhere in conda's documentation about the pitfalls of not activating, and it should probably explain a bit more about what activation actually does. It's also annoying that conda activate (the command the conda documentation recommends) does not work on AppVeyor (since conda init does not work) - but that's AppVeyor's problem.

I think it would also be good to mention this pitfall in the text that numpy emits in the ImportError (the section IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!): It's telling me to check that I'm indeed using C:\Miniconda3-x64\envs\testenv\python.exe, which I was. Maybe it should also say "If you are using a conda environment, you MUST explicitly activate that environment. Simply invoking the environment's python executable is not sufficient."

In any case, this likely moves the responsibility for this "bug" to the tox-conda plugin: it must not be activating the conda envs under its management. Strange that it works with Python 3.6 though.

I'll leave this issue open for a bit until I get some feedback from the tox-conda devs about this (and maybe for numpy to improve the error message).

[mattip]

I think this is new to python3.7, definitely needed on python3.8. If we add such a message, it would only start appearing in numpy 1.19 though. I guess we could check the sys.platform, I wonder if there is a way to test "anaconda and not anaconda_activated"

[isuruf]

not anaconda_activated

os.getenv("CONDA_PREFIX") == sys.prefix

[rossbar]

Closing as resolved. Please check this guide or open a new issue with system information if similar problems are encountered.