Numpy WIN 64-bit v1.13.1 |Python 3.6.1 |Anaconda 4.4.0 suddenly broken

[tobigithub]

Numpy WIN 64-bit v1.13.1 just worked fine in Anaconda navigator and Spyder, but since July 2017
and some weird updates, NUMPY under Windows and Python 3.6.1 does not work anymore. (worked for couple of month before, really liked it).

---

Spyder with numpy v1.13.1
Python 3.6.1 |Anaconda 4.4.0 (64-bit)| (default, May 11 2017, 13:25:24) [MSC v.1900 64 bit (AMD64)]

import numpy
ImportError:
Importing the multiarray numpy extension module failed. Most
likely you are trying to import a failed build of numpy.
If you're working with a numpy git repo, try git clean -xdf (removes all
files not under version control). Otherwise reinstall numpy.

---

Another error just for conda Numpy WIN 64-bit v1.13.1.

Python 3.6.2 |Continuum Analytics, Inc.| (default, Jul 20 2017, 12:30:02) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.

import numpy
Traceback (most recent call last):
File "C:\Anaconda3\envs\tensorflow-cpu\lib\site-packages\numpy\core_init_.py", line 16, in
from . import multiarray
ImportError: DLL load failed: The specified procedure could not be found.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "", line 1, in
File "C:\Anaconda3\envs\tensorflow-cpu\lib\site-packages\numpy_init_.py", line 142, in
from . import add_newdocs
File "C:\Anaconda3\envs\tensorflow-cpu\lib\site-packages\numpy\add_newdocs.py", line 13, in
from numpy.lib import add_newdoc
File "C:\Anaconda3\envs\tensorflow-cpu\lib\site-packages\numpy\lib_init_.py", line 8, in
from .type_check import *
File "C:\Anaconda3\envs\tensorflow-cpu\lib\site-packages\numpy\lib\type_check.py", line 11, in
import numpy.core.numeric as nx
File "C:\Anaconda3\envs\tensorflow-cpu\lib\site-packages\numpy\core_init.py", line 26, in
raise ImportError(msg)
ImportError:
Importing the multiarray numpy extension module failed. Most
likely you are trying to import a failed build of numpy.
If you're working with a numpy git repo, try git clean -xdf (removes all
files not under version control). Otherwise reinstall numpy.

Original error was: DLL load failed: The specified procedure could not be found.

---

This worked fine with an old install of Anaconda3-4.3.1-Windows-x86_64.exe for a couple of weeks,
a new install after everything was broken based on update/upgrade procedures gives the same errors.

[tobigithub]

Some further investigation (only based on numpy and python 3.6.2 leads to
C:\Anaconda3\envs\tensorflow-cpu\Lib\site-packages\numpy\core\multiarray.cp36-win_amd64.pyd
(compiled python DLL) from 7/7/2017 which loads
C:\Windows\System32\mkl_intel_thread.dll (v2009)
C:\Windows\System32\mkl_core.dll (v2009)
C:\Windows\System32\libiomp5md.dll (v2009)

now the problem is, that there are multiple versions for each of the DLLs (DLL hell)
all different dates, versions, file sizes. Basically the Anaconda Navigator or Spyder or python installed through different environments do not load the correct DLL versions.

---

mkl_intel_thread.dll
C:\Anaconda3\Library\bin\mkl_intel_thread.dll (october 2016)
C:\Anaconda3\pkgs\mkl-2017.0.3-0\Library\bin\mkl_intel_thread.dll (april 2017)
C:\Windows\System32\mkl_intel_thread.dll (v2009)

---

mkl_core.dll
C:\Anaconda3\pkgs\mkl-2017.0.3-0\Library\bin\mkl_core.dll (april 2017)
C:\Anaconda3\Library\bin\mkl_core.dll (october 2016)
C:\Windows\System32\mkl_core.dll (v2009)

---

libiomp5md.dll
C:\Anaconda3\pkgs\mkl-2017.0.3-0\Library\bin\libiomp5md.dll
C:\Anaconda3\Library\bin\libiomp5md.dll
C:\Anaconda3\envs\tensorflow-cpu\Library\bin\libiomp5md.dll
C:\Program Files (x86)\Common Files\Intel\Shared Libraries\redist\ia32\compiler\libiomp5md.dll
C:\Program Files (x86)\Common Files\Intel\Shared Libraries\redist\intel64\compiler\libiomp5md.dll
C:\Windows\System32\libiomp5md.dll

each of them different versions, file sizes and versions. Basically the libraries that are loaded refer
to very old versions ranging from 2009 to 2017. This is an issue with numpy.

[mingwandroid]

This is an issue with your system. You should never have any DLLs in C:\Windows\System32 that are not provided as part of Windows by Microsoft. Non-system DLLs in the system DLL folder is the worst form of DLL hell.

Do you know what software you installed that put those DLLs in that location? Anaconda does not do that.

[tobigithub]

@mingwandroid
thanks, I agree.

But reality is that Microsoft and many other vendors ship these DLLs as part of different packages and they are sometimes installed in System32 and sometimes not. These libraries are around since many years dating back to 1994. Out of the 70,000 DLLs on my small test system around 4000 DLLs are in System32. Who installed what, probably can only be found by a dependency walker. Is it good practice to have all DLLs in local directories, sure it is. It is commonly done, probably not.

Example for MS DLLs shipped (libiomp5md.dll)
https://docs.microsoft.com/en-us/cognitive-toolkit/cntk-library-evaluation-on-windows
https://www.google.com/search?q=libiomp5md.dll+site%3Amicrosoft.com

As I said, this is (probably) an issue with numpy, not only observed by me and I just report it here,
because for two month Anaconda Navigator with different environments was working fine and then it stopped, I am not sure if the Anaconda Navigator automatically updates packages (i doubt). Nevertheless the upgrade of Anaconda Navigator to a later version nuked my formerly working solution (an error I have not posted). Even with a clean re-install of Anaconda Navigator (without the "path" option) numpy does not work anymore.

2017 numpy error under windows:
https://www.google.com/#q=numpy+%22Importing+the+multiarray+numpy+extension+module+failed.+Most%22

The numpy package owners recommended to post that error here, I will see if this can get resolved.
numpy/numpy#9272

[tobigithub]

Its a numpy/conda/python issue under windows. Both examples are executed at the same time using the same Anaconda Navigator (conda) version. Installing different python versions and then testing all different dependencies and version numbers can help find the issue. After that code changes have to be investigated.

Working
Python 3.5.3 and numpy 1.10.1 and mkl 2017.0.3

Python 3.5.3 |Continuum Analytics, Inc.| (default, May 15 2017, 10:43:23) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> print (sys.version)
3.5.3 |Continuum Analytics, Inc.| (default, May 15 2017, 10:43:23) [MSC v.1900 64 bit (AMD64)]
>>>
>>> import numpy
>>> numpy.version.version
'1.10.1'
>>>
>>> import numpy as np
>>> a = np.array([1, 2, 3, 4, 5])
>>> print(a)
[1 2 3 4 5]
>>>

Broken
Python 3.5.3 and numpy 1.13.1 and mkl 2017.0.3

Python 3.5.3 |Continuum Analytics, Inc.| (default, May 15 2017, 10:43:23) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> print (sys.version)
3.5.3 |Continuum Analytics, Inc.| (default, May 15 2017, 10:43:23) [MSC v.1900 64 bit (AMD64)]
>>>
>>> import numpy
Traceback (most recent call last):
  File "C:\Anaconda3\envs\python35-numpy-test\lib\site-packages\numpy\core\__init__.py", line 16, in <module>
    from . import multiarray
ImportError: DLL load failed: The specified procedure could not be found.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Anaconda3\envs\python35-numpy-test\lib\site-packages\numpy\__init__.py", line 142, in <module>
    from . import add_newdocs
  File "C:\Anaconda3\envs\python35-numpy-test\lib\site-packages\numpy\add_newdocs.py", line 13, in <module>
    from numpy.lib import add_newdoc
  File "C:\Anaconda3\envs\python35-numpy-test\lib\site-packages\numpy\lib\__init__.py", line 8, in <module>
    from .type_check import *
  File "C:\Anaconda3\envs\python35-numpy-test\lib\site-packages\numpy\lib\type_check.py", line 11, in <module>
    import numpy.core.numeric as _nx
  File "C:\Anaconda3\envs\python35-numpy-test\lib\site-packages\numpy\core\__init__.py", line 26, in <module>
    raise ImportError(msg)
ImportError:
Importing the multiarray numpy extension module failed.  Most
likely you are trying to import a failed build of numpy.
If you're working with a numpy git repo, try `git clean -xdf` (removes all
files not under version control).  Otherwise reinstall numpy.

Original error was: DLL load failed: The specified procedure could not be found.

[mingwandroid]

But reality is that Microsoft and many other vendors ship these DLLs as part of different packages and they are sometimes installed in System32 and sometimes not.

I am afraid you are mistaken here with regards to Microsoft. I have seen other software vendors doing it, but as a tip, anyone who does this cares not a jot for whether their software breaks the rest of the software on your system and that software should not be used. Please read https://docs.microsoft.com/en-gb/cpp/build/search-path-used-by-windows-to-locate-a-dll

The tl;dr is: Any DLLs in System32 are found in preference to those on PATH, so non-system DLLs in that folder is, as I said, the worst form of DLL hell. There is literally nothing a software distributor such as ourselves can do in the face of the same named DLL in System32.

Example for MS DLLs shipped (libiomp5md.dll)

'Shipped' does not mean 'installed into C:\Windows\System32'. MS knows better than that.

[mingwandroid]

For your reference, here are the instructions for installing CNTK: https://docs.microsoft.com/en-us/cognitive-toolkit/Setup-Windows-Binary-Manual

[tobigithub]

The solution is indeed related to some DLL depencency issues under WIN with Intel MKL libraries
the numpy import error was under python 3.6.2 and numpy 1.13.1 :

"import numpy"
numpy\core_init_.py", line 16, in
from . import multiarray
ImportError: DLL load failed: The specified procedure could not be found.

The quick and dirty solution is to copy all Intel mkl*.dll libraries from the specific environments (env) subfolder (seen at stackoverflow)

<\Library\bin> to
<\Lib\site-packages\numpy\core>

04/13/2017  12:50 AM        30,277,392 mkl_avx.dll
04/13/2017  12:50 AM        38,253,840 mkl_avx2.dll
04/13/2017  12:50 AM        42,234,640 mkl_avx512.dll
04/13/2017  12:50 AM        45,722,896 mkl_avx512_mic.dll
04/13/2017  12:50 AM        25,369,360 mkl_core.dll
04/13/2017  12:50 AM        24,541,456 mkl_def.dll
04/13/2017  12:50 AM        24,506,640 mkl_intel_thread.dll
04/13/2017  12:50 AM        29,744,400 mkl_mc.dll
04/13/2017  12:50 AM        30,594,320 mkl_mc3.dll
04/13/2017  12:50 AM        12,567,824 mkl_rt.dll
04/13/2017  12:50 AM        11,388,688 mkl_sequential.dll
04/13/2017  12:50 AM        12,575,504 mkl_tbb_thread.dll
04/13/2017  12:50 AM        11,250,960 mkl_vml_avx.dll
04/13/2017  12:50 AM        11,463,952 mkl_vml_avx2.dll
04/13/2017  12:50 AM        10,285,840 mkl_vml_avx512.dll
04/13/2017  12:50 AM        11,692,304 mkl_vml_avx512_mic.dll
04/13/2017  12:50 AM         5,224,208 mkl_vml_cmpt.dll
04/13/2017  12:50 AM         5,345,552 mkl_vml_def.dll
04/13/2017  12:50 AM         9,477,904 mkl_vml_mc.dll
04/13/2017  12:50 AM         9,368,336 mkl_vml_mc2.dll
04/13/2017  12:50 AM         9,593,616 mkl_vml_mc3.dll

now numpy can be imported and runs fine.

[mingwandroid]

The reason they're not being read from your PATH is probably because either you have them in System32 or you are not seeing your PATH, best done by using activate.bat.

While I'm glad you are up and running again your fix is not the correct solution and you haven't got to the root of the problem (ie it is likely you'll need to do something similar next time you have an import failure).

[j08lue]

Same problem on Windows64, Miniconda 3 with these versions:

numba                     0.36.2          np114py36h12cb543_0
numpy                     1.14.0          py36_blas_openblas_200  [blas_openblas]  conda-forge
openblas                  0.2.20                   vc14_7  [vc14]  conda-forge

Downgrading numpy to 1.13 (no openblas) helped.

[mingwandroid]

@j08lue, this is probably a conda-forge packaging issue and unlikely related to this MKL issue.

[j08lue]

Right, thanks. conda-forge/numpy-feedstock#78