Numpy 1.10.2 + MKL = crash (Windows) #6923
Comments
|
Hmm, don't know what is going on here. Intel made some changes to the build, but I don't think those are involved. Python 3.5 is a possible culprit as it is compiled with a newer VS on windows. @cgohlke thoughts? |
|
I think your best bet is to provide a reproducer (what are the "some functions" that produce this?), and either hope that @cgohlke pops up with the answer or else file a bug with Anaconda (since they have the resources to support their MKL builds and we don't). |
|
Sorry for the "some functions" :) - it crashed indirectly though matplotlib. This crashes just the same: |
|
I did a quick check with the latest @cgohlke numpy (1.10.2) python-3.4 (64bit) as well as python-3.5 (64bit) on my laptop: >>> np.show_config()
openblas_lapack_info:
NOT AVAILABLE
lapack_mkl_info:
include_dirs = ['C:/Program Files (x86)/IntelSWTools/compilers_and_libraries/windows/mkl/include']
libraries = ['mkl_lapack95_lp64', 'mkl_blas95_lp64', 'mkl_rt', 'mkl_lapack95_lp64', 'mkl_blas95_lp64', 'mkl_rt']
library_dirs = ['C:/Program Files (x86)/IntelSWTools/compilers_and_libraries/windows/mkl/lib/intel64_win']
define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
lapack_opt_info:
include_dirs = ['C:/Program Files (x86)/IntelSWTools/compilers_and_libraries/windows/mkl/include']
libraries = ['mkl_lapack95_lp64', 'mkl_blas95_lp64', 'mkl_rt', 'mkl_lapack95_lp64', 'mkl_blas95_lp64', 'mkl_rt']
library_dirs = ['C:/Program Files (x86)/IntelSWTools/compilers_and_libraries/windows/mkl/lib/intel64_win']
define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
mkl_info:
include_dirs = ['C:/Program Files (x86)/IntelSWTools/compilers_and_libraries/windows/mkl/include']
libraries = ['mkl_lapack95_lp64', 'mkl_blas95_lp64', 'mkl_rt']
library_dirs = ['C:/Program Files (x86)/IntelSWTools/compilers_and_libraries/windows/mkl/lib/intel64_win']
define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
blas_mkl_info:
include_dirs = ['C:/Program Files (x86)/IntelSWTools/compilers_and_libraries/windows/mkl/include']
libraries = ['mkl_lapack95_lp64', 'mkl_blas95_lp64', 'mkl_rt']
library_dirs = ['C:/Program Files (x86)/IntelSWTools/compilers_and_libraries/windows/mkl/lib/intel64_win']
define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
blas_opt_info:
include_dirs = ['C:/Program Files (x86)/IntelSWTools/compilers_and_libraries/windows/mkl/include']
libraries = ['mkl_lapack95_lp64', 'mkl_blas95_lp64', 'mkl_rt']
library_dirs = ['C:/Program Files (x86)/IntelSWTools/compilers_and_libraries/windows/mkl/lib/intel64_win']
define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]>>> import numpy as np
>>> A = np.arange(2*3).reshape(2,3).astype(np.float32)
>>> B = np.arange(2*3*2731).reshape(2,3,2731).astype(np.int16)
>>> np.dot(A, B)
array([[[ 13655., 13658., 13661., ..., 21839., 21842., 21845.],
[ 38234., 38237., 38240., ..., 46418., 46421., 46424.]],
[[ 38234., 38246., 38258., ..., 70970., 70982., 70994.],
[ 136550., 136562., 136574., ..., 169286., 169298., 169310.]]], dtype=float32)Both works as expected for me. I guess the error |
|
Are there any |
|
@cgohlke, in numpy+MKL you introduced Did you consider the following approach from https://gist.github.com/matham/f6a07f8fb5e403feb440 to change the DLL search order? See also: |
|
@cgohlke These are the DLLs found in my Windows directory: Trying to reset my PATH with |
|
@cgohlke, I propose following change to test with numpy+MKL's def _init_numpy_mkl():
# Numpy+MKL on Windows only
import os
import ctypes
if os.name != 'nt':
return
# disable Intel Fortran default console event handler
env = 'FOR_DISABLE_CONSOLE_CTRL_HANDLER'
if env not in os.environ:
os.environ[env] = '1'
# extend DLL search order with numpy.core
# See https://github.com/numpy/numpy/wiki/windows-dll-notes#python-dlls
# and https://pytools.codeplex.com/workitem/1627
try:
_AddDllDirectory = ctypes.windll.kernel32.AddDllDirectory
_AddDllDirectory.argtypes = [ctypes.c_wchar_p]
# Needed to initialize AddDllDirectory modifications
ctypes.windll.kernel32.SetDefaultDllDirectories(0x1000)
except AttributeError:
_AddDllDirectory = ctypes.windll.kernel32.SetDllDirectoryW
_AddDllDirectory.argtypes = [ctypes.c_wchar_p]
try:
_core = os.path.join(os.path.abspath(os.path.dirname(__file__)), 'core')
_AddDllDirectory(_core)
except Exception:
pass
_init_numpy_mkl()
del _init_numpy_mkl |
Good riddance.
That won't really work on older systems and more importantly it breaks packages that depend on the system's DLL search order. It might be acceptable for distributions like Anaconda or WinPython. |
|
I find it amusing that I initially found the Anyway, I did want to note for others who consider using this, that the line However, because python calls Anyway, hopefully this wasn't too far off topic. |
|
@matham, this code snippet should be credited to zooba - see https://pytools.codeplex.com/workitem/1627. The line log from depency walker during |
|
@carlkl true, but I was referring to the wiki: The fragility I was referring to is that in general there may be some other code executed between the |
|
@cgohlke, is there something wrong with preloading |
|
the following snippet ensures(?) that the MKL DLLs are loaded from numpy.core and not from any windows system or Intel redist folder. The order of loading the DLLs matters. At least it works for Windows 7 python-3.4 bit. def _init_numpy_mkl():
# Numpy+MKL on Windows only
import os
import ctypes
if os.name != 'nt':
return
# disable Intel Fortran default console event handler
env = 'FOR_DISABLE_CONSOLE_CTRL_HANDLER'
if env not in os.environ:
os.environ[env] = '1'
# preload MKL DLLs from numpy.core
try:
_core = os.path.join(os.path.abspath(os.path.dirname(__file__)), 'core')
for _dll in ('mkl_rt', 'libiomp5md', 'mkl_core', 'mkl_intel_thread',
'libmmd', 'libifcoremd', 'libimalloc'):
ctypes.cdll.LoadLibrary(os.path.join(_core,_dll))
except Exception:
pass
_init_numpy_mkl()
del _init_numpy_mkl |
|
I had the same Problem and traced it back to installed |
|
@sebastian-schmidt, can you try out, if explicit loading MKL DLLs from the numpy/core folder works for you, even with installed outdated MKL DLLs in the Windows system folder. See above: #6923 (comment) |
|
@carlkl , where do I have to add the snippet? |
|
The function Here is the version found in def _init_numpy_mkl():
# Numpy+MKL on Windows only
import os
if os.name != 'nt':
return
# disable Intel Fortran default console event handler
env = 'FOR_DISABLE_CONSOLE_CTRL_HANDLER'
if env not in os.environ:
os.environ[env] = '1'
# prepend the path of the Intel runtime DLLs to os.environ['PATH']
try:
path = os.path.join(os.path.abspath(os.path.dirname(__file__)), 'core')
if path not in os.environ.get('PATH', ''):
os.environ['PATH'] = os.pathsep.join((path, os.environ.get('PATH', '')))
except Exception:
pass
_init_numpy_mkl()
del _init_numpy_mklThis function adds the The alternative version of this function #6923 (comment) loads the MKL DLLs from However, I didn't care about frozen numpy in this code snippet. |
|
I changed the |
|
Great to hear! The code in _init_numpy_mkl() is part of @cgohlke's numpy distribution and under his responibility. |
|
@sebastian-schmidt, can you test the version using AddDllDirectory (or SetDllDirectoryW as a fallback): see above: #6923 (comment) |
|
@carlkl , the AddDllDirectory / SetDllDirectoryW version also works. |
|
I, too, traced back the issue to Amplitube. |
|
when i tried : So I switched back to the Anaconda original version and and leave Amplitube desinstalled. |
|
@sebastian-schmidt, at least it works for me. Importing numpy, scipy, matplotlib._png works without problems on my box. However, deinstalling MKL DLLs from the windows system folders seems the best solution for this kind of problems. Time to close this issue? |
|
Hi guys. Just in case someone is still with this problem. I have numpy 1.12 + mkl and had the same issue. I uninstalled Amplitube 3 and now everything its working! |
|
Hi guys, I just installed the latest numpy+mkl 1.13 package from here by @cgohlke . I'm running Anaconda w/ Python 3.6.1 64-bit on Windows 7. After that I'm getting this DLL issue as well, it was working fine for v1.12 from the same site before. To add, actually all code works in a Jupyter notebook.... This only happens in a console. @carlkl I searched in my Anaconda folder for Any other suggestions? Thanks. |
|
when i imported skimage , i met the problem "cannot load the mkl_intel_thread.dll". |
|
I wonder if #6923 (comment) works in that case without the need to remove MKL DLLs from windows/system32. This snippet shows, that it is possible to load seven (but not all) MKL DLLs in the numpy/core folder without the need to pass extra libs to the PATH environment variable or use other tricks. It is important to load these 7 DLLs in the correct sequence. All other required MKL DLLs are then automatically loaded from numpy/core instead of the windows/system32 folder. It is yet unclear if this works in all conceivable szenarios. For example frozen python applications. Also Windows OS versions may differ in the loading behaviour of the DLL. It seems to me advantageous to further investigate this variant. |
Whether using Anacondas Numpy or the one packaged here Numpy crashes with some functions. The error messages are
Intel MKL FATAL ERROR: Cannot load mkl_intel_thread.dll.and
The procedure entry point mkl_aa_fw_get_max_memory could not be located in the dynamic link library C:\Program Files\Python 3.5\lib\site-packages\numpy\core\mkl_intel_thread.dll..Numpy 1.9.2 (tested on Python 3.3.5) works fine (but doesn't have above .dll in its directory, either).
For the time being I've downgraded to Numpy 1.9. I know you don't test against MKL but maybe you've got some hints?
System Info
The text was updated successfully, but these errors were encountered: