Problem with MacOSX, conda, and INTEL MKL ERROR

[bt2901]

So, I pre-built some provisional wheel files for Mac and uploaded them on PyPi. That means you can install BigARTM with pip: pip install -i https://test.pypi.org/simple/ bigartm.

However, there could be a problem. This is not confirmed, but it appears the wheels conflict with virtual environments created with conda.

What happens: any attempt of using ARTM library fails with the error similar to

INTEL MKL ERROR: dlopen(/Users/<...>/opt/anaconda3/lib/libmkl_intel_thread.dylib, 9): Library not loaded: @rpath/libiomp5.dylib
Referenced from: /Users/<...>/opt/anaconda3/lib/libmkl_intel_thread.dylib
Reason: image not found.
Intel MKL FATAL ERROR: Cannot load libmkl_intel_thread.dylib.

The explanation

By default, conda installs mkl, which is a BLAS implementation tuned for high performance on Intel CPUs. It seems that this particular BigARTM binary is either incompatible with mkl, or is linked against wrong version, or just fails to find it.

I'm out of my depth here; I need help troubleshooting the problem.

Workarounds

1. The simplest way is just removing mkl:

conda remove mkl mkl-service
conda install nomkl numpy scipy scikit-learn numexpr

This isn't ideal because you could lose some performance gains (Note that conda install nomkl means "Remove mkl and replace by a (slow) standard version")

2. Exporting some environment variables:

In BVLC/caffe#3884 it is suggested to tweak LD_PRELOAD.

From https://software.intel.com/pt-br/forums/intel-fortran-compiler/topic/269694:

I think the trick is to find libiomp5.dylib and then run export DYLD_LIBRARY_PATH=<...>

The somewhat relevant discussion here suggests doing a similar thing with LD_LIBRARY_PATH.

3. The issue could be related to boost (which is pre-compiled and included in the wheels). In that case, it is possible that brew install boost will help (but I cannot verify this).

[bt2901]

Possibly related: #341