4.22 - Failed to load the native TensorFlow runtime.

[Tokusei]

Ok so, to confirm the machine can build TF. I installed it via Anaconda
Windows 10
Python 3.6.9
install tensorflow-gpu=1.12.0 will grab CUDA9.0/cuDNN AND CUDA9.1 as part of the prerequisite libraries.
This works. Builds and trains models in VS.

Using the plugin, It grabs TF1.12.0 and the associated dependencies, But it will not run using the same CUDA/cuDNN configuration (I have tried9.0 with various versions of cuDNN) It gives a missing library error. I have the path variable pointing to cuDNN as explained on the installation guide.

full log:

Traceback (most recent call last): File "C:\Users\Administrator\Desktop\tensorflow-ue4-examples-master\Plugins\UnrealEnginePython\Binaries\Win64\Lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 58, in <module> from tensorflow.python.pywrap_tensorflow_internal import * File "C:\Users\Administrator\Desktop\tensorflow-ue4-examples-master\Plugins\UnrealEnginePython\Binaries\Win64\Lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 28, in <module> _pywrap_tensorflow_internal = swig_import_helper() File "C:\Users\Administrator\Desktop\tensorflow-ue4-examples-master\Plugins\UnrealEnginePython\Binaries\Win64\Lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 24, in swig_import_helper _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description) File "imp.py", line 243, in load_module File "imp.py", line 343, in load_dynamic ImportError: DLL load failed: A dynamic link library (DLL) initialization routine failed. Failed to load the native TensorFlow runtime. See https://www.tensorflow.org/install/errors for some common reasons and solutions. Include the entire stack trace above this error message when asking for help. Traceback (most recent call last): File "C:\Users\Administrator\Desktop\tensorflow-ue4-examples-master\Plugins\UnrealEnginePython\Binaries\Win64\Lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 58, in <module> from tensorflow.python.pywrap_tensorflow_internal import * File "C:\Users\Administrator\Desktop\tensorflow-ue4-examples-master\Plugins\UnrealEnginePython\Binaries\Win64\Lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 28, in <module> _pywrap_tensorflow_internal = swig_import_helper() File "C:\Users\Administrator\Desktop\tensorflow-ue4-examples-master\Plugins\UnrealEnginePython\Binaries\Win64\Lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 24, in swig_import_helper _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description) File "imp.py", line 243, in load_module File "imp.py", line 343, in load_dynamic ImportError: DLL load failed: A dynamic link library (DLL) initialization routine failed. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "C:\Users\Administrator\Desktop\tensorflow-ue4-examples-master\Plugins\tensorflow-ue4\Content\Scripts\TensorFlowComponent.py", line 2, in <module> import tensorflow as tf File "C:\Users\Administrator\Desktop\tensorflow-ue4-examples-master\Plugins\UnrealEnginePython\Binaries\Win64\Lib\site-packages\tensorflow\__init__.py", line 24, in <module> from tensorflow.python import pywrap_tensorflow # pylint: disable=unused-import File "C:\Users\Administrator\Desktop\tensorflow-ue4-examples-master\Plugins\UnrealEnginePython\Binaries\Win64\Lib\site-packages\tensorflow\python\__init__.py", line 49, in <module> from tensorflow.python import pywrap_tensorflow File "C:\Users\Administrator\Desktop\tensorflow-ue4-examples-master\Plugins\UnrealEnginePython\Binaries\Win64\Lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 74, in <module> raise ImportError(msg) ImportError: Traceback (most recent call last): File "C:\Users\Administrator\Desktop\tensorflow-ue4-examples-master\Plugins\UnrealEnginePython\Binaries\Win64\Lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 58, in <module> from tensorflow.python.pywrap_tensorflow_internal import * File "C:\Users\Administrator\Desktop\tensorflow-ue4-examples-master\Plugins\UnrealEnginePython\Binaries\Win64\Lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 28, in <module> _pywrap_tensorflow_internal = swig_import_helper() File "C:\Users\Administrator\Desktop\tensorflow-ue4-examples-master\Plugins\UnrealEnginePython\Binaries\Win64\Lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 24, in swig_import_helper _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description) File "imp.py", line 243, in load_module File "imp.py", line 343, in load_dynamic ImportError: DLL load failed: A dynamic link library (DLL) initialization routine failed. Failed to load the native TensorFlow runtime. See https://www.tensorflow.org/install/errors for some common reasons and solutions. Include the entire stack trace above this error message when asking for help.

[getnamo]

Have you tried the cpu build first and can confirm it works? This would eliminate one potential problem area.

Couple of thoughts:

• It does appear you have a short enough path, so it isn't that.
• Keep in mind that this is setup to use an embedded python build and should require not external python installations.
• It is possible some system environment variables may be picking the wrong cuda/cudnn if you have multiple installed. Potentially re-order them so it picks the 9.0 version.

[Tokusei]

Hi, Yep, I also tried the CPU build and it does not work. It gives the same error. I understand it has an embedded python setup, I was using an external one to make sure the machine was capable of running 1.12.0.

Seems like it is failing to find the CUDA/cuDNN installation on the machine (The Anaconda install downloads and refs local nvidia binaries for tensorflow, and does not modify the PATH variables. (It downloads 9.0,9.1 and cuDNN7.6))

Explicitly which versions of CUDA and cuDNN should be used? As 9.0 has 4 patches, and cuDNN has multiple versions too. I have installed a new OS without Python and will try the CPU version again.

[Tokusei]

did the following on a fresh install (W10 :18362.418)

• Installed VS/UE4.22
• Grab the example 4.22 project: https://github.com/getnamo/tensorflow-ue4-examples/releases/tag/0.6.0
• Added the CPU only plugin: https://github.com/getnamo/tensorflow-ue4/releases/tag/0.13.0
• Launched the project, Waited for PIP to download dependencies.

Output log has the following warnings:

ERROR:

LogPython: C:\Users\Administrator\Desktop\TF\Plugins\UnrealEnginePython\Binaries\Win64>echo Getting pip...
Getting pip...
C:\Users\Administrator\Desktop\TF\Plugins\UnrealEnginePython\Binaries\Win64>python get-pip.py
Collecting pip
Downloading https://files.pythonhosted.org/packages/00/b6/9cfa56b4081ad13874b0c6f96af8ce16cfbc1cb06bedf8e9164ce5551ec1/pip-19.3.1-py2.py3-none-any.whl (1.4MB)
Collecting setuptools
Downloading https://files.pythonhosted.org/packages/6a/9a/50fadfd53ec909e4399b67c74cc7f4e883488035cfcdb90b685758fa8b34/setuptools-41.4.0-py2.py3-none-any.whl (580kB)
Collecting wheel
Downloading https://files.pythonhosted.org/packages/00/83/b4a77d044e78ad1a45610eb88f745be2fd2c6d658f9798a15e384b7d57c9/wheel-0.33.6-py2.py3-none-any.whl
Installing collected packages: pip, setuptools, wheel
The script wheel.exe is installed in 'C:\Users\Administrator\Desktop\TF\Plugins\UnrealEnginePython\Binaries\Win64\Scripts' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed pip-19.3.1 setuptools-41.4.0 wheel-0.33.6
C:\Users\Administrator\Desktop\TF\Plugins\UnrealEnginePython\Binaries\Win64>Scripts\pip install -U -I pip
Collecting pip
Using cached https://files.pythonhosted.org/packages/00/b6/9cfa56b4081ad13874b0c6f96af8ce16cfbc1cb06bedf8e9164ce5551ec1/pip-19.3.1-py2.py3-none-any.whl
ERROR: To modify pip, please run the following command:
c:\users\administrator\desktop\tf\plugins\unrealenginepython\binaries\win64\python.exe -m pip install -U -I pip

WARNING:

LogPython: Collecting tensorflow==1.12.0
Downloading https://files.pythonhosted.org/packages/05/cd/c171d2e33c0192b04560ce864c26eba83fed888fe5cd9ded661b2702f2ae/tensorflow-1.12.0-cp36-cp36m-win_amd64.whl (45.9MB)
Collecting keras-preprocessing>=1.0.5
Downloading https://files.pythonhosted.org/packages/28/6a/8c1f62c37212d9fc441a7e26736df51ce6f0e38455816445471f10da4f0a/Keras_Preprocessing-1.1.0-py2.py3-none-any.whl (41kB)
Collecting tensorboard<1.13.0,>=1.12.0
Downloading https://files.pythonhosted.org/packages/07/53/8d32ce9471c18f8d99028b7cef2e5b39ea8765bd7ef250ca05b490880971/tensorboard-1.12.2-py3-none-any.whl (3.0MB)
Collecting astor>=0.6.0
Downloading https://files.pythonhosted.org/packages/d1/4f/950dfae467b384fc96bc6469de25d832534f6b4441033c39f914efd13418/astor-0.8.0-py2.py3-none-any.whl
Collecting numpy>=1.13.3
Downloading https://files.pythonhosted.org/packages/55/7a/f32b39164262765b069b0fe3ec5d4b47580c9c60f7bd3588b58ba8e93a4c/numpy-1.17.3-cp36-cp36m-win_amd64.whl (12.7MB)
Collecting grpcio>=1.8.6
Downloading https://files.pythonhosted.org/packages/4b/75/35bb3a14f671c34ecda9d621b5f363b02011baf67c4c0c6ce6b9e9aa4ddc/grpcio-1.24.1-cp36-cp36m-win_amd64.whl (1.8MB)
Collecting six>=1.10.0
Downloading https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Collecting termcolor>=1.1.0
Downloading https://files.pythonhosted.org/packages/8a/48/a76be51647d0eb9f10e2a4511bf3ffb8cc1e6b14e9e4fab46173aa79f981/termcolor-1.1.0.tar.gz
Requirement already satisfied: wheel>=0.26 in c:\users\administrator\desktop\tf\plugins\unrealenginepython\binaries\win64\lib\site-packages (from tensorflow==1.12.0) (0.33.6)
Collecting keras-applications>=1.0.6
Downloading https://files.pythonhosted.org/packages/71/e3/19762fdfc62877ae9102edf6342d71b28fbfd9dea3d2f96a882ce099b03f/Keras_Applications-1.0.8-py3-none-any.whl (50kB)
Collecting gast>=0.2.0
Downloading https://files.pythonhosted.org/packages/1f/04/4e36c33f8eb5c5b6c622a1f4859352a6acca7ab387257d4b3c191d23ec1d/gast-0.3.2.tar.gz
Collecting protobuf>=3.6.1
Downloading https://files.pythonhosted.org/packages/2d/73/4a14606fa26f186e23015bc974f9010e2bbf1607f372e3bd5e82d2a62f1b/protobuf-3.10.0-cp36-cp36m-win_amd64.whl (1.1MB)
Collecting absl-py>=0.1.6
Downloading https://files.pythonhosted.org/packages/3b/72/e6e483e2db953c11efa44ee21c5fdb6505c4dffa447b4263ca8af6676b62/absl-py-0.8.1.tar.gz (103kB)
Collecting markdown>=2.6.8
Downloading https://files.pythonhosted.org/packages/c0/4e/fd492e91abdc2d2fcb70ef453064d980688762079397f779758e055f6575/Markdown-3.1.1-py2.py3-none-any.whl (87kB)
Collecting werkzeug>=0.11.10
Downloading https://files.pythonhosted.org/packages/ce/42/3aeda98f96e85fd26180534d36570e4d18108d62ae36f87694b476b83d6f/Werkzeug-0.16.0-py2.py3-none-any.whl (327kB)
Collecting h5py
Downloading https://files.pythonhosted.org/packages/0b/fa/bee65d2dbdbd3611702aafd128139c53c90a1285f169ba5467aab252e27a/h5py-2.10.0-cp36-cp36m-win_amd64.whl (2.4MB)
Requirement already satisfied: setuptools in c:\users\administrator\desktop\tf\plugins\unrealenginepython\binaries\win64\lib\site-packages (from protobuf>=3.6.1->tensorflow==1.12.0) (41.4.0)
Building wheels for collected packages: termcolor, gast, absl-py
Building wheel for termcolor (setup.py): started
Building wheel for termcolor (setup.py): finished with status 'done'
Created wheel for termcolor: filename=termcolor-1.1.0-cp36-none-any.whl size=4835 sha256=809d27398657e9c6847ca64f7978946ac24cbbf72c3cbac34a48075b558514a0
Stored in directory: C:\Users\Administrator\AppData\Local\pip\Cache\wheels\7c\06\54\bc84598ba1daf8f970247f550b175aaaee85f68b4b0c5ab2c6
Building wheel for gast (setup.py): started
Building wheel for gast (setup.py): finished with status 'done'
Created wheel for gast: filename=gast-0.3.2-cp36-none-any.whl size=9685 sha256=cfdde2bb4475edec5ce5c41f714a4f79694dbac12dc595e1541d058d7ad7ef0a
Stored in directory: C:\Users\Administrator\AppData\Local\pip\Cache\wheels\59\38\c6\234dc39b4f6951a0768fbc02d5b7207137a5b1d9094f0d54bf
Building wheel for absl-py (setup.py): started
Building wheel for absl-py (setup.py): finished with status 'done'
Created wheel for absl-py: filename=absl_py-0.8.1-cp36-none-any.whl size=121171 sha256=8f65c84279ac07610f4ac9a7f977fb2cf3df60a9aa7bfc8d80bc7764df7c82a5
Stored in directory: C:\Users\Administrator\AppData\Local\pip\Cache\wheels\a7\15\a0\0a0561549ad11cdc1bc8fa1191a353efd30facf6bfb507aefc
Successfully built termcolor gast absl-py
Installing collected packages: numpy, six, keras-preprocessing, grpcio, protobuf, markdown, werkzeug, tensorboard, astor, termcolor, h5py, keras-applications, gast, absl-py, tensorflow
WARNING: The script f2py.exe is installed in 'c:\users\administrator\desktop\tf\plugins\unrealenginepython\binaries\win64\Scripts' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
WARNING: The script markdown_py.exe is installed in 'c:\users\administrator\desktop\tf\plugins\unrealenginepython\binaries\win64\Scripts' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
WARNING: The script tensorboard.exe is installed in 'c:\users\administrator\desktop\tf\plugins\unrealenginepython\binaries\win64\Scripts' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
WARNING: The scripts freeze_graph.exe, saved_model_cli.exe, tensorboard.exe, tflite_convert.exe, toco.exe and toco_from_protos.exe are installed in 'c:\users\administrator\desktop\tf\plugins\unrealenginepython\binaries\win64\Scripts' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed absl-py-0.8.1 astor-0.8.0 gast-0.3.2 grpcio-1.24.1 h5py-2.10.0 keras-applications-1.0.8 keras-preprocessing-1.1.0 markdown-3.1.1 numpy-1.17.3 protobuf-3.10.0 six-1.12.0 tensorboard-1.12.2 tensorflow-1.12.0 termcolor-1.1.0 werkzeug-0.16.0

[Tokusei]

Here is a screengrab of the site-packages directory. Can you confirm that all the dependencies have been downloaded?

Again, When I launch the project is gives a missing library error. If I try to run import tensorflow in the UE Python console, it returns the same error. There may be some misconfiguration with embedded setup, Is it possible to modify the parameters to download a newer version of Tensorflow, Or is the API incompatible with the blueprint interface?

[Tokusei]

Ok, So I have narrowed down the issue a little more. I am running a i7-965x CPU which does not have AVX Support. This explains why it fails to load TF in CPU mode. However, I am able to run it on my GPU in an environment setup in conda (CUDA9.0/cuDNN7.6.0). If I set up the same environment globally, The GPU plugin still fails to load the TF library.

[getnamo]

Hmm interesting. I can confirm the dependencies looks the same as I have when running 1.12.

I would defer to issues like: tensorflow/tensorflow#19584 and maybe try some custom wheels: https://github.com/metral/tensorflow-wheels. Potentially look into your system environment variables to make sure the 9.0 is set to the cudapath (for gpu version).

Also you can downgrade the tensorflow version to pre-avx for cpu (e.g. try 1.5) which can be adjusted here: https://github.com/getnamo/tensorflow-ue4/blob/master/Content/Scripts/upymodule.json

If you do have an environment working however you can disable the embedded feature by changing: https://github.com/getnamo/UnrealEnginePython/blob/master/Source/UnrealEnginePython/UnrealEnginePython.Build.cs#L13 this should lookup your global python environment instead which should run if you know it runs externally.

[Tokusei]

Hi, Thanks for the response. here are the results

AVX-Enabled machine will run the CPU and GPU plugin.

Non-AVX machine: CUDA9.0 patch 4 + cudnn-9.0-windows10-x64-v7

Installing TF1.5.0 CPU by modifying the config will initalise in the basic map and return the addition and subtraction functions. in the Mnist map, it gives some errors because it is calling funcs which dont exist in 1.5.0

Installing 1.12.0 cannot init CPU as expected, It also cannot initalise the GPU version, however it is possible to initalise the GPU version of 1.12.0 on a non-avx macchine