Illegal instruction when importing Keras/Tensorflow with CPU not supporting AVX

[zgm0407]

I run sct_check_dependencies after launch sct_4.0.0 on Win10's docker DeskTop, it seemed something wrong. how can I resolve the problem?

Description

sct@35acfd3d29fa:~/sct_4.0.0/bin$  sct_check_dependencies

--
Spinal Cord Toolbox (4.0.0)

SCT info:
- version: 4.0.0
- path: /home/sct/sct_4.0.0
OS: linux (Linux-4.9.184-linuxkit-x86_64-with-debian-buster-sid)
CPU cores: Available: 2, Used by SCT: 2
RAM: MemTotal:        2047012 kB
total        used        free      shared  buff/cache   available
Mem:           1999         348         167           0        1482        1465
Swap:          1023           2        1021
Check Python executable.............................[OK]
  Using bundled python 3.6.9 |Anaconda, Inc.| (default, Jul 30 2019, 19:07:31)
[GCC 7.3.0] at /home/sct/sct_4.0.0/python/envs/venv_sct/bin/python
Check if data are installed.........................[OK]
Check if numpy is installed.........................[OK] (1.17.0)
Check if cryptography is installed..................[OK] (2.7)
Check if dipy is installed..........................[OK] (0.16.0)
Check if futures is installed.......................[OK]
Check if h5py is installed..........................[OK] (2.9.0)
Check if Keras is installed.........................Illegal instruction
sct@35acfd3d29fa:~/sct_4.0.0/bin$

[jcohenadad]

Hi,
There might be incompatibilities with your CPU. TF >1.5 requires CPU that support AVX. Some info here:

• Illegal instruction (core dumped) after running import tensorflow tensorflow/tensorflow#17411
• Illegal instruction (core dumped) keras-team/keras#9630
• https://tech.amikelive.com/node-887/how-to-resolve-error-illegal-instruction-core-dumped-when-running-import-tensorflow-in-a-python-program/
• Illegal instruction (core dumped) tensorflow/tensorflow#30114

What is your hardware configuration (CPU type)?

Before digging too extensively, could you please try the following:

• Install SCT via virtual machine (instructions here)
• Use another computer (to see if your issue is computer-specific)

[zgm0407]

I tried install SCT via virtual machine, but failed again.
SCT info:

• version: 4.2.0
• path: /home/brain/sct_4.2.0
  OS: linux (Linux-3.16.0-4-amd64-x86_64-with-debian-8.0)
  CPU cores: Available: 1, Used by SCT: 1
  RAM: MemTotal: 4059052 kB
  total used free shared buffers cached
  Mem: 3963 2748 1215 11 76 2350
  -/+ buffers/cache: 320 3643
  Swap: 1951 0 1951
  Check Python executable.............................[OK]
  Using bundled python 3.6.9 |Anaconda, Inc.| (default, Jul 30 2019, 19:07:31)
  [GCC 7.3.0] at /home/brain/sct_4.2.0/python/envs/venv_sct/bin/python
  Check if data are installed.........................[OK]
  Check if numpy is installed.........................[OK] (1.17.4)
  Check if cryptography is installed..................[OK] (2.8)
  Check if dipy is installed..........................[OK] (1.0.0)
  Check if futures is installed.......................[OK]
  Check if h5py is installed..........................[OK] (2.10.0)
  Check if Keras is installed........................../install_sct: line 680: 1647 Illegal instruction sct_check_dependencies

Installation validation Failed!

Please copy the historic of this Terminal (starting with the command install_sct) and paste it in the SCT Help forum (create a new discussion):

http://forum.spinalcordmri.org/c/sct

Installation failed

brain@neurodebian:~/spinalcordtoolbox-4.2.0$

[zgm0407]

I tried install SCT via virtual machine, but failed again.

SCT info:
- version: 4.2.0
- path: /home/brain/sct_4.2.0
OS: linux (Linux-3.16.0-4-amd64-x86_64-with-debian-8.0)
CPU cores: Available: 1, Used by SCT: 1
RAM: MemTotal:        4059052 kB
total       used       free     shared    buffers     cached
Mem:          3963       2748       1215         11         76       2350
-/+ buffers/cache:        320       3643
Swap:         1951          0       1951
Check Python executable.............................[OK]
  Using bundled python 3.6.9 |Anaconda, Inc.| (default, Jul 30 2019, 19:07:31) 
[GCC 7.3.0] at /home/brain/sct_4.2.0/python/envs/venv_sct/bin/python
Check if data are installed.........................[OK]
Check if numpy is installed.........................[OK] (1.17.4)
Check if cryptography is installed..................[OK] (2.8)
Check if dipy is installed..........................[OK] (1.0.0)
Check if futures is installed.......................[OK]
Check if h5py is installed..........................[OK] (2.10.0)
Check if Keras is installed........................../install_sct: line 680:  1647 Illegal instruction     sct_check_dependencies

Installation validation Failed!

Please copy the historic of this Terminal (starting with the command install_sct) and paste it in the SCT Help forum (create a new discussion):

http://forum.spinalcordmri.org/c/sct


Installation failed

brain@neurodebian:~/spinalcordtoolbox-4.2.0$ 

[jcohenadad]

@zgm0407 likely related to this issue.

It is possible that your CPU version does not support AVX, which is used by TF v1.5 and higher, as explained here.

Could you please try to downgrade to TF 1.5 to see if it works? Please run the following in your virtual machine environment (and then copy/paste your entire Terminal output):

source ${SCT_DIR}/python/etc/profile.d/conda.sh
conda activate venv_sct
python -c "import tensorflow as tf; print(tf.__version__)"
yes | pip uninstall tensorflow
pip install tensorflow==1.5
python -c "import tensorflow as tf; print(tf.__version__)"

[jcohenadad]

It would be nice to check if CPU supports AVX before installing SCT.

One way to do it would be to check if the output of the following command is not empty:

lscpu | grep avx
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a rdseed adx smap intel_pt xsaveopt cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm arat pln pts md_clear flush_l1d

But the above command does not work on OSX (and possibly not on all Linux distros). @bcdarwin @cfhammill any insight/idea how to proceed best?

[bcdarwin]

Maybe either depend on https://github.com/workhorsy/py-cpuinfo, which uses no command-line programs which might not be installed or other Python libraries, or read the code there to find out how to get OSX (/Windows) CPU info.

[jcohenadad]

right, the only thing is that ideally, we would do the check before installing Python+SCT, as done here.

On the other hand, doing it in Python would be easier/cleaner. We could add this check under sct_check_dependencies

[bcdarwin]

I guess there are three options:

• check using 'cat /proc/cpuinfo | grep ... ' on Linux and the equivalent on OSX at the beginning of the installer;
• check using py-cpuinfo as soon as conda is available and possibly exit from the installer with an error (a bit messy)
• check in sct_check_dependencies by detecting the Keras segfault and/or detecting AVX with Python

[jcohenadad]

ok, i think we should go for the 3rd one. The pros are:

• no refactoring of the install shell script (less friendly to read than python code)
• no need to deal with platform-dependent CPU-check functions
• users' report is centralized in the sct_check_dependencies (easier for us to assist users if all the info is spitted out from this script)

the only cons i see is:

• adds 1-2 minutes on the user side (has to wait for the install to complete)

[jcohenadad]

@zgm0407,

I'm following up on the issue you had while installing SCT, you experienced an ERROR: Illegal instruction (core dumped) related to the incompatibility with Keras/Tensoflow.

We have solved that issue in a pre-release, and we would like to ask users who have experienced that issue to make sure this solution works in most cases.

Could you please go back to the computer that produced that error, and try the following (copy/paste the Terminal output):

git clone --branch=jca/2562-tensorflow-1.5 --depth=1 https://github.com/neuropoly/spinalcordtoolbox.git sct-tf1.5
cd sct-tf1.5/
yes | ./install_sct

Open a new Terminal and run (send us this terminal output if at least one test fails):

sct_testing

Thanks!

[ms-elphi]

Inside conda virtual env
first installed keras-application using pip
then install tensorflow again.
(downgrade not required)
it worked for me