Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

core dump when import tensorflow #2129

Closed
wangyongliang opened this issue Apr 27, 2016 · 11 comments
Closed

core dump when import tensorflow #2129

wangyongliang opened this issue Apr 27, 2016 · 11 comments

Comments

@wangyongliang
Copy link

wangyongliang commented Apr 27, 2016 •
edited

GitHub issues are for bugs / installation problems / feature requests.
For general support from the community, see StackOverflow.
To make bugs and feature requests more easy to find and organize, we close issues that are deemed
out of scope for GitHub Issues and point people to StackOverflow.

For bugs or installation issues, please provide the following information.
The more information you provide, the more easily we will be able to offer
help and advice.

Environment info

Operating System: Centos 7

Installed version of CUDA and cuDNN: cuda 7.5 cuDNNv4
(please attach the output of ls -l /path/to/cuda/lib/libcud*):
total 987720
-rw-r--r--. 1 root root 28585480 Aug 16 2015 libcublas_device.a
lrwxrwxrwx. 1 root root 16 Sep 9 2015 libcublas.so -> libcublas.so.7.5
lrwxrwxrwx. 1 root root 19 Sep 9 2015 libcublas.so.7.5 -> libcublas.so.7.5.18
-rwxr-xr-x. 1 root root 23938736 Aug 16 2015 libcublas.so.7.5.18
-rw-r--r--. 1 root root 28220076 Aug 16 2015 libcublas_static.a
-rw-r--r--. 1 root root 322936 Aug 16 2015 libcudadevrt.a
lrwxrwxrwx. 1 root root 16 Sep 9 2015 libcudart.so -> libcudart.so.7.5
lrwxrwxrwx. 1 root root 19 Sep 9 2015 libcudart.so.7.5 -> libcudart.so.7.5.18
-rwxr-xr-x. 1 root root 383336 Aug 16 2015 libcudart.so.7.5.18
-rw-r--r--. 1 root root 720192 Aug 16 2015 libcudart_static.a
-rwxr-xr-x. 1 root root 61453024 Apr 27 15:55 libcudnn.so
-rwxr-xr-x. 1 root root 61453024 Apr 27 15:55 libcudnn.so.4
-rwxr-xr-x. 1 root root 61453024 Apr 27 15:55 libcudnn.so.4.0.7
-rw-r--r--. 1 root root 62025862 Apr 27 15:55 libcudnn_static.a
lrwxrwxrwx. 1 root root 15 Sep 9 2015 libcufft.so -> libcufft.so.7.5
lrwxrwxrwx. 1 root root 18 Sep 9 2015 libcufft.so.7.5 -> libcufft.so.7.5.18
-rwxr-xr-x. 1 root root 111231960 Aug 16 2015 libcufft.so.7.5.18
-rw-r--r--. 1 root root 115104400 Aug 16 2015 libcufft_static.a
lrwxrwxrwx. 1 root root 16 Sep 9 2015 libcufftw.so -> libcufftw.so.7.5
lrwxrwxrwx. 1 root root 19 Sep 9 2015 libcufftw.so.7.5 -> libcufftw.so.7.5.18
-rwxr-xr-x. 1 root root 447664 Aug 16 2015 libcufftw.so.7.5.18
-rw-r--r--. 1 root root 42206 Aug 16 2015 libcufftw_static.a
lrwxrwxrwx. 1 root root 17 Sep 9 2015 libcuinj64.so -> libcuinj64.so.7.5
lrwxrwxrwx. 1 root root 20 Sep 9 2015 libcuinj64.so.7.5 -> libcuinj64.so.7.5.18
-rwxr-xr-x. 1 root root 5751400 Aug 16 2015 libcuinj64.so.7.5.18
-rw-r--r--. 1 root root 1649726 Aug 16 2015 libculibos.a
lrwxrwxrwx. 1 root root 16 Sep 9 2015 libcurand.so -> libcurand.so.7.5
lrwxrwxrwx. 1 root root 19 Sep 9 2015 libcurand.so.7.5 -> libcurand.so.7.5.18
-rwxr-xr-x. 1 root root 51765952 Aug 16 2015 libcurand.so.7.5.18
-rw-r--r--. 1 root root 51992564 Aug 16 2015 libcurand_static.a
lrwxrwxrwx. 1 root root 18 Sep 9 2015 libcusolver.so -> libcusolver.so.7.5
lrwxrwxrwx. 1 root root 21 Sep 9 2015 libcusolver.so.7.5 -> libcusolver.so.7.5.18
-rwxr-xr-x. 1 root root 37034328 Aug 16 2015 libcusolver.so.7.5.18
-rw-r--r--. 1 root root 16613348 Aug 16 2015 libcusolver_static.a
lrwxrwxrwx. 1 root root 18 Sep 9 2015 libcusparse.so -> libcusparse.so.7.5
lrwxrwxrwx. 1 root root 21 Sep 9 2015 libcusparse.so.7.5 -> libcusparse.so.7.5.18
-rwxr-xr-x. 1 root root 36816424 Aug 16 2015 libcusparse.so.7.5.18
-rw-r--r--. 1 root root 44445334 Aug 16 2015 libcusparse_static.a
lrwxrwxrwx. 1 root root 14 Sep 9 2015 libnppc.so -> libnppc.so.7.5
lrwxrwxrwx. 1 root root 17 Sep 9 2015 libnppc.so.7.5 -> libnppc.so.7.5.18
-rwxr-xr-x. 1 root root 427168 Aug 16 2015 libnppc.so.7.5.18
-rw-r--r--. 1 root root 20664 Aug 16 2015 libnppc_static.a
lrwxrwxrwx. 1 root root 14 Sep 9 2015 libnppi.so -> libnppi.so.7.5
lrwxrwxrwx. 1 root root 17 Sep 9 2015 libnppi.so.7.5 -> libnppi.so.7.5.18
-rwxr-xr-x. 1 root root 63516808 Aug 16 2015 libnppi.so.7.5.18
-rw-r--r--. 1 root root 90106200 Aug 16 2015 libnppi_static.a
lrwxrwxrwx. 1 root root 14 Sep 9 2015 libnpps.so -> libnpps.so.7.5
lrwxrwxrwx. 1 root root 17 Sep 9 2015 libnpps.so.7.5 -> libnpps.so.7.5.18
-rwxr-xr-x. 1 root root 6047400 Aug 16 2015 libnpps.so.7.5.18
-rw-r--r--. 1 root root 8647292 Aug 16 2015 libnpps_static.a
lrwxrwxrwx. 1 root root 16 Sep 9 2015 libnvblas.so -> libnvblas.so.7.5
lrwxrwxrwx. 1 root root 19 Sep 9 2015 libnvblas.so.7.5 -> libnvblas.so.7.5.18
-rwxr-xr-x. 1 root root 456112 Aug 16 2015 libnvblas.so.7.5.18
lrwxrwxrwx. 1 root root 24 Sep 9 2015 libnvrtc-builtins.so -> libnvrtc-builtins.so.7.5
lrwxrwxrwx. 1 root root 27 Sep 9 2015 libnvrtc-builtins.so.7.5 -> libnvrtc-builtins.so.7.5.18
-rwxr-xr-x. 1 root root 22408512 Aug 16 2015 libnvrtc-builtins.so.7.5.18
lrwxrwxrwx. 1 root root 15 Sep 9 2015 libnvrtc.so -> libnvrtc.so.7.5
lrwxrwxrwx. 1 root root 18 Sep 9 2015 libnvrtc.so.7.5 -> libnvrtc.so.7.5.17
-rwxr-xr-x. 1 root root 18199288 Aug 16 2015 libnvrtc.so.7.5.17
lrwxrwxrwx. 1 root root 18 Sep 9 2015 libnvToolsExt.so -> libnvToolsExt.so.1
lrwxrwxrwx. 1 root root 22 Sep 9 2015 libnvToolsExt.so.1 -> libnvToolsExt.so.1.0.0
-rwxr-xr-x. 1 root root 37936 Aug 16 2015 libnvToolsExt.so.1.0.0
-rw-r--r--. 1 root root 25840 Aug 16 2015 libOpenCL.so
lrwxrwxrwx. 1 root root 12 Sep 9 2015 libOpenCL.so.1 -> libOpenCL.so
drwxr-xr-x. 2 root root 4096 Sep 9 2015 stubs

If installed from binary pip package, provide:

  1. Which pip package you installed.
  2. The output from python -c "import tensorflow; print(tensorflow.version)".

If installed from sources, provide the commit hash:

Steps to reproduce

  1. write a python script test.py which only has one line: import tensorflow

2, gdb python

3, run test.py

4, core dumped, use bt to get stacktrace

What have you tried?

  1. install from pip
    2, install form source
    3, try cudnn V4, V5

Logs or other output that would be helpful

stacktrace from gdb

rogram received signal SIGSEGV, Segmentation fault.
0x00007fffd50e0220 in PyArray_API () from /usr/lib64/python2.7/site-packages/numpy/core/multiarray.so
(gdb) bt
#0 0x00007fffd50e0220 in PyArray_API () from /usr/lib64/python2.7/site-packages/numpy/core/multiarray.so
#1 0x00007fffc513f3c4 in initspecfun () from /usr/lib64/python2.7/site-packages/scipy/special/specfun.so
#2 0x00007ffff7b09eb9 in PyImport_LoadDynamicModule () from /lib64/libpython2.7.so.1.0
#3 0x00007ffff7b07f91 in import_submodule () from /lib64/libpython2.7.so.1.0
#4 0x00007ffff7b0848f in ensure_fromlist () from /lib64/libpython2.7.so.1.0
#5 0x00007ffff7b08cca in PyImport_ImportModuleLevel () from /lib64/libpython2.7.so.1.0
#6 0x00007ffff7aef3bf in builtin___import_ () from /lib64/libpython2.7.so.1.0
#7 0x00007ffff7a5f073 in PyObject_Call () from /lib64/libpython2.7.so.1.0
#8 0x00007ffff7af0fd7 in PyEval_CallObjectWithKeywords () from /lib64/libpython2.7.so.1.0
#9 0x00007ffff7af2aa3 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#10 0x00007ffff7af718d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#11 0x00007ffff7af7292 in PyEval_EvalCode () from /lib64/libpython2.7.so.1.0
#12 0x00007ffff7b0707c in PyImport_ExecCodeModuleEx () from /lib64/libpython2.7.so.1.0
#13 0x00007ffff7b072f8 in load_source_module () from /lib64/libpython2.7.so.1.0
#14 0x00007ffff7b07f91 in import_submodule () from /lib64/libpython2.7.so.1.0
#15 0x00007ffff7b081dd in load_next () from /lib64/libpython2.7.so.1.0
#16 0x00007ffff7b08bbe in PyImport_ImportModuleLevel () from /lib64/libpython2.7.so.1.0
#17 0x00007ffff7aef3bf in builtin___import__ () from /lib64/libpython2.7.so.1.0
#18 0x00007ffff7a5f073 in PyObject_Call () from /lib64/libpython2.7.so.1.0
#19 0x00007ffff7af0fd7 in PyEval_CallObjectWithKeywords () from /lib64/libpython2.7.so.1.0
#20 0x00007ffff7af2aa3 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#21 0x00007ffff7af718d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#22 0x00007ffff7af7292 in PyEval_EvalCode () from /lib64/libpython2.7.so.1.0
#23 0x00007ffff7b0707c in PyImport_ExecCodeModuleEx () from /lib64/libpython2.7.so.1.0
#24 0x00007ffff7b072f8 in load_source_module () from /lib64/libpython2.7.so.1.0
#25 0x00007ffff7b0878a in load_package () from /lib64/libpython2.7.so.1.0
#26 0x00007ffff7b07f91 in import_submodule () from /lib64/libpython2.7.so.1.0
#27 0x00007ffff7b081dd in load_next () from /lib64/libpython2.7.so.1.0
#28 0x00007ffff7b08bf8 in PyImport_ImportModuleLevel () from /lib64/libpython2.7.so.1.0
#29 0x00007ffff7aef3bf in builtin___import__ () from /lib64/libpython2.7.so.1.0
#30 0x00007ffff7a5f073 in PyObject_Call () from /lib64/libpython2.7.so.1.0
#31 0x00007ffff7af0fd7 in PyEval_CallObjectWithKeywords () from /lib64/libpython2.7.so.1.0
#32 0x00007ffff7af2aa3 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#33 0x00007ffff7af718d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#34 0x00007ffff7af7292 in PyEval_EvalCode () from /lib64/libpython2.7.so.1.0
#35 0x00007ffff7b0707c in PyImport_ExecCodeModuleEx () from /lib64/libpython2.7.so.1.0
#36 0x00007ffff7b072f8 in load_source_module () from /lib64/libpython2.7.so.1.0
#37 0x00007ffff7b07f91 in import_submodule () from /lib64/libpython2.7.so.1.0
#38 0x00007ffff7b081dd in load_next () from /lib64/libpython2.7.so.1.0
#39 0x00007ffff7b08bf8 in PyImport_ImportModuleLevel () from /lib64/libpython2.7.so.1.0
@poxvoculi
Copy link
Contributor

Your stack trace shows a failure in numpy, yet it sounds like your source program is just the one line. What if you import numpy first, ie. place
import numpy
above
import tensorflow
?

@yaroslavvb
Copy link
Contributor

possibly related: #2034

@stephenroller
Copy link
Contributor

stephenroller commented Apr 28, 2016 •
edited

I seem to be having this problem too, or maybe the one in #2034. Tensorflow runs just fine, as long as numpy has already been imported.

I'm using cuda 7.5, cudnn 5.0, numpy 1.11.0, python 2.7.11, and tensorflow HEAD (c38157c). Everything was compiled by me.

Test case

$ python
Python 2.7.11 (default, Apr 25 2016, 20:02:17)
[GCC 4.9.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcurand.so.7.5 locally
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcufft.so.7.5 locally
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcudnn.so.5.0 locally
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcublas.so.7.5 locally
zsh: segmentation fault (core dumped)  python

But it doesn't crash as long as numpy is imported first:

$ python
Python 2.7.11 (default, Apr 25 2016, 20:02:17)
[GCC 4.9.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> import tensorflow
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcurand.so.7.5 locally
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcufft.so.7.5 locally
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcudnn.so.5.0 locally
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcublas.so.7.5 locally
>>>

GDB stacktrace of the segfault.

$ gdb python
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-83.el6)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /work/01813/roller/maverick/packages/python/bin/python...done.
(gdb) r
Starting program: /work/01813/roller/maverick/packages/python/bin/python

[Thread debugging using libthread_db enabled]
Python 2.7.11 (default, Apr 25 2016, 20:02:17)
[GCC 4.9.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>
>>> import tensorflow
warning: File "/opt/apps/gcc/4.9.1/lib64/libstdc++.so.6.0.20-gdb.py" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".
To enable execution of this file add
        add-auto-load-safe-path /opt/apps/gcc/4.9.1/lib64/libstdc++.so.6.0.20-gdb.py
line to your configuration file "/home/01813/roller/.gdbinit".
To completely disable this security protection add
        set auto-load safe-path /
line to your configuration file "/home/01813/roller/.gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
        info "(gdb)Auto-loading safe path"
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcurand.so.7.5 locally
Missing separate debuginfo for /usr/lib64/libcuda.so.1
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcufft.so.7.5 locally
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcudnn.so.5.0 locally
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcublas.so.7.5 locally
Detaching after fork from child process 10971.
Missing separate debuginfo for /work/01813/roller/maverick/packages/python/lib/python2.7/site-packages/scipy/special/../.libs/libgfortran-ed201abd.so.3.0.0

Program received signal SIGSEGV, Segmentation fault.
0x00002aaac9d91a40 in PyArray_API () from /work/01813/roller/maverick/packages/python/lib/python2.7/site-packages/numpy-1.11.0-py2.7-linux-x86_64.egg/numpy/core/multiarray.so
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.166.el6_7.1.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 libuuid-2.17.2-12.18.el6.x86_64 ncurses-libs-5.7-4.20090207.el6.x86_64 openssl-1.0.1e-42.el6_7.4.x86_64 readline-6.0-4.el6.x86_64 zlib-1.2.3-29.el6.x86_64

Edit: I tried following the advice of #1373 and reinstalling everything, but it didn't help.

@wangyongliang
Copy link
Author

@poxvoculi , the program has only one line "import tensorflow"

@wangyongliang
Copy link
Author

@stephenroller , mine is different, I got a core dump even import tensorflow, have you solved this issue?

@wangyongliang
Copy link
Author

it seems work now if I "import numpy" before "import tensorflow".
according to #2034

@stephenroller stephenroller mentioned this issue Apr 29, 2016
Closed
@ushnish
Copy link

ushnish commented Jun 16, 2016

Yes! Incredibly import numpy before tensorflow fixes this issue. What's weird is this issue started happening today, and I've run this same code 50+ times before this started happening.

@mijung-kim
Copy link

I had the same segfault and even I tried import numpy before import tensorflow, it did not work. My environment is ubuntu 16.04, python 3.5, and TF 0.9 in a docker container. Anyone knows?

@arita37
Copy link

arita37 commented Aug 16, 2017 •
edited

Same issue, even I import numpy before tensorflow.
tensorflow 1.2
Issue happening after VM copy on other machine...

I suspect this related to the source compiled version on machine 1 (some Intel CPU).
When copying on different hardware the source compiled is not working anymore.....

@dexhunter
Copy link

Hmm... another interesting thing I met is that I need to import numpy before tensorflow and tensorflow before pytorch

If not, I got segmentation fault (core dumped)

@vinnitu
Copy link

vinnitu commented Mar 16, 2018 •
edited

tensorflow 1.6

(gdb) run test.py
Starting program: /usr/bin/python test.py
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff3d42700 (LWP 4263)]

Thread 1 "python" received signal SIGILL, Illegal instruction.
0x00007fffea257880 in std::pair<std::__detail::_Node_iterator<std::pair<tensorflow::StringPiece const, std::function<bool (tensorflow::Variant*)> >, false, true>, bool> std::_Hashtable<tensorflow::StringPiece, std::pair<tensorflow::StringPiece const, std::function<bool (tensorflow::Variant*)> >, std::allocator<std::pair<tensorflow::StringPiece const, std::function<bool (tensorflow::Variant*)> > >, std::__detail::_Select1st, std::equal_to<tensorflow::StringPiece>, tensorflow::StringPieceHasher, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_M_emplace<std::pair<tensorflow::StringPiece, std::function<bool (tensorflow::Variant*)> > >(std::integral_constant<bool, true>, std::pair<tensorflow::StringPiece, std::function<bool (tensorflow::Variant*)> >&&) () from /usr/local/lib/python2.7/dist-packages/tensorflow/python/../libtensorflow_framework.so

(gdb) bt
#0  0x00007fffea257880 in std::pair<std::__detail::_Node_iterator<std::pair<tensorflow::StringPiece const, std::function<bool (tensorflow::Variant*)> >, false, true>, bool> std::_Hashtable<tensorflow::StringPiece, std::pair<tensorflow::StringPiece const, std::function<bool (tensorflow::Variant*)> >, std::allocator<std::pair<tensorflow::StringPiece const, std::function<bool (tensorflow::Variant*)> > >, std::__detail::_Select1st, std::equal_to<tensorflow::StringPiece>, tensorflow::StringPieceHasher, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_M_emplace<std::pair<tensorflow::StringPiece, std::function<bool (tensorflow::Variant*)> > >(std::integral_constant<bool, true>, std::pair<tensorflow::StringPiece, std::function<bool (tensorflow::Variant*)> >&&) () from /usr/local/lib/python2.7/dist-packages/tensorflow/python/../libtensorflow_framework.so
#1  0x00007fffea259735 in tensorflow::UnaryVariantOpRegistry::RegisterDecodeFn(std::string const&, std::function<bool (tensorflow::Variant*)> const&) ()
   from /usr/local/lib/python2.7/dist-packages/tensorflow/python/../libtensorflow_framework.so
#2  0x00007fffea234a7c in tensorflow::variant_op_registry_fn_registration::UnaryVariantDecodeRegistration<tensorflow::Tensor>::UnaryVariantDecodeRegistration(std::string const&) ()
   from /usr/local/lib/python2.7/dist-packages/tensorflow/python/../libtensorflow_framework.so
#3  0x00007fffea1ae165 in _GLOBAL__sub_I_tensor.cc () from /usr/local/lib/python2.7/dist-packages/tensorflow/python/../libtensorflow_framework.so
#4  0x00007ffff7de76ba in call_init (l=<optimized out>, argc=argc@entry=2, argv=argv@entry=0x7fffffffdbc8, env=env@entry=0xa99dc0) at dl-init.c:72
#5  0x00007ffff7de77cb in call_init (env=0xa99dc0, argv=0x7fffffffdbc8, argc=2, l=<optimized out>) at dl-init.c:30
#6  _dl_init (main_map=main_map@entry=0xedcf10, argc=2, argv=0x7fffffffdbc8, env=0xa99dc0) at dl-init.c:120
#7  0x00007ffff7dec8e2 in dl_open_worker (a=a@entry=0x7fffffffbbc0) at dl-open.c:575
#8  0x00007ffff7de7564 in _dl_catch_error (objname=objname@entry=0x7fffffffbbb0, errstring=errstring@entry=0x7fffffffbbb8, mallocedp=mallocedp@entry=0x7fffffffbbaf, 
    operate=operate@entry=0x7ffff7dec4d0 <dl_open_worker>, args=args@entry=0x7fffffffbbc0) at dl-error.c:187
#9  0x00007ffff7debda9 in _dl_open (file=0x7fffef22ba54 "/usr/local/lib/python2.7/dist-packages/tensorflow/python/_pywrap_tensorflow_internal.so", mode=-2147483646, 
    caller_dlopen=0x51ad19 <_PyImport_GetDynLoadFunc+233>, nsid=-2, argc=<optimized out>, argv=<optimized out>, env=0xa99dc0) at dl-open.c:660
#10 0x00007ffff75ecf09 in dlopen_doit (a=a@entry=0x7fffffffbdf0) at dlopen.c:66
#11 0x00007ffff7de7564 in _dl_catch_error (objname=0xa5ee40, errstring=0xa5ee48, mallocedp=0xa5ee38, operate=0x7ffff75eceb0 <dlopen_doit>, args=0x7fffffffbdf0) at dl-error.c:187
#12 0x00007ffff75ed571 in _dlerror_run (operate=operate@entry=0x7ffff75eceb0 <dlopen_doit>, args=args@entry=0x7fffffffbdf0) at dlerror.c:163
#13 0x00007ffff75ecfa1 in __dlopen (file=<optimized out>, mode=<optimized out>) at dlopen.c:87
#14 0x000000000051ad19 in _PyImport_GetDynLoadFunc ()
#15 0x000000000051a8e4 in _PyImport_LoadDynamicModule ()
#16 0x00000000005b7b1b in ?? ()
#17 0x00000000004bc3fa in PyEval_EvalFrameEx ()
#18 0x00000000004c136f in PyEval_EvalFrameEx ()
#19 0x00000000004b9ab6 in PyEval_EvalCodeEx ()
#20 0x00000000004b97a6 in PyEval_EvalCode ()
#21 0x00000000004b96df in PyImport_ExecCodeModuleEx ()
#22 0x00000000004b2b06 in ?? ()
#23 0x00000000004a4ae1 in ?? ()
#24 0x00000000004a4513 in PyImport_ImportModuleLevel ()
#25 0x00000000004a59e4 in ?? ()
#26 0x00000000004a577e in PyObject_Call ()
#27 0x00000000004c5e10 in PyEval_CallObjectWithKeywords ()
#28 0x00000000004be6d7 in PyEval_EvalFrameEx ()
#29 0x00000000004b9ab6 in PyEval_EvalCodeEx ()
#30 0x00000000004b97a6 in PyEval_EvalCode ()
#31 0x00000000004b96df in PyImport_ExecCodeModuleEx ()
#32 0x00000000004b2b06 in ?? ()
#33 0x00000000004a4ae1 in ?? ()
#34 0x00000000004a7fd5 in ?? ()
#35 0x00000000004a3f38 in PyImport_ImportModuleLevel ()
#36 0x00000000004a59e4 in ?? ()
#37 0x00000000004a577e in PyObject_Call ()
#38 0x00000000004c5e10 in PyEval_CallObjectWithKeywords ()
#39 0x00000000004be6d7 in PyEval_EvalFrameEx ()
#40 0x00000000004b9ab6 in PyEval_EvalCodeEx ()
#41 0x00000000004b97a6 in PyEval_EvalCode ()
#42 0x00000000004b96df in PyImport_ExecCodeModuleEx ()
#43 0x00000000004b2b06 in ?? ()
#44 0x00000000004b402c in ?? ()
#45 0x00000000004a4ae1 in ?? ()
#46 0x00000000004a4513 in PyImport_ImportModuleLevel ()
#47 0x00000000004a59e4 in ?? ()
#48 0x00000000004a577e in PyObject_Call ()
#49 0x00000000004c5e10 in PyEval_CallObjectWithKeywords ()
#50 0x00000000004be6d7 in PyEval_EvalFrameEx ()
---Type <return> to continue, or q <return> to quit---
#51 0x00000000004b9ab6 in PyEval_EvalCodeEx ()
#52 0x00000000004b97a6 in PyEval_EvalCode ()
#53 0x00000000004b96df in PyImport_ExecCodeModuleEx ()
#54 0x00000000004b2b06 in ?? ()
#55 0x00000000004b402c in ?? ()
#56 0x00000000004a4ae1 in ?? ()
#57 0x00000000004a3e84 in PyImport_ImportModuleLevel ()
#58 0x00000000004a59e4 in ?? ()
#59 0x00000000004a577e in PyObject_Call ()
#60 0x00000000004c5e10 in PyEval_CallObjectWithKeywords ()
#61 0x00000000004be6d7 in PyEval_EvalFrameEx ()
#62 0x00000000004b9ab6 in PyEval_EvalCodeEx ()
#63 0x00000000004eb30f in ?? ()
#64 0x00000000004e5422 in PyRun_FileExFlags ()
#65 0x00000000004e3cd6 in PyRun_SimpleFileExFlags ()
#66 0x0000000000493ae2 in Py_Main ()
#67 0x00007ffff7810830 in __libc_start_main (main=0x4934c0 <main>, argc=2, argv=0x7fffffffdbc8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffdbb8)
    at ../csu/libc-start.c:291
#68 0x00000000004933e9 in _start ()

fsx950223 pushed a commit to fsx950223/tensorflow that referenced this issue Dec 22, 2023
@jayfurmanek
[ROCM] Adjust run_cpu.sh to match pycpp config in cpu.bazelrc (tensor…
5dd5fae 
…flow#2129)

* [ROCM] Adjust run_cpu.sh to match pycpp config in cpu.bazelrc

* remove no_rocm from cpu tests; adjust pycpp in cpu.bazelrc

* Use no_rocm for cpu test configs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
9 participants
@yaroslavvb @stephenroller @vinnitu @wangyongliang @ushnish @mijung-kim @dexhunter @poxvoculi @arita37