Error installing graphkernels/GKextCPy: Unable to locate python.h header file.

[git-tea]

Hello!

I want to install graphkernels, and am receiving the output at the following file when using the command: pip --no-cache-dir install graphkernels:
graphkernels_install_err.txt

The main error from this seems to be "Python.h: No such file or directory". My understanding is that this usually means that pythonX.Y-dev must be installed according to the python version used.

I have confirmed that I do have python3.9-dev installed corresponding to my version of python using the command sudo apt-get install python3.9-dev, which gave the output "python3.9-dev is already the newest version (3.9.13-1+jammy1)." And the Python.h file exists on my system at /usr/include/python3.9/Python.h.

Is there something else I should do to install the package?

[Pseudomanifold]

Sorry to hear this—not aware of this problem, but it's been a while since I was working on this project. Can you try installing everything from source (i.e. from the src directory of this repo)?

[git-tea]

Thank you for the quick response!

I have tried to install via src, and I get the output in the following file:
graphkernels_src_install.txt

I do not see any obvious errors during the installation. However, I seem to come across a second error later on.

When I try to call CalculateWLKernel from my script, I get the following error:
kernel_mat = gk.CalculateWLKernel( relabeled_graphs, n_iters )
File "/opt/GraphKernels/src/graphkernels/graphkernels/kernels.py", line 160, in CalculateWLKernel
E, V_label, V_count, E_count, D_max = GetGKInput(G)
File "/opt/GraphKernels/src/graphkernels/graphkernels/utilities.py", line 71, in GetGKInput
E = gkCpy.VecMatrixXi()
AttributeError: module 'GKextCPy' has no attribute 'VecMatrixXi'

Is there a separate path that I need to link to be able to locate these attributes?

[Pseudomanifold]

That's weird—can you load the module and tell me the output of the following command, please:

import pathlib, GKextCPy; pathlib.Path(GKextCPy.__file__).resolve().parent

[git-tea]

The following is what is returned from running that command in the python interpreter:

">>> import pathlib, GKextCPy; pathlib.Path(GKextCPy.file).resolve().parent
Traceback (most recent call last):
File "/opt/GraphKernels/src/GKextCPy/GKextCPy.py", line 14, in swig_import_helper
return importlib.import_module(mname)
File "/opt/anaconda3/lib/python3.9/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1030, in _gcd_import
File "", line 1007, in _find_and_load
File "", line 984, in _find_and_load_unlocked
ModuleNotFoundError: No module named '_GKextCPy'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "", line 1, in
File "/opt/GraphKernels/src/GKextCPy/GKextCPy.py", line 17, in
_GKextCPy = swig_import_helper()
File "/opt/GraphKernels/src/GKextCPy/GKextCPy.py", line 16, in swig_import_helper
return importlib.import_module('_GKextCPy')
File "/opt/anaconda3/lib/python3.9/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
ModuleNotFoundError: No module named '_GKextCPy'"

[Pseudomanifold]

This looks to me like the extension module was not installed. Can you try to install this manually as well (from the GKextCPy folder)?

[git-tea]

I believe that I have now installed GKextCPy from src using the corresponding setup.py file.

I ran python setup.py build and then python setup.py install --user like with graphkernels. I then repeated the build and installation of graphkernels.

I get the same error when I repeat the test from above (import pathlib, GKextCPy; pathlib.Path(GKextCPy.__file__).resolve().parent)

[Pseudomanifold]

I don't understand this fully. Is the module maybe already installed (if somewhat partially)? The setup script might refuse to overwrite it.

[git-tea]

I have cleared all the references I can find to the module and tried to install it from source from scratch.

After clearing everything, the entire local GraphKernels directory was gone, and there was no listing of either GKextCPy or graphkernels in pip list.

I re-cloned the repository and both built and installed GKextCPy and then graphkernels. I also built and installed with sudo permissions to ensure no access problems. The following files correspond to each command, such that the setup.py files are the ones inside their corresponding package.
GKextCPy - sudo ./setup.py build: gkcpy_build.txt
GKextCPy - sudo ./setup.py install --user: gkcpy_install.txt
graphkernels - sudo ./setup.py build: gk_build.txt
graphkernels - sudo ./setup.py install --user: gk_install.txt

When I then run the command above import pathlib, GKextCPy; pathlib.Path(GKextCPy.__file__).resolve().parent from a python interpreter in my home directory, I get the following error:
->>> import pathlib, GKextCPy; pathlib.Path(GKextCPy.file).resolve().parent
Traceback (most recent call last):
File "", line 1, in
ModuleNotFoundError: No module named 'GKextCPy'

[Pseudomanifold]

Sorry for the hassle here...

It seems that the packages are being installed for the root user only. If you use sudo, you can drop the --user flag. This should install the packages globally (might cause other problems, though; maybe a virtual environment would be best).

[git-tea]

It's no problem. I appreciate you taking the time to help with this.

I've deleted and re-installed it again to make sure there's no conflict, and I installed it without the --user flag. I seem to be getting the same ModuleNotFoundError when running the test command.

Do I need to run the command from within a specific directory to ensure that all packages are locate-able?

When I run the test from somewhere outside of the GraphKernels directories, I get the ModuleNotFoundError above:
->>> import pathlib, GKextCPy; pathlib.Path(GKextCPy.file).resolve().parent
Traceback (most recent call last):
File "", line 1, in
ModuleNotFoundError: No module named 'GKextCPy'

When I run within GraphKernels/src, I get the following error:
->>> import pathlib, GKextCPy; pathlib.Path(GKextCPy.file).resolve().parent
Traceback (most recent call last):
File "", line 1, in
File "/opt/anaconda3/lib/python3.9/pathlib.py", line 1082, in new
self = cls._from_parts(args, init=False)
File "/opt/anaconda3/lib/python3.9/pathlib.py", line 707, in _from_parts
drv, root, parts = self._parse_args(args)
File "/opt/anaconda3/lib/python3.9/pathlib.py", line 691, in _parse_args
a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not NoneType

And when I run within the GraphKernels/src/GKextCPy directory, I get the following errors:
->>> import pathlib, GKextCPy; pathlib.Path(GKextCPy.file).resolve().parent
Traceback (most recent call last):
File "/opt/GraphKernels/src/GKextCPy/GKextCPy.py", line 14, in swig_import_helper
return importlib.import_module(mname)
File "/opt/anaconda3/lib/python3.9/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1030, in _gcd_import
File "", line 1007, in _find_and_load
File "", line 984, in _find_and_load_unlocked
ModuleNotFoundError: No module named '_GKextCPy'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "", line 1, in
File "/opt/GraphKernels/src/GKextCPy/GKextCPy.py", line 17, in
_GKextCPy = swig_import_helper()
File "/opt/GraphKernels/src/GKextCPy/GKextCPy.py", line 16, in swig_import_helper
return importlib.import_module('_GKextCPy')
File "/opt/anaconda3/lib/python3.9/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
ModuleNotFoundError: No module named '_GKextCPy'

[Pseudomanifold]

Still looks like there's something wrong with the installation. I see that you have Anaconda active as well. Can you try installing it in the Anaconda environment using --user and no use of sudo?

[Pseudomanifold]

(I just pushed a simplified module installation fix, but that should not apply to your issue, unfortunately. If we can't get anywhere, the main developers @mahito-sugiyama or @eghisu might want to weigh in.)

[git-tea]

When you say to install it in the Anaconda environment, do you mean using pip to install it? Or do you mean to install it into the directories for the conda packages?

[Pseudomanifold]

I mean installing it using setup.py while the Anaconda environment is active, but without using sudo. From what I saw in the logs, your previous installation used a different Python version than the one your Anaconda environment uses.

[git-tea]

The package is within my /opt directory, so it has limited permissions by default.

To account for that, I have now added write permissions (using chmod +w) to each of the directories of GraphKernels so that the build process can create what it needs to.

Afterward, I ran python3 setup.py build on the setup.py file in GKextCPy, and it returned the following error. It appears to be the same error that I saw when installing via pip:

Requirement already satisfied: numpy in /opt/anaconda3/lib/python3.9/site-packages (1.21.5)
Requirement already satisfied: pkgconfig in /opt/anaconda3/lib/python3.9/site-packages (1.5.5)
running build
running build_py
creating build/lib.linux-x86_64-3.9
copying GKextCPy.py -> build/lib.linux-x86_64-3.9
running build_ext
building '_GKextCPy' extension
creating build/temp.linux-x86_64-3.9
gcc -pthread -B /opt/anaconda3/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /opt/anaconda3/include -I/opt/anaconda3/include -fPIC -O2 -isystem /opt/anaconda3/include -fPIC -I/opt/anaconda3/lib/python3.9/site-packages/numpy/core/include -I -I/opt/anaconda3/include/python3.9 -c GKextCPy.cpp -o build/temp.linux-x86_64-3.9/GKextCPy.o -std=c++11 -O3
gcc -pthread -B /opt/anaconda3/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /opt/anaconda3/include -I/opt/anaconda3/include -fPIC -O2 -isystem /opt/anaconda3/include -fPIC -I/opt/anaconda3/lib/python3.9/site-packages/numpy/core/include -I -I/opt/anaconda3/include/python3.9 -c GKextCPy_wrap.cxx -o build/temp.linux-x86_64-3.9/GKextCPy_wrap.o -std=c++11 -O3
GKextCPy_wrap.cxx:173:11: fatal error: Python.h: No such file or directory
173 | # include <Python.h>
| ^~~~~~~~~~
compilation terminated.
error: command '/usr/bin/gcc' failed with exit code 1

[Pseudomanifold]

Do you have the dev files for Python installed within your Anaconda environment? I think the command you ran at the very beginning only installs them for your global installation.

I am not sure I understand the discussion on the /opt dir. What I would suggest is to clone the package into your home, activate your conda env, then run setup.py from there (after potentially installing dev headers; the package is named python-devtools or something).

[git-tea]

I confirmed that I already have the devtools within the conda environment:

$ conda install -c conda-forge python-devtools
Collecting package metadata (current_repodata.json): done
Solving environment: done
All requested packages already installed.

Additionally, I have cloned the repository into my home directory and tried to build GKextCPy from there, but I seem to get the same 'Python.h: No such file or directory' error.

Specifically, when I run python setup.py build from the GKextCPy directory, I get the output at the following file:
gkcpy_build.txt

[Pseudomanifold]

Things are getting curiouser and curiouser. Can you check where the file exists for your current Python installation? (It seems at this point that this is not necessary a problem with our setup.py since we cannot do more than rely on the paths provided in the system.)

[git-tea]

The following are the paths I know of related to Python:

Python binary: /opt/anaconda3/bin/python
Python.h header: /usr/include/python3.9/Python.h

Is there a separate python file or directory that would be helpful to locate?

[Pseudomanifold]

What I find weird is that the second directory is not being used during the setup process. Can you try providing this directory as an additional CFLAGS argument before calling setup.py?

[git-tea]

Thank you for being patient while I work on this.

It seems that the place that /usr/include/python3.9 was not included was in the sysconfig settings. So I updated the GKextCPy setup file such that the extra compile arguments takes this form:
extra_compile_args = ['-std=c++11', '-O3', '-I/usr/include/python3.9/']

This update allowed GKextCPy and graphkernels to build and install from src seemingly without errors.

However, when I try to run the test import command above (import pathlib, GKextCPy; pathlib.Path(GKextCPy.__file__).resolve().parent), I get the following error. The test command was run from outside of the Graphkernels directory after the installation completed:

>>> import pathlib, GKextCPy; pathlib.Path(GKextCPy.file).resolve().parent
Traceback (most recent call last):
File "/opt/GraphKernels/src/GKextCPy/GKextCPy.py", line 14, in swig_import_helper
return importlib.import_module(mname)
File "/opt/anaconda3/lib/python3.9/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1030, in _gcd_import
File "", line 1007, in _find_and_load
File "", line 984, in _find_and_load_unlocked
ModuleNotFoundError: No module named '_GKextCPy'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "", line 1, in
File "/opt/GraphKernels/src/GKextCPy/GKextCPy.py", line 17, in
_GKextCPy = swig_import_helper()
File "/opt/GraphKernels/src/GKextCPy/GKextCPy.py", line 16, in swig_import_helper
return importlib.import_module('_GKextCPy')
File "/opt/anaconda3/lib/python3.9/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
ModuleNotFoundError: No module named '_GKextCPy'

[Pseudomanifold]

Hmm, what I don't understand about this issue is that the Anaconda headers should 'live' under a different path than /usr/include. But in any case: the module seems to build and install now—but I am guessing that from your output, not even the normal import worked...

Can you please attach the output of python setup.by install --user for the GKextCPy module? Seems like something is very wrong with the paths.

(Not the main developer here; at this point I would really appreciate any additional input from @mahito-sugiyama or @eghisu)

[git-tea]

I was able to locate a separate version of the Python headers under the Anaconda installation at /opt/anaconda3/pkgs/python-3.9.12-h12debd9_0/include/python3.9/Python.h.

I also removed any existing versions of the Graphkernels repository on my system to ensure that there is not an existing installation elsewhere on the system being identified. And I recloned it into my home directory.

Then I take the following steps:

1. I update the setup.py file with the additional python header to cflags as listed above within Anaconda. And I run python setup.py build. This produces the following output, which doesn't seem to have any obvious errors: gkcpy_build.txt
2. Afterward, I run the command python setup.py install --user and get the following output: gkcpy_install.txt
3. Finally, I change back to my home directory outside of the GraphKernels directory to run the test command import pathlib, GKextCPy; pathlib.Path(GKextCPy.__file__).resolve().parent from a python interpreter. It produces the following output:

>>> import pathlib, GKextCPy; pathlib.Path(GKextCPy.__file__).resolve().parent
Traceback (most recent call last):
  File "/home/exouser/.local/lib/python3.9/site-packages/GKextCPy-0.4.1-py3.9-linux-x86_64.egg/GKextCPy.py", line 14, in swig_import_helper
    return importlib.import_module(mname)
  File "/opt/anaconda3/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 666, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 565, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 1173, in create_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
ImportError: /home/exouser/.local/lib/python3.9/site-packages/GKextCPy-0.4.1-py3.9-linux-x86_64.egg/_GKextCPy.cpython-39-x86_64-linux-gnu.so: undefined symbol: _ZSt28__throw_bad_array_new_lengthv
    
During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/exouser/.local/lib/python3.9/site-packages/GKextCPy-0.4.1-py3.9-linux-x86_64.egg/GKextCPy.py", line 17, in <module>
    _GKextCPy = swig_import_helper()
  File "/home/exouser/.local/lib/python3.9/site-packages/GKextCPy-0.4.1-py3.9-linux-x86_64.egg/GKextCPy.py", line 16, in swig_import_helper
    return importlib.import_module('_GKextCPy')
  File "/opt/anaconda3/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
ImportError: /home/exouser/.local/lib/python3.9/site-packages/GKextCPy-0.4.1-py3.9-linux-x86_64.egg/_GKextCPy.cpython-39-x86_64-linux-gnu.so: undefined symbol: _ZSt28__throw_bad_array_new_lengthv

[Pseudomanifold]

OK, now there appears to be an issue with the linker dependencies. Is this the same environment? Might be an issue between the system C++ compiler and the Anaconda one. Can you check stan-dev/pystan#294 for a potential workaround?

At this point, it seems that we are fighting a losing battle—maybe consider trying the GraKeL library?