PDM and Poetry can't install symengine properly #474

martinruefenacht · 2024-02-21T09:57:41Z

I am trying to run symengine in a PDM or Poetry project and while I can install it via "pdm add" or "poetry add" and I find it in the virtualenv site-packages, the lib directory is empty compared to my globally installed version with my OS python.

When I run python directly from the venv and import symengine I get:

Python 3.12.1 (main, Dec 18 2023, 00:00:00) [GCC 13.2.1 20231205 (Red Hat 13.2.1-6)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import symengine
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/****/****/****/.venv/lib/python3.12/site-packages/symengine/__init__.py", line 12, in <module>
    import symengine.lib.symengine_wrapper as wrapper
ModuleNotFoundError: No module named 'symengine.lib.symengine_wrapper'

Is this a problem with both Poetry and PDM or with symengine not picking something up?

The text was updated successfully, but these errors were encountered:

rikardn · 2024-02-22T07:46:11Z

I tested to create a venv with Python 3.12 and install symengine without using poetry and it works fine on my computer:

$ python3.12 --version
Python 3.12.0
[~/testing] 
$ python3.12 -m venv py312venv
[~/testing] 
$ cd py312venv
[~/testing/py312venv] 
$ . ./bin/activate
(py312venv) [~/testing/py312venv] 
$ pip install symengine
Collecting symengine
  Obtaining dependency information for symengine from https://files.pythonhosted.org/packages/b5/58/d22492431cefd539a3655ec298752ad56a11c349bb32c668d2567172118f/symengine-0.11.0-cp312-cp312-manylinux_2_12_x86_64.manylinux2010_x86_64.whl.metadata
  Downloading symengine-0.11.0-cp312-cp312-manylinux_2_12_x86_64.manylinux2010_x86_64.whl.metadata (1.2 kB)
Downloading symengine-0.11.0-cp312-cp312-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (39.3 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 39.3/39.3 MB 19.7 MB/s eta 0:00:00
Installing collected packages: symengine
Successfully installed symengine-0.11.0

[notice] A new release of pip is available: 23.2.1 -> 24.0
[notice] To update, run: pip install --upgrade pip
(py312venv) [~/testing/py312venv] 
$ python
Python 3.12.0 (main, Nov 14 2023, 08:54:25) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import symengine
>>>

The files installed:

[~/testing/py312venv/lib/python3.12/site-packages] 
$ tree symengine*              
symengine
├── functions.py
├── __init__.py
├── lib
│   ├── __init__.py
│   ├── __pycache__
│   │   └── __init__.cpython-312.pyc
│   ├── pywrapper.h
│   ├── symengine.pxd
│   ├── symengine_wrapper.cpython-312-x86_64-linux-gnu.so
│   └── symengine_wrapper.pxd
├── printing.py
├── __pycache__
│   ├── functions.cpython-312.pyc
│   ├── __init__.cpython-312.pyc
│   ├── printing.cpython-312.pyc
│   ├── sympy_compat.cpython-312.pyc
│   ├── test_utilities.cpython-312.pyc
│   └── utilities.cpython-312.pyc
├── sympy_compat.py
├── tests
│   ├── __init__.py
│   ├── __pycache__
│   │   ├── __init__.cpython-312.pyc
│   │   ├── test_arit.cpython-312.pyc
│   │   ├── test_cse.cpython-312.pyc
│   │   ├── test_dict_basic.cpython-312.pyc
│   │   ├── test_eval.cpython-312.pyc
│   │   ├── test_expr.cpython-312.pyc
│   │   ├── test_functions.cpython-312.pyc
│   │   ├── test_lambdify.cpython-312.pyc
│   │   ├── test_logic.cpython-312.pyc
│   │   ├── test_matrices.cpython-312.pyc
│   │   ├── test_ntheory.cpython-312.pyc
│   │   ├── test_number.cpython-312.pyc
│   │   ├── test_pickling.cpython-312.pyc
│   │   ├── test_printing.cpython-312.pyc
│   │   ├── test_sage.cpython-312.pyc
│   │   ├── test_series_expansion.cpython-312.pyc
│   │   ├── test_sets.cpython-312.pyc
│   │   ├── test_solve.cpython-312.pyc
│   │   ├── test_subs.cpython-312.pyc
│   │   ├── test_symbol.cpython-312.pyc
│   │   ├── test_sympify.cpython-312.pyc
│   │   ├── test_sympy_compat.cpython-312.pyc
│   │   ├── test_sympy_conv.cpython-312.pyc
│   │   └── test_var.cpython-312.pyc
│   ├── test_arit.py
│   ├── test_cse.py
│   ├── test_dict_basic.py
│   ├── test_eval.py
│   ├── test_expr.py
│   ├── test_functions.py
│   ├── test_lambdify.py
│   ├── test_logic.py
│   ├── test_matrices.py
│   ├── test_ntheory.py
│   ├── test_number.py
│   ├── test_pickling.py
│   ├── test_printing.py
│   ├── test_sage.py
│   ├── test_series_expansion.py
│   ├── test_sets.py
│   ├── test_solve.py
│   ├── test_subs.py
│   ├── test_symbol.py
│   ├── test_sympify.py
│   ├── test_sympy_compat.py
│   ├── test_sympy_conv.py
│   └── test_var.py
├── test_utilities.py
└── utilities.py
symengine-0.11.0.dist-info
├── AUTHORS
├── INSTALLER
├── LICENSE
├── METADATA
├── RECORD
├── REQUESTED
├── top_level.txt
└── WHEEL
symengine.libs
├── libflint-b75bea04.so.15.0.1
├── libgmp-ba3d8273.so.10.5.0
├── libmpc-5c556745.so.3.3.1
├── libmpfr-7f7b9c9c.so.6.2.1
└── libzstd-52a5ab61.so.1.5.5

martinruefenacht · 2024-02-22T08:38:25Z

Yes, I got the same with just a normal venv.

What I notice is that several directories in the site-packages are missing in the pdm version:

symengine
├── functions.py
├── __init__.py
├── lib
│   ├── __init__.py
│   └── __pycache__
│       └── __init__.cpython-312.pyc
├── printing.py
├── __pycache__
│   └── __init__.cpython-312.pyc
├── sympy_compat.py
├── tests
│   ├── __init__.py
│   ├── test_arit.py
│   ├── test_cse.py
│   ├── test_dict_basic.py
│   ├── test_eval.py
│   ├── test_expr.py
│   ├── test_functions.py
│   ├── test_lambdify.py
│   ├── test_logic.py
│   ├── test_matrices.py
│   ├── test_ntheory.py
│   ├── test_number.py
│   ├── test_pickling.py
│   ├── test_printing.py
│   ├── test_sage.py
│   ├── test_series_expansion.py
│   ├── test_sets.py
│   ├── test_solve.py
│   ├── test_subs.py
│   ├── test_symbol.py
│   ├── test_sympify.py
│   ├── test_sympy_compat.py
│   ├── test_sympy_conv.py
│   └── test_var.py
├── test_utilities.py
└── utilities.py

The lib/ folder is empty, symengine.libs doesn't get created, and symengine-0.11.

Perhaps this is more a question for pdm and poetry than symengine? They seem to be missing the external libraries.

rikardn · 2024-02-22T09:13:54Z

Could still be a problem with symengine. Does it work if you use poetry to install some other library that also uses compiled code, like for example numpy?

martinruefenacht · 2024-02-22T10:24:53Z

Yes, numpy works without issue...

martinruefenacht · 2024-02-22T10:32:04Z

I just discovered that the lib/, symengine-0.11.0.dist-info, symengine.libs are placed in the lib64 directory.

.venv/lib
└── python3.12
    └── site-packages
        ├── __pycache__
        │   └── _virtualenv.cpython-312.pyc
        ├── symengine
        │   ├── functions.py
        │   ├── __init__.py
        │   ├── lib
        │   │   └── __init__.py
        │   ├── printing.py
        │   ├── sympy_compat.py
        │   ├── tests
        │   │   ├── __init__.py
        │   │   ├── test_arit.py
        │   │   ├── test_cse.py
        │   │   ├── test_dict_basic.py
        │   │   ├── test_eval.py
        │   │   ├── test_expr.py
        │   │   ├── test_functions.py
        │   │   ├── test_lambdify.py
        │   │   ├── test_logic.py
        │   │   ├── test_matrices.py
        │   │   ├── test_ntheory.py
        │   │   ├── test_number.py
        │   │   ├── test_pickling.py
        │   │   ├── test_printing.py
        │   │   ├── test_sage.py
        │   │   ├── test_series_expansion.py
        │   │   ├── test_sets.py
        │   │   ├── test_solve.py
        │   │   ├── test_subs.py
        │   │   ├── test_symbol.py
        │   │   ├── test_sympify.py
        │   │   ├── test_sympy_compat.py
        │   │   ├── test_sympy_conv.py
        │   │   └── test_var.py
        │   ├── test_utilities.py
        │   └── utilities.py
        ├── _virtualenv.pth
        └── _virtualenv.py
.venv/lib64
└── python3.12
    └── site-packages
        ├── symengine
        │   └── lib
        │       ├── pywrapper.h
        │       ├── symengine.pxd
        │       ├── symengine_wrapper.cpython-312-x86_64-linux-gnu.so
        │       └── symengine_wrapper.pxd
        ├── symengine-0.11.0.dist-info
        │   ├── AUTHORS
        │   ├── LICENSE
        │   ├── METADATA
        │   ├── RECORD
        │   ├── top_level.txt
        │   └── WHEEL
        └── symengine.libs
            ├── libflint-b75bea04.so.15.0.1
            ├── libgmp-ba3d8273.so.10.5.0
            ├── libmpc-5c556745.so.3.3.1
            ├── libmpfr-7f7b9c9c.so.6.2.1
            └── libzstd-52a5ab61.so.1.5.5

Whereas numpy gets placed entirely in the lib64 site-packages not spread across.

martinruefenacht · 2024-02-22T10:37:53Z

And when I move all the files that are placed into the .venv/lib/python3.12/site-packages/ into the lib64 then everything works are expected.

rikardn · 2024-02-22T10:56:32Z

Good find @martinruefenacht! I think this is the crucial problem. Now we just need to figure out if this is caused by the symengine wheel being composed in a faulty way or if this is the fault of the installation procedure.

martinruefenacht · 2024-02-22T11:13:46Z

Well I am not familiar with this stuff. I unpacked the numpy and symengine wheels to have a look. What stood out was that the symengine has a .data directory and that contains what I see in the lib/ instead of the lib64/.

symengine
└── lib
    ├── pywrapper.h
    ├── symengine.pxd
    ├── symengine_wrapper.cpython-312-x86_64-linux-gnu.so
    └── symengine_wrapper.pxd
symengine-0.11.0.data
└── purelib
    └── symengine
        ├── functions.py
        ├── __init__.py
        ├── lib
        │   └── __init__.py
        ├── printing.py
        ├── sympy_compat.py
        ├── tests
        │   ├── __init__.py
        │   ├── test_arit.py
        │   ├── test_cse.py
        │   ├── test_dict_basic.py
        │   ├── test_eval.py
        │   ├── test_expr.py
        │   ├── test_functions.py
        │   ├── test_lambdify.py
        │   ├── test_logic.py
        │   ├── test_matrices.py
        │   ├── test_ntheory.py
        │   ├── test_number.py
        │   ├── test_pickling.py
        │   ├── test_printing.py
        │   ├── test_sage.py
        │   ├── test_series_expansion.py
        │   ├── test_sets.py
        │   ├── test_solve.py
        │   ├── test_subs.py
        │   ├── test_symbol.py
        │   ├── test_sympify.py
        │   ├── test_sympy_compat.py
        │   ├── test_sympy_conv.py
        │   └── test_var.py
        ├── test_utilities.py
        └── utilities.py
symengine-0.11.0.dist-info
├── AUTHORS
├── LICENSE
├── METADATA
├── RECORD
├── top_level.txt
└── WHEEL
symengine.libs
├── libflint-b75bea04.so.15.0.1
├── libgmp-ba3d8273.so.10.5.0
├── libmpc-5c556745.so.3.3.1
├── libmpfr-7f7b9c9c.so.6.2.1
└── libzstd-52a5ab61.so.1.5.5

mrossinek · 2024-03-06T09:11:28Z

I just ran into this myself while symengine is being installed as part of a tox workflow.
I am running Fedora 39. It is odd that everything works fine outside of tox.

I performed a git bisection to find the first bad commit based on the fact that I knew v0.9.2 to be working.
I made sure to always have the correct symengine version as well as compatible Cython installed. I used Python 3.10 since this is supported by all versions between 0.9 and 0.11.

The commit which introduced the failure to build as expected is: bc84086

I am not quite sure what to make of this but hopefully this helps in finding a solution.

rikardn · 2024-03-06T09:15:03Z

@mrossinek are you also using poetry?

mrossinek · 2024-03-06T09:16:44Z

No I am using pure pip for the installation. But the case in which it breaks is when this gets executed as part of tox

rikardn · 2024-03-06T09:25:15Z

Strange. I use symengine myself with pip and tox and that works fine. I wonder if it is the same cause as in the poetry case.

martinruefenacht · 2024-03-06T09:27:43Z

I am also on Fedora if that helps.

mrossinek · 2024-03-06T09:28:41Z

I had a colleague point out to me, that it could possibly be related to how the OS (in this case Fedora) handles lib vs lib64 in Python. I did not find much that would be helpful on this, hence I did the bisection.

mrossinek · 2024-03-06T09:45:59Z

Okay, here a few more insights.

that first bad commit was part of Fix build to avoid duplicate files in wheel #402 which also includes an option to re-enable the previous behavior by setting SYMENGINE_INSTALL_PY_FILES=ON. Inside setup.py this is hard-coded to OFF when calling cmake. Changing this to ON does indeed make the latest commit (2223b99) work as expected with my tox
digging into the lib vs lib64 got me thinking, why things worked for me outside of tox. I noticed that my venv has a symlink lib64 -> lib which is not what tox is doing. The creation of this symlink seems to have changed at some point: see lib64 directory is no longer symlinked to lib? pypa/virtualenv#1751 which is also linked to by a poetry issue, possibly explaining Martin's observations
and indeed, tox uses virtualenv > 20 which could explain this behavior

mrossinek · 2024-03-06T09:53:28Z

Even more information:

We seem to be running into the distinction of platlib and purelib which differs on Fedora: https://peps.python.org/pep-0427/#what-s-the-deal-with-purelib-vs-platlib

In setup.py, this project uses platlib consistently. With the symlink between lib and lib64 gone in virtualenv>=20 (which is what both, tox and poetry, use under the hood to manage their environment), this is likely the reason why symengine fails to install properly in both cases.

I have not worked with Python + cmake before, so the question remains on how to resolve this problem.

mrossinek · 2024-03-06T10:43:51Z

What I do not understand, is why the wheels that are built for symengine place their files in purelib.
I would expect them to use platlib given that root_is_pure = False is set inside of setup.py and that the C module is obviously not pure Python...

martinruefenacht · 2024-03-06T17:05:56Z

I just replicated the finding of @mrossinek with both tox and nox (3.8-3.12). Same problem as I found with poetry and pdm. This was all on Fedora. For fun I tested on Debian quickly with poetry and tox (3.11) and those worked.

JMuff22 · 2024-04-23T17:11:10Z

Also have this error on rockylinux:8.8. Any fixes so far? Problem seems to be that pip install symengine places files in /usr/local/lib64 and not /usr/local/lib

martinruefenacht · 2024-04-23T18:12:41Z

My hotfix is to copy everything in my virtual environment to the /lib... manually.

I don't know how to fix this issue, I assume the setup.py has to be adjusted somehow to modify the packaging for fedora specifically?

JMuff22 · 2024-04-23T18:32:22Z

This works

git clone https://github.com/symengine/symengine.py.git
 cd symengine.py
 git checkout v0.11.0
 sed -i 's/-DSYMENGINE_INSTALL_PY_FILES=OFF/-DSYMENGINE_INSTALL_PY_FILES=ON/' setup.py
 python setup.py install

rikardn transferred this issue from symengine/symengine Feb 22, 2024

JMuff22 mentioned this issue Apr 24, 2024

[qiskit-aer-gpu] ImportError: libcustatevec.so.1 Qiskit/qiskit-aer#1874

Open

mrossinek mentioned this issue May 16, 2024

The provided wheels faultily report to be pure Python symengine/symengine-wheels#16

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PDM and Poetry can't install symengine properly #474

PDM and Poetry can't install symengine properly #474

martinruefenacht commented Feb 21, 2024

rikardn commented Feb 22, 2024

martinruefenacht commented Feb 22, 2024 •

edited

rikardn commented Feb 22, 2024

martinruefenacht commented Feb 22, 2024

martinruefenacht commented Feb 22, 2024

martinruefenacht commented Feb 22, 2024

rikardn commented Feb 22, 2024

martinruefenacht commented Feb 22, 2024

mrossinek commented Mar 6, 2024

rikardn commented Mar 6, 2024

mrossinek commented Mar 6, 2024

rikardn commented Mar 6, 2024

martinruefenacht commented Mar 6, 2024

mrossinek commented Mar 6, 2024

mrossinek commented Mar 6, 2024

mrossinek commented Mar 6, 2024

mrossinek commented Mar 6, 2024

martinruefenacht commented Mar 6, 2024

JMuff22 commented Apr 23, 2024

martinruefenacht commented Apr 23, 2024

JMuff22 commented Apr 23, 2024

PDM and Poetry can't install symengine properly #474

PDM and Poetry can't install symengine properly #474

Comments

martinruefenacht commented Feb 21, 2024

rikardn commented Feb 22, 2024

martinruefenacht commented Feb 22, 2024 • edited

rikardn commented Feb 22, 2024

martinruefenacht commented Feb 22, 2024

martinruefenacht commented Feb 22, 2024

martinruefenacht commented Feb 22, 2024

rikardn commented Feb 22, 2024

martinruefenacht commented Feb 22, 2024

mrossinek commented Mar 6, 2024

rikardn commented Mar 6, 2024

mrossinek commented Mar 6, 2024

rikardn commented Mar 6, 2024

martinruefenacht commented Mar 6, 2024

mrossinek commented Mar 6, 2024

mrossinek commented Mar 6, 2024

mrossinek commented Mar 6, 2024

mrossinek commented Mar 6, 2024

martinruefenacht commented Mar 6, 2024

JMuff22 commented Apr 23, 2024

martinruefenacht commented Apr 23, 2024

JMuff22 commented Apr 23, 2024

martinruefenacht commented Feb 22, 2024 •

edited