Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

delocate-wheel sets invalid relative paths to delocated dylibs (related to "purelib" subdirectory) #149

Open
maxhgerlach opened this issue Mar 15, 2022 · 1 comment
Open

delocate-wheel sets invalid relative paths to delocated dylibs (related to "purelib" subdirectory) #149

maxhgerlach opened this issue Mar 15, 2022 · 1 comment
Labels
bug

Comments

@maxhgerlach
Copy link

maxhgerlach commented Mar 15, 2022 •
edited

Describe the bug / To Reproduce
My original wheel contains these files:

$ unzip -l dist/PyMyLib-0.1.17-cp39-cp39-macosx_12_0_arm64.whl
Archive:  dist/PyMyLib-0.1.17-cp39-cp39-macosx_12_0_arm64.whl
  Length      Date    Time    Name
---------  ---------- -----   ----
  4444243  03-15-2022 11:19   PyMyLib-0.1.17.data/purelib/PyMyLib/PyMyLib.cpython-39-darwin.so
      105  03-15-2022 11:19   PyMyLib-0.1.17.data/purelib/PyMyLib/__init__.py
      215  03-15-2022 11:19   PyMyLib-0.1.17.dist-info/METADATA
      108  03-15-2022 11:19   PyMyLib-0.1.17.dist-info/WHEEL
       18  03-15-2022 11:19   PyMyLib-0.1.17.dist-info/top_level.txt
      619  03-15-2022 11:19   PyMyLib-0.1.17.dist-info/RECORD
---------                     -------
  4445308                     6 files

It depends on Boost, ICU, and PCRE, which have been installed via Homebrew:

$ delocate-listdeps dist/PyMyLib-0.1.17-cp39-cp39-macosx_12_0_arm64.whl
/opt/homebrew/Cellar/boost/1.78.0_1/lib/libboost_atomic-mt.dylib
/opt/homebrew/Cellar/boost/1.78.0_1/lib/libboost_chrono-mt.dylib
/opt/homebrew/Cellar/boost/1.78.0_1/lib/libboost_date_time-mt.dylib
/opt/homebrew/Cellar/boost/1.78.0_1/lib/libboost_filesystem-mt.dylib
/opt/homebrew/Cellar/boost/1.78.0_1/lib/libboost_iostreams-mt.dylib
/opt/homebrew/Cellar/boost/1.78.0_1/lib/libboost_log-mt.dylib
/opt/homebrew/Cellar/boost/1.78.0_1/lib/libboost_random-mt.dylib
/opt/homebrew/Cellar/boost/1.78.0_1/lib/libboost_regex-mt.dylib
/opt/homebrew/Cellar/boost/1.78.0_1/lib/libboost_serialization-mt.dylib
/opt/homebrew/Cellar/boost/1.78.0_1/lib/libboost_system-mt.dylib
/opt/homebrew/Cellar/boost/1.78.0_1/lib/libboost_thread-mt.dylib
/opt/homebrew/Cellar/boost/1.78.0_1/lib/libboost_unit_test_framework-mt.dylib
/opt/homebrew/Cellar/icu4c/70.1/lib/libicudata.70.1.dylib
/opt/homebrew/Cellar/icu4c/70.1/lib/libicui18n.70.1.dylib
/opt/homebrew/Cellar/icu4c/70.1/lib/libicuuc.70.1.dylib
/opt/homebrew/Cellar/pcre/8.45/lib/libpcre.1.dylib

I call delocate-wheel like this:

$ delocate-wheel -w dist-patched -v dist/*whl

The dylibs are properly copied into the delocated wheel:

$ unzip -l dist-patched/PyMyLib-0.1.17-cp39-cp39-macosx_12_0_arm64.whl
Archive:  dist-patched/PyMyLib-0.1.17-cp39-cp39-macosx_12_0_arm64.whl
  Length      Date    Time    Name
---------  ---------- -----   ----
  1040912  03-15-2022 12:31   PyMyLib.dylibs/libboost_unit_test_framework-mt.dylib
    88048  03-15-2022 12:31   PyMyLib.dylibs/libboost_chrono-mt.dylib
   288224  03-15-2022 12:31   PyMyLib.dylibs/libpcre.1.dylib
   227408  03-15-2022 12:31   PyMyLib.dylibs/libboost_filesystem-mt.dylib
   180704  03-15-2022 12:31   PyMyLib.dylibs/libboost_thread-mt.dylib
    35456  03-15-2022 12:31   PyMyLib.dylibs/libboost_system-mt.dylib
  2780192  03-15-2022 12:31   PyMyLib.dylibs/libicui18n.70.1.dylib
    89968  03-15-2022 12:31   PyMyLib.dylibs/libboost_atomic-mt.dylib
   154208  03-15-2022 12:31   PyMyLib.dylibs/libboost_iostreams-mt.dylib
  1748992  03-15-2022 12:31   PyMyLib.dylibs/libicuuc.70.1.dylib
 29723472  03-15-2022 12:31   PyMyLib.dylibs/libicudata.70.1.dylib
  1337616  03-15-2022 12:31   PyMyLib.dylibs/libboost_log-mt.dylib
    84288  03-15-2022 12:31   PyMyLib.dylibs/libboost_random-mt.dylib
   449360  03-15-2022 12:31   PyMyLib.dylibs/libboost_regex-mt.dylib
    35536  03-15-2022 12:31   PyMyLib.dylibs/libboost_date_time-mt.dylib
   634416  03-15-2022 12:31   PyMyLib.dylibs/libboost_serialization-mt.dylib
      105  03-15-2022 11:19   PyMyLib-0.1.17.data/purelib/PyMyLib/__init__.py
  4462352  03-15-2022 12:31   PyMyLib-0.1.17.data/purelib/PyMyLib/PyMyLib.cpython-39-darwin.so
     2376  03-15-2022 12:31   PyMyLib-0.1.17.dist-info/RECORD
      108  03-15-2022 11:19   PyMyLib-0.1.17.dist-info/WHEEL
       18  03-15-2022 11:19   PyMyLib-0.1.17.dist-info/top_level.txt
      215  03-15-2022 11:19   PyMyLib-0.1.17.dist-info/METADATA
---------                     -------
 43363974                     22 files

The output of delocate-listdeps looks reasonable at first sight:

$ delocate-listdeps dist-patched/PyMyLib-0.1.17-cp39-cp39-macosx_12_0_arm64.whl
PyMyLib.dylibs/libboost_atomic-mt.dylib
PyMyLib.dylibs/libboost_chrono-mt.dylib
PyMyLib.dylibs/libboost_date_time-mt.dylib
PyMyLib.dylibs/libboost_filesystem-mt.dylib
PyMyLib.dylibs/libboost_iostreams-mt.dylib
PyMyLib.dylibs/libboost_log-mt.dylib
PyMyLib.dylibs/libboost_random-mt.dylib
PyMyLib.dylibs/libboost_regex-mt.dylib
PyMyLib.dylibs/libboost_serialization-mt.dylib
PyMyLib.dylibs/libboost_system-mt.dylib
PyMyLib.dylibs/libboost_thread-mt.dylib
PyMyLib.dylibs/libboost_unit_test_framework-mt.dylib
PyMyLib.dylibs/libicudata.70.1.dylib
PyMyLib.dylibs/libicui18n.70.1.dylib
PyMyLib.dylibs/libicuuc.70.1.dylib
PyMyLib.dylibs/libpcre.1.dylib

However, otool -L shows me these relative paths:

$ cd dist-patched/
dist-patched$ unzip PyMyLib-0.1.17-cp39-cp39-macosx_12_0_arm64.whl
dist-patched$ otool -L PyMyLib-0.1.17.data/purelib/PyMyLib/PyMyLib.cpython-39-darwin.so
PyMyLib-0.1.17.data/purelib/PyMyLib/PyMyLib.cpython-39-darwin.so:
	@loader_path/../../../PyMyLib.dylibs/libboost_date_time-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
	@loader_path/../../../PyMyLib.dylibs/libboost_iostreams-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
	@loader_path/../../../PyMyLib.dylibs/libboost_random-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
	@loader_path/../../../PyMyLib.dylibs/libboost_serialization-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
	@loader_path/../../../PyMyLib.dylibs/libboost_system-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
	@loader_path/../../../PyMyLib.dylibs/libboost_unit_test_framework-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
	@loader_path/../../../PyMyLib.dylibs/libboost_log-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
	@loader_path/../../../PyMyLib.dylibs/libicudata.70.1.dylib (compatibility version 70.0.0, current version 70.1.0)
	@loader_path/../../../PyMyLib.dylibs/libicui18n.70.1.dylib (compatibility version 70.0.0, current version 70.1.0)
	@loader_path/../../../PyMyLib.dylibs/libicuuc.70.1.dylib (compatibility version 70.0.0, current version 70.1.0)
	@loader_path/../../../PyMyLib.dylibs/libpcre.1.dylib (compatibility version 4.0.0, current version 4.13.0)
	@loader_path/../../../PyMyLib.dylibs/libboost_chrono-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
	@loader_path/../../../PyMyLib.dylibs/libboost_regex-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
	@loader_path/../../../PyMyLib.dylibs/libboost_filesystem-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
	@loader_path/../../../PyMyLib.dylibs/libboost_thread-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
	@loader_path/../../../PyMyLib.dylibs/libboost_atomic-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1200.3.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.0.0)

Note all the relative paths @loader_path/../../../PyMyLib.dylibs. While these match the files as they are placed inside the wheel archive, after pip install these relative paths don't point to the dylibs under site-packages:

dist-patched$ pip install -U --force-reinstall PyMyLib-0.1.17-cp39-cp39-macosx_12_0_arm64.whl
# in my virtual environment:
.venv-devel-macos-arm64-tf26/lib/python3.9/site-packages$ ls PyMyLib*
PyMyLib:
PyMyLib.cpython-39-darwin.so __init__.py                            __pycache__

PyMyLib-0.1.17.dist-info:
INSTALLER       METADATA        RECORD          REQUESTED       WHEEL           direct_url.json top_level.txt

PyMyLib.dylibs:
libboost_atomic-mt.dylib              libboost_iostreams-mt.dylib           libboost_serialization-mt.dylib       libicudata.70.1.dylib
libboost_chrono-mt.dylib              libboost_log-mt.dylib                 libboost_system-mt.dylib              libicui18n.70.1.dylib
libboost_date_time-mt.dylib           libboost_random-mt.dylib              libboost_thread-mt.dylib              libicuuc.70.1.dylib
libboost_filesystem-mt.dylib          libboost_regex-mt.dylib               libboost_unit_test_framework-mt.dylib libpcre.1.dylib

Note how PyMyLib.cpython-39-darwin.so and __init__.py have been installed directly to PyMyLib/, not to PyMyLib-0.1.17.data/purelib/PyMyLib/ (as it looks in the wheel archive).

Consequently, I get import errors because the dynamic libraries are not found:

$ python -c 'import PyMyLib'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "...../.venv-devel-macos-arm64-tf26/lib/python3.9/site-packages/PyMyLib/__init__.py", line 1, in <module>
    from .PyMyLib import PyMyLib as MyLib
ImportError: dlopen(...../.venv-devel-macos-arm64-tf26/lib/python3.9/site-packages/PyMyLib/PyMyLib.cpython-39-darwin.so, 0x0002): Library not loaded: @loader_path/../../../PyMyLib.dylibs/libboost_date_time-mt.dylib
  Referenced from: ...../.venv-devel-macos-arm64-tf26/lib/python3.9/site-packages/PyMyLib/PyMyLib.cpython-39-darwin.so
  Reason: tried: '...../.venv-devel-macos-arm64-tf26/lib/python3.9/site-packages/PyMyLib/../../../PyMyLib.dylibs/libboost_date_time-mt.dylib' (no such file), '/usr/local/lib/libboost_date_time-mt.dylib' (no such file), '/usr/lib/libboost_date_time-mt.dylib' (no such file)

Expected behavior
delocate-wheel should set relative paths to dylibs in such a way that they agree with their ultimate location after pip install.

Wheels used
Unfortunately not open source. But the __init__.py contains just these lines:

from .PyMyLib import PyMyLib as MyLib

if __name__ == "__main__":
    pass

Platform (please complete the following information):

  • OS version: macOS 12.2.1 (Apple Silicon)
  • Delocate version: 0.10.2
$ pip --version
pip 22.0.4 from ...../.venv-devel-macos-arm64-tf26/lib/python3.9/site-packages/pip (python 3.9)

Additional context
I actually don't know why our (cmake-based) build process produces wheels with these subdirectories PyMyLib-0.1.17.data/purelib. Pointers that could help me simplify that would also be appreciated. 🙂
@planetmarshall
Copy link

I know this bug report is a bit old but came across this today whilst packaging an existing CMake project into a wheel. It hapens because delocate doesn't take account of the final locations of files installed into the "data" folder in the wheel, as these go directly into the root of the environment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@planetmarshall @maxhgerlach