BUG: pyzmq v26.0.0 installation fails on Bullseye (build from sources) #1976

AlvinSchiller · 2024-04-16T11:32:03Z

What pyzmq version?**

26.0.0

What libzmq version?**

4.3.5 (seperatly build and installed)

Python version (and how it was installed)

python 3.9 via apt-get

OS

Raspberry Pi OS Bullseye 32-bit / Github CI Docker Debian Bullseye

What happened?

In our projekt we build pyzmq from sources with a seperatly prebuild libzmq installation.
Prior to the v26.0.0 release this succeeded on Bullseye and Bookworm. Since the new version release this fails on Bullseye.

Traceback, if applicable

Collecting pyzmq
  Downloading pyzmq-26.0.0.tar.gz (266 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 266.6/266.6 kB 4.9 MB/s eta 0:00:00
  Installing build dependencies: started
  Running command pip subprocess to install build dependencies
  Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple, https://www.piwheels.org/simple
  Ignoring cffi: markers 'implementation_name == "pypy"' don't match your environment
  Collecting cython>=3.0.0
    Downloading https://www.piwheels.org/simple/cython/Cython-3.0.10-cp39-cp39-linux_armv7l.whl (10.7 MB)
       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.7/10.7 MB 3.7 MB/s eta 0:00:00
  Collecting packaging
    Downloading https://www.piwheels.org/simple/packaging/packaging-24.0-py3-none-any.whl (53 kB)
       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 53.5/53.5 kB 5.0 MB/s eta 0:00:00
  Collecting scikit-build-core
    Downloading https://www.piwheels.org/simple/scikit-build-core/scikit_build_core-0.8.2-py3-none-any.whl (140 kB)
       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 140.5/140.5 kB 8.8 MB/s eta 0:00:00
  Collecting exceptiongroup (from scikit-build-core)
    Downloading https://www.piwheels.org/simple/exceptiongroup/exceptiongroup-1.2.0-py3-none-any.whl (16 kB)
  Collecting tomli>=1.1 (from scikit-build-core)
    Downloading https://www.piwheels.org/simple/tomli/tomli-2.0.1-py3-none-any.whl (12 kB)
  Installing collected packages: tomli, packaging, exceptiongroup, cython, scikit-build-core
  Successfully installed cython-3.0.10 exceptiongroup-1.2.0 packaging-24.0 scikit-build-core-0.8.2 tomli-2.0.1
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Running command Getting requirements to build wheel
  Getting requirements to build wheel: finished with status 'done'
  Installing backend dependencies: started
  Running command pip subprocess to install backend dependencies
  Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple, https://www.piwheels.org/simple
  Collecting pyproject_metadata
    Downloading https://www.piwheels.org/simple/pyproject-metadata/pyproject_metadata-0.7.1-py3-none-any.whl (7.4 kB)
  Collecting pathspec
    Downloading https://www.piwheels.org/simple/pathspec/pathspec-0.12.1-py3-none-any.whl (31 kB)
  Collecting cmake>=3.14
    Downloading cmake-3.29.2.tar.gz (30 kB)
    Installing build dependencies: started
    Installing build dependencies: finished with status 'done'
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'done'
    Installing backend dependencies: started
    Installing backend dependencies: finished with status 'error'
    error: subprocess-exited-with-error

    × pip subprocess to install backend dependencies did not run successfully.
    │ exit code: 2
    ╰─> [55 lines of output]
        Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple, https://www.piwheels.org/simple, https://www.piwheels.org/simple
        Collecting ninja>=1.5
          Downloading https://www.piwheels.org/simple/ninja/ninja-1.11.1.1-cp39-cp39-linux_armv7l.whl (132 kB)
             ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 132.5/132.5 kB 406.0 kB/s eta 0:00:00
        Collecting cmake
          Using cached cmake-3.29.2.tar.gz (30 kB)
        ERROR: Exception:
        Traceback (most recent call last):
          File "/home/testuser/RPi-Jukebox-RFID/.venv/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 180, in exc_logging_wrapper
            status = run_func(*args)
          File "/home/testuser/RPi-Jukebox-RFID/.venv/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 245, in wrapper
            return func(self, options, args)
          File "/home/testuser/RPi-Jukebox-RFID/.venv/lib/python3.9/site-packages/pip/_internal/commands/install.py", line 377, in run
            requirement_set = resolver.resolve(
          File "/home/testuser/RPi-Jukebox-RFID/.venv/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 95, in resolve
            result = self._result = resolver.resolve(
          File "/home/testuser/RPi-Jukebox-RFID/.venv/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 546, in resolve
            state = resolution.resolve(requirements, max_rounds=max_rounds)
          File "/home/testuser/RPi-Jukebox-RFID/.venv/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 397, in resolve
            self._add_to_criteria(self.state.criteria, r, parent=None)
          File "/home/testuser/RPi-Jukebox-RFID/.venv/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 173, in _add_to_criteria
            if not criterion.candidates:
          File "/home/testuser/RPi-Jukebox-RFID/.venv/lib/python3.9/site-packages/pip/_vendor/resolvelib/structs.py", line 156, in __bool__
            return bool(self._sequence)
          File "/home/testuser/RPi-Jukebox-RFID/.venv/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 155, in __bool__
            return any(self)
          File "/home/testuser/RPi-Jukebox-RFID/.venv/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 143, in <genexpr>
            return (c for c in iterator if id(c) not in self._incompatible_ids)
          File "/home/testuser/RPi-Jukebox-RFID/.venv/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 47, in _iter_built
            candidate = func()
          File "/home/testuser/RPi-Jukebox-RFID/.venv/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 182, in _make_candidate_from_link
            base: Optional[BaseCandidate] = self._make_base_candidate_from_link(
          File "/home/testuser/RPi-Jukebox-RFID/.venv/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 228, in _make_base_candidate_from_link
            self._link_candidate_cache[link] = LinkCandidate(
          File "/home/testuser/RPi-Jukebox-RFID/.venv/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 290, in __init__
            super().__init__(
          File "/home/testuser/RPi-Jukebox-RFID/.venv/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 156, in __init__
            self.dist = self._prepare()
          File "/home/testuser/RPi-Jukebox-RFID/.venv/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 222, in _prepare
            dist = self._prepare_distribution()
          File "/home/testuser/RPi-Jukebox-RFID/.venv/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 301, in _prepare_distribution
            return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
          File "/home/testuser/RPi-Jukebox-RFID/.venv/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 525, in prepare_linked_requirement
            return self._prepare_linked_requirement(req, parallel_builds)
          File "/home/testuser/RPi-Jukebox-RFID/.venv/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 640, in _prepare_linked_requirement
            dist = _get_prepared_distribution(
          File "/home/testuser/RPi-Jukebox-RFID/.venv/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 70, in _get_prepared_distribution
            with build_tracker.track(req, tracker_id):
          File "/usr/lib/python3.9/contextlib.py", line 117, in __enter__
            return next(self.gen)
          File "/home/testuser/RPi-Jukebox-RFID/.venv/lib/python3.9/site-packages/pip/_internal/operations/build/build_tracker.py", line 137, in track
            self.add(req, tracker_id)
          File "/home/testuser/RPi-Jukebox-RFID/.venv/lib/python3.9/site-packages/pip/_internal/operations/build/build_tracker.py", line 103, in add
            raise LookupError(message)
        LookupError: https://files.pythonhosted.org/packages/80/bf/4f9a9f754507992be28b985d1e9b17f93a2271106b5916a212efe1d65205/cmake-3.29.2.tar.gz (from https://pypi.org/simple/cmake/) (requires-python:>=3.7) is already being built: cmake>=3.14 from https://files.pythonhosted.org/packages/80/bf/4f9a9f754507992be28b985d1e9b17f93a2271106b5916a212efe1d65205/cmake-3.29.2.tar.gz
        [end of output]

    note: This error originates from a subprocess, and is likely not a problem with pip.
  error: subprocess-exited-with-error

  × pip subprocess to install backend dependencies did not run successfully.
  │ exit code: 2
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
  error: subprocess-exited-with-error
  
  × pip subprocess to install backend dependencies did not run successfully.
  │ exit code: 1
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /home/testuser/RPi-Jukebox-RFID/.venv/bin/python3 /home/testuser/RPi-Jukebox-RFID/.venv/lib/python3.9/site-packages/pip/__pip-runner__.py install --ignore-installed --no-user --prefix /tmp/pip-build-env-vnwo_md6/normal --no-warn-script-location --no-binary pyzmq --only-binary :none: -i https://pypi.org/simple --extra-index-url https://www.piwheels.org/simple -- pyproject_metadata pathspec 'cmake>=3.14'
  cwd: [inherit]
  Installing backend dependencies: finished with status 'error'
error: subprocess-exited-with-error

× pip subprocess to install backend dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

More info

libzmq prebuild and copied to "/usr/local".
Build pyzmq
ZMQ_PREFIX="/usr/local"
ZMQ_DRAFT_API=1
pip install -v pyzmq --no-binary pyzmq

We had also problems with the prerelease some time ago: #1937

The text was updated successfully, but these errors were encountered:

minrk · 2024-04-16T11:58:52Z

building pyzmq 26 from source requires cmake. If you don't have cmake, there is a Python cmake package that will fetch cmake as a wheel, but if there isn't a wheel for your platform, you need cmake installed separately. It looks like you should be getting this one, I'm not sure how piwheels works, though.

This is the biggest downside of the new build system, and there's nothing really that can be done about it until someone can address scikit-build/cmake-python-distributions#33. If you have the ability to install cmake, that's what is required.

minrk · 2024-04-16T13:21:14Z

I believe #1977 fixed the issue preventing the piwheels builds of pyzmq 26, I'll do 26.0.1 with that tonight or tomorrow.

gg-mmill · 2024-04-19T09:53:32Z

Hello, there seem to be an issue, many builds are missing from https://pypi.org/project/pyzmq/26.0.1/#files , especially amd64 build, which were present in https://pypi.org/project/pyzmq/26.0.0/#files

minrk · 2024-04-19T14:53:25Z

The CI upload took a while today, should be all there now.

minrk · 2024-04-20T14:07:28Z

The 26.0.2 builds appeared to work fine on piwheels: https://www.piwheels.org/project/pyzmq/

AlvinSchiller · 2024-04-20T22:54:08Z

The problem on bullseye appears to be that cmake 3.29.x has no wheel package for python 3.9 for arm, so the installation fails. It works with cmake 3.28.x though.
https://www.piwheels.org/project/cmake/json/
Seems to be a problem on there side.

Opened scikit-build/cmake-python-distributions#494

AlvinSchiller · 2024-04-20T23:01:00Z

With 26.0.2 we have a new problem on Bookworm now.
The installation works, but a python call for a version check after fails, with the prebuild libzmq.
Using the bundled libzmq succeeds.

local zmq_version=$(python -c 'import zmq; print(f"{zmq.zmq_version()}")')

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/testuser/RPi-Jukebox-RFID/.venv/lib/python3.11/site-packages/zmq/__init__.py", line 52, in <module>
    from zmq import backend
  File "/home/testuser/RPi-Jukebox-RFID/.venv/lib/python3.11/site-packages/zmq/backend/__init__.py", line 30, in <module>
    raise original_error from None
  File "/home/testuser/RPi-Jukebox-RFID/.venv/lib/python3.11/site-packages/zmq/backend/__init__.py", line 25, in <module>
    _ns = select_backend(first)
          ^^^^^^^^^^^^^^^^^^^^^
  File "/home/testuser/RPi-Jukebox-RFID/.venv/lib/python3.11/site-packages/zmq/backend/select.py", line 31, in select_backend
    mod = import_module(name)
          ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/testuser/RPi-Jukebox-RFID/.venv/lib/python3.11/site-packages/zmq/backend/cython/__init__.py", line 6, in <module>
    from . import _zmq
ImportError: /home/testuser/RPi-Jukebox-RFID/.venv/lib/python3.11/site-packages/zmq/backend/cython/_zmq.cpython-311-arm-linux-gnueabihf.so: undefined symbol: zmq_msg_set_routing_id

minrk · 2024-04-21T17:32:44Z

Do you have a link to build commands, context, and output? This is what you would get e.g. if you built against libzmq with drafts but the at runtime found a different libzmq built without drafts. It might be missing rpath in your link flags.

AlvinSchiller · 2024-04-24T20:08:23Z

We build libzmq and pyzmq like this (as described here)

JUKEBOX_ZMQ_TMP_DIR="${HOME_PATH}/libzmq"
JUKEBOX_ZMQ_PREFIX="/usr/local"
JUKEBOX_ZMQ_VERSION="4.3.5"

_build_libzmq_pyzmq_with_drafts() {
  local zmq_filename="zeromq-${JUKEBOX_ZMQ_VERSION}"
  local zmq_tar_filename="${zmq_filename}.tar.gz"
  local cpu_count=${CPU_COUNT:-$(python3 -c "import os; print(os.cpu_count())")}

  cd "${JUKEBOX_ZMQ_TMP_DIR}"
  wget --quiet https://github.com/zeromq/libzmq/releases/download/v${JUKEBOX_ZMQ_VERSION}/${zmq_tar_filename}
  tar -xzf ${zmq_tar_filename}
  rm -f ${zmq_tar_filename}
  cd ${zmq_filename} 
  ./configure --prefix=${JUKEBOX_ZMQ_PREFIX} --enable-drafts --disable-Werror
  make -j${cpu_count} && sudo make install

  ZMQ_PREFIX="${JUKEBOX_ZMQ_PREFIX}" ZMQ_DRAFT_API=1 pip install -v pyzmq --no-binary pyzmq
}

Note: This problem existed most likely already with v26.0.0 but pops up only at runtime. We added this version check just recently, so it now pops up directly in the ci run.

minrk · 2024-04-25T15:00:30Z

Can you add sudo ldconfig between installing libzmq and building pyzmq? Not sure if it would help, but worth a try.

it would also help to have the full build output of pyzmq, just to see what libzmq it is finding, as well as ldd output for the .so file in pyzmq.

based on the output, it really seems like pyzmq is built with a libzmq with drafts, but a different libzmq is being found at runtime. Check if there is more than one on your system. It may be that you need to add rpath arguments if there’s more than one (ldconfig might also help with this) to ensure that the right one is loaded.

AlvinSchiller · 2024-04-26T12:06:22Z

Adding sudo ldconfig didn't make a difference.

Here is the build log for pyzmq from the ci

Buildlog pyzmq

Using pip 24.0 from /home/testuser/RPi-Jukebox-RFID/.venv/lib/python3.11/site-packages/pip (python 3.11)
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Collecting pyzmq
  Downloading pyzmq-26.0.2.tar.gz (266 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 266.7/266.7 kB 1.4 MB/s eta 0:00:00
  Installing build dependencies: started
  Running command pip subprocess to install build dependencies
  Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple, https://www.piwheels.org/simple
  Ignoring cffi: markers 'implementation_name == "pypy"' don't match your environment
  Collecting cython>=3.0.0
    Downloading https://www.piwheels.org/simple/cython/Cython-3.0.10-cp311-cp311-linux_armv7l.whl (11.8 MB)
       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.8/11.8 MB 13.8 MB/s eta 0:00:00
  Collecting packaging
    Downloading https://www.piwheels.org/simple/packaging/packaging-24.0-py3-none-any.whl (53 kB)
       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 53.5/53.5 kB 5.6 MB/s eta 0:00:00
  Collecting scikit-build-core
    Downloading https://www.piwheels.org/simple/scikit-build-core/scikit_build_core-0.9.2-py3-none-any.whl (151 kB)
       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 151.5/151.5 kB 10.2 MB/s eta 0:00:00
  Collecting pathspec>=0.10.1 (from scikit-build-core)
    Downloading https://www.piwheels.org/simple/pathspec/pathspec-0.12.1-py3-none-any.whl (31 kB)
  Installing collected packages: pathspec, packaging, cython, scikit-build-core
  Successfully installed cython-3.0.10 packaging-24.0 pathspec-0.12.1 scikit-build-core-0.9.2
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Running command Getting requirements to build wheel
  Getting requirements to build wheel: finished with status 'done'
  Installing backend dependencies: started
  Running command pip subprocess to install backend dependencies
  Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple, https://www.piwheels.org/simple
  Collecting cmake>=3.14
    Downloading https://www.piwheels.org/simple/cmake/cmake-3.29.2-py3-none-manylinux_2_36_armv7l.whl (19.9 MB)
       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 19.9/19.9 MB 14.2 MB/s eta 0:00:00
  Installing collected packages: cmake
  Successfully installed cmake-3.29.2
  Installing backend dependencies: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Running command Preparing metadata (pyproject.toml)
  *** scikit-build-core 0.9.2 using CMake 3.29.2 (metadata_wheel)
  Preparing metadata (pyproject.toml): finished with status 'done'
Building wheels for collected packages: pyzmq
  Building wheel for pyzmq (pyproject.toml): started
  Running command Building wheel for pyzmq (pyproject.toml)
  *** scikit-build-core 0.9.2 using CMake 3.29.2 (wheel)
  *** Configuring CMake...
  loading initial cache file /tmp/tmp1_jajl26/build/CMakeInit.txt
  -- The C compiler identification is GNU 12.2.0
  -- The CXX compiler identification is GNU 12.2.0
  -- Detecting C compiler ABI info
  -- Detecting C compiler ABI info - done
  -- Check for working C compiler: /usr/bin/cc - skipped
  -- Detecting C compile features
  -- Detecting C compile features - done
  -- Detecting CXX compiler ABI info
  -- Detecting CXX compiler ABI info - done
  -- Check for working CXX compiler: /usr/bin/c++ - skipped
  -- Detecting CXX compile features
  -- Detecting CXX compile features - done
  -- Found Python: /home/testuser/RPi-Jukebox-RFID/.venv/bin/python3 (found version "3.11.2") found components: Interpreter Development.Module
  -- enabling ZMQ_DRAFT_API
  -- Looking for libzmq in /usr/local
  -- Looking for libzmq in /usr/local - /usr/local/lib/libzmq.so
  -- Using Python Python /home/testuser/RPi-Jukebox-RFID/.venv/bin/python3
  -- Building Cython backend
  -- Linking libzmq target libzmq
  -- Configuring done (7.0s)
  -- Generating done (0.0s)
  -- Build files have been written to: /tmp/tmp1_jajl26/build
  *** Building project with Unix Makefiles...
  [ 33%] Generating _src/_zmq.c
  [ 66%] Building C object CMakeFiles/_zmq.dir/_src/_zmq.c.o
  [100%] Linking C shared module _zmq.cpython-311-arm-linux-gnueabihf.so
  [100%] Built target _zmq
  [100%] Built target pyzmq
  *** Installing project into wheel...
  -- Install configuration: "Release"
  -- Installing: /tmp/tmp1_jajl26/wheel/platlib/zmq/backend/cython/_zmq.cpython-311-arm-linux-gnueabihf.so
  -- Set non-toolchain portion of runtime path of "/tmp/tmp1_jajl26/wheel/platlib/zmq/backend/cython/_zmq.cpython-311-arm-linux-gnueabihf.so" to ""
  *** Making wheel...
  *** Created pyzmq-26.0.2-cp311-cp311-linux_armv7l.whl...
  Building wheel for pyzmq (pyproject.toml): finished with status 'done'
  Created wheel for pyzmq: filename=pyzmq-26.0.2-cp311-cp311-linux_armv7l.whl size=303439 sha256=5bf5c71db73dc26fd7d79213d6a57ad1735388ccff0dcd1259ae3b250a2ea862
  Stored in directory: /home/testuser/.cache/pip/wheels/3c/04/91/34f7182033fe51d08c78918f20e611b65920ffa91c4594a93e
Successfully built pyzmq
Installing collected packages: pyzmq
Successfully installed pyzmq-26.0.2

And the ldd output for /home/testuser/RPi-Jukebox-RFID/.venv/lib/python3.11/site-packages/zmq/backend/cython/_zmq.cpython-311-arm-linux-gnueabihf.so

Output ldd

 	libzmq.so.5 => /lib/arm-linux-gnueabihf/libzmq.so.5 (0x3ef45000)
	libc.so.6 => /lib/arm-linux-gnueabihf/libc.so.6 (0x3edcc000)
	libbsd.so.0 => /lib/arm-linux-gnueabihf/libbsd.so.0 (0x3edbe000)
	libsodium.so.23 => /lib/arm-linux-gnueabihf/libsodium.so.23 (0x3ed70000)
	libpgm-5.3.so.0 => /lib/arm-linux-gnueabihf/libpgm-5.3.so.0 (0x3ed10000)
	libnorm.so.1 => /lib/arm-linux-gnueabihf/libnorm.so.1 (0x3ec00000)
	libgssapi_krb5.so.2 => /lib/arm-linux-gnueabihf/libgssapi_krb5.so.2 (0x3ebd0000)
	libstdc++.so.6 => /lib/arm-linux-gnueabihf/libstdc++.so.6 (0x3ea00000)
	/lib/ld-linux-armhf.so.3 (0x40000000)
	libgcc_s.so.1 => /lib/arm-linux-gnueabihf/libgcc_s.so.1 (0x3e9c0000)
	libmd.so.0 => /lib/arm-linux-gnueabihf/libmd.so.0 (0x3e990000)
	libpthread.so.0 => /lib/arm-linux-gnueabihf/libpthread.so.0 (0x3e98b000)
	libm.so.6 => /lib/arm-linux-gnueabihf/libm.so.6 (0x3e944000)
	libkrb5.so.3 => /lib/arm-linux-gnueabihf/libkrb5.so.3 (0x3e8b7000)
	libk5crypto.so.3 => /lib/arm-linux-gnueabihf/libk5crypto.so.3 (0x3e895000)
	libcom_err.so.2 => /lib/arm-linux-gnueabihf/libcom_err.so.2 (0x3e891000)
	libkrb5support.so.0 => /lib/arm-linux-gnueabihf/libkrb5support.so.0 (0x3e888000)
	libkeyutils.so.1 => /lib/arm-linux-gnueabihf/libkeyutils.so.1 (0x3e860000)
	libresolv.so.2 => /lib/arm-linux-gnueabihf/libresolv.so.2 (0x3e851000)

I added some output and apparently the system has libzmq already installed in a different version, which would support your statement that the wrong lib is linked.

$apt list --installed | grep "zmq"
   libzmq5/stable,now 4.3.4-6 armhf [installed,automatic]

AlvinSchiller · 2024-04-26T12:41:50Z

If the installation is run on a real pi, no libzmq version is installed by default, so the output is like this instead:

$ python -c 'import zmq; print(f"{zmq.zmq_version()}")'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/pi/testenv/lib/python3.11/site-packages/zmq/__init__.py", line 52, in <module>
    from zmq import backend
  File "/home/pi/testenv/lib/python3.11/site-packages/zmq/backend/__init__.py", line 30, in <module>
    raise original_error from None
  File "/home/pi/testenv/lib/python3.11/site-packages/zmq/backend/__init__.py", line 25, in <module>
    _ns = select_backend(first)
          ^^^^^^^^^^^^^^^^^^^^^
  File "/home/pi/testenv/lib/python3.11/site-packages/zmq/backend/select.py", line 31, in select_backend
    mod = import_module(name)
          ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pi/testenv/lib/python3.11/site-packages/zmq/backend/cython/__init__.py", line 6, in <module>
    from . import _zmq
ImportError: libzmq.so.5: cannot open shared object file: No such file or directory

$ ldd testenv/lib/python3.11/site-packages/zmq/backend/cython/_zmq.cpython-311-arm-linux-gnueabihf.so
        linux-vdso.so.1 (0x7ef38000)
        /usr/lib/arm-linux-gnueabihf/libarmmem-${PLATFORM}.so => /usr/lib/arm-linux-gnueabihf/libarmmem-v7l.so (0x76ea0000)
        libzmq.so.5 => not found
        libc.so.6 => /lib/arm-linux-gnueabihf/libc.so.6 (0x76d20000)
        /lib/ld-linux-armhf.so.3 (0x76efb000)

Here sudo ldconfig seems to fix the situation, if a prebuild libzmq is used.

minrk · 2024-04-26T12:50:22Z

Can you add

export LDFLAGS="-Wl,-rpath,/usr/local/lib"
export SKBUILD_CMAKE_VERBOSE=true

and share the output again?

EDIT: added missing /lib on rpath

AlvinSchiller · 2024-04-26T13:38:38Z

Can you add
export LDFLAGS="-Wl,-rpath,/usr/local"
export SKBUILD_CMAKE_VERBOSE=true
and share the output again?

Buildlog pyzmq

Using pip 24.0 from /home/testuser/RPi-Jukebox-RFID/.venv/lib/python3.11/site-packages/pip (python 3.11)
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Collecting pyzmq
  Downloading pyzmq-26.0.2.tar.gz (266 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 266.7/266.7 kB 944.0 kB/s eta 0:00:00
  Installing build dependencies: started
  Running command pip subprocess to install build dependencies
  Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple, https://www.piwheels.org/simple
  Ignoring cffi: markers 'implementation_name == "pypy"' don't match your environment
  Collecting cython>=3.0.0
    Downloading https://www.piwheels.org/simple/cython/Cython-3.0.10-cp311-cp311-linux_armv7l.whl (11.8 MB)
       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.8/11.8 MB 13.6 MB/s eta 0:00:00
  Collecting packaging
    Downloading https://www.piwheels.org/simple/packaging/packaging-24.0-py3-none-any.whl (53 kB)
       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 53.5/53.5 kB 5.0 MB/s eta 0:00:00
  Collecting scikit-build-core
    Downloading https://www.piwheels.org/simple/scikit-build-core/scikit_build_core-0.9.2-py3-none-any.whl (151 kB)
       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 151.5/151.5 kB 9.4 MB/s eta 0:00:00
  Collecting pathspec>=0.10.1 (from scikit-build-core)
    Downloading https://www.piwheels.org/simple/pathspec/pathspec-0.12.1-py3-none-any.whl (31 kB)
  Installing collected packages: pathspec, packaging, cython, scikit-build-core
  Successfully installed cython-3.0.10 packaging-24.0 pathspec-0.12.1 scikit-build-core-0.9.2
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Running command Getting requirements to build wheel
  Getting requirements to build wheel: finished with status 'done'
  Installing backend dependencies: started
  Running command pip subprocess to install backend dependencies
  Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple, https://www.piwheels.org/simple
  Collecting cmake>=3.14
    Downloading https://www.piwheels.org/simple/cmake/cmake-3.29.2-py3-none-manylinux_2_36_armv7l.whl (19.9 MB)
       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 19.9/19.9 MB 14.4 MB/s eta 0:00:00
  Installing collected packages: cmake
  Successfully installed cmake-3.29.2
  Installing backend dependencies: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Running command Preparing metadata (pyproject.toml)
  *** scikit-build-core 0.9.2 using CMake 3.29.2 (metadata_wheel)
  Preparing metadata (pyproject.toml): finished with status 'done'
Building wheels for collected packages: pyzmq
  Building wheel for pyzmq (pyproject.toml): started
  Running command Building wheel for pyzmq (pyproject.toml)
  *** scikit-build-core 0.9.2 using CMake 3.29.2 (wheel)
  *** Configuring CMake...
  loading initial cache file /tmp/tmpfbgp3nxd/build/CMakeInit.txt
  -- The C compiler identification is GNU 12.2.0
  -- The CXX compiler identification is GNU 12.2.0
  -- Detecting C compiler ABI info
  -- Detecting C compiler ABI info - done
  -- Check for working C compiler: /usr/bin/cc - skipped
  -- Detecting C compile features
  -- Detecting C compile features - done
  -- Detecting CXX compiler ABI info
  -- Detecting CXX compiler ABI info - done
  -- Check for working CXX compiler: /usr/bin/c++ - skipped
  -- Detecting CXX compile features
  -- Detecting CXX compile features - done
  -- Found Python: /home/testuser/RPi-Jukebox-RFID/.venv/bin/python3 (found version "3.11.2") found components: Interpreter Development.Module
  -- enabling ZMQ_DRAFT_API
  -- Looking for libzmq in /usr/local
  -- Looking for libzmq in /usr/local - /usr/local/lib/libzmq.so
  -- Using Python Python /home/testuser/RPi-Jukebox-RFID/.venv/bin/python3
  -- Building Cython backend
  -- Linking libzmq target libzmq
  -- Configuring done (7.0s)
  -- Generating done (0.0s)
  -- Build files have been written to: /tmp/tmpfbgp3nxd/build
  *** Building project with Unix Makefiles...
  Change Dir: '/tmp/tmpfbgp3nxd/build'

  Run Build Command(s): /tmp/pip-build-env-bjo2rz6s/normal/lib/python3.11/site-packages/cmake/data/bin/cmake -E env VERBOSE=1 /usr/bin/gmake -f Makefile pyzmq
  /tmp/pip-build-env-bjo2rz6s/normal/lib/python3.11/site-packages/cmake/data/bin/cmake -S/tmp/pip-install-q2v6cu4p/pyzmq_6be7afd26bcb4d5685c99760702f234f -B/tmp/tmpfbgp3nxd/build --check-build-system CMakeFiles/Makefile.cmake 0
  /usr/bin/gmake  -f CMakeFiles/Makefile2 pyzmq
  gmake[1]: Entering directory '/tmp/tmpfbgp3nxd/build'
  /tmp/pip-build-env-bjo2rz6s/normal/lib/python3.11/site-packages/cmake/data/bin/cmake -S/tmp/pip-install-q2v6cu4p/pyzmq_6be7afd26bcb4d5685c99760702f234f -B/tmp/tmpfbgp3nxd/build --check-build-system CMakeFiles/Makefile.cmake 0
  /tmp/pip-build-env-bjo2rz6s/normal/lib/python3.11/site-packages/cmake/data/bin/cmake -E cmake_progress_start /tmp/tmpfbgp3nxd/build/CMakeFiles 3
  /usr/bin/gmake  -f CMakeFiles/Makefile2 CMakeFiles/pyzmq.dir/all
  gmake[2]: Entering directory '/tmp/tmpfbgp3nxd/build'
  /usr/bin/gmake  -f CMakeFiles/_zmq.dir/build.make CMakeFiles/_zmq.dir/depend
  gmake[3]: Entering directory '/tmp/tmpfbgp3nxd/build'
  [ 33%] Generating _src/_zmq.c
  /home/testuser/RPi-Jukebox-RFID/.venv/bin/python3 -mcython --3str --output-file /tmp/tmpfbgp3nxd/build/_src/_zmq.c --module-name zmq.backend.cython._zmq /tmp/pip-install-q2v6cu4p/pyzmq_6be7afd26bcb4d5685c99760702f234f/zmq/backend/cython/_zmq.py
  cd /tmp/tmpfbgp3nxd/build && /tmp/pip-build-env-bjo2rz6s/normal/lib/python3.11/site-packages/cmake/data/bin/cmake -E cmake_depends "Unix Makefiles" /tmp/pip-install-q2v6cu4p/pyzmq_6be7afd26bcb4d5685c99760702f234f /tmp/pip-install-q2v6cu4p/pyzmq_6be7afd26bcb4d5685c99760702f234f /tmp/tmpfbgp3nxd/build /tmp/tmpfbgp3nxd/build /tmp/tmpfbgp3nxd/build/CMakeFiles/_zmq.dir/DependInfo.cmake "--color="
  gmake[3]: Leaving directory '/tmp/tmpfbgp3nxd/build'
  /usr/bin/gmake  -f CMakeFiles/_zmq.dir/build.make CMakeFiles/_zmq.dir/build
  gmake[3]: Entering directory '/tmp/tmpfbgp3nxd/build'
  [ 66%] Building C object CMakeFiles/_zmq.dir/_src/_zmq.c.o
  /usr/bin/cc -DZMQ_BUILD_DRAFT_API=1 -D_zmq_EXPORTS -I/tmp/pip-install-q2v6cu4p/pyzmq_6be7afd26bcb4d5685c99760702f234f/zmq/utils -isystem /usr/include/python3.11 -O3 -DNDEBUG -fPIC -MD -MT CMakeFiles/_zmq.dir/_src/_zmq.c.o -MF CMakeFiles/_zmq.dir/_src/_zmq.c.o.d -o CMakeFiles/_zmq.dir/_src/_zmq.c.o -c /tmp/tmpfbgp3nxd/build/_src/_zmq.c
  [100%] Linking C shared module _zmq.cpython-311-arm-linux-gnueabihf.so
  /tmp/pip-build-env-bjo2rz6s/normal/lib/python3.11/site-packages/cmake/data/bin/cmake -E cmake_link_script CMakeFiles/_zmq.dir/link.txt --verbose=1
  /usr/bin/cc -fPIC -O3 -DNDEBUG -Wl,-rpath,/usr/local -shared  -o _zmq.cpython-311-arm-linux-gnueabihf.so CMakeFiles/_zmq.dir/_src/_zmq.c.o  -Wl,-rpath,/usr/local/lib: /usr/local/lib/libzmq.so
  gmake[3]: Leaving directory '/tmp/tmpfbgp3nxd/build'
  [100%] Built target _zmq
  /usr/bin/gmake  -f CMakeFiles/pyzmq.dir/build.make CMakeFiles/pyzmq.dir/depend
  gmake[3]: Entering directory '/tmp/tmpfbgp3nxd/build'
  cd /tmp/tmpfbgp3nxd/build && /tmp/pip-build-env-bjo2rz6s/normal/lib/python3.11/site-packages/cmake/data/bin/cmake -E cmake_depends "Unix Makefiles" /tmp/pip-install-q2v6cu4p/pyzmq_6be7afd26bcb4d5685c99760702f234f /tmp/pip-install-q2v6cu4p/pyzmq_6be7afd26bcb4d5685c99760702f234f /tmp/tmpfbgp3nxd/build /tmp/tmpfbgp3nxd/build /tmp/tmpfbgp3nxd/build/CMakeFiles/pyzmq.dir/DependInfo.cmake "--color="
  gmake[3]: Leaving directory '/tmp/tmpfbgp3nxd/build'
  /usr/bin/gmake  -f CMakeFiles/pyzmq.dir/build.make CMakeFiles/pyzmq.dir/build
  gmake[3]: Entering directory '/tmp/tmpfbgp3nxd/build'
  gmake[3]: Nothing to be done for 'CMakeFiles/pyzmq.dir/build'.
  gmake[3]: Leaving directory '/tmp/tmpfbgp3nxd/build'
  [100%] Built target pyzmq
  gmake[2]: Leaving directory '/tmp/tmpfbgp3nxd/build'
  /tmp/pip-build-env-bjo2rz6s/normal/lib/python3.11/site-packages/cmake/data/bin/cmake -E cmake_progress_start /tmp/tmpfbgp3nxd/build/CMakeFiles 0
  gmake[1]: Leaving directory '/tmp/tmpfbgp3nxd/build'

  *** Installing project into wheel...
  -- Install configuration: "Release"
  -- Installing: /tmp/tmpfbgp3nxd/wheel/platlib/zmq/backend/cython/_zmq.cpython-311-arm-linux-gnueabihf.so
  -- Set non-toolchain portion of runtime path of "/tmp/tmpfbgp3nxd/wheel/platlib/zmq/backend/cython/_zmq.cpython-311-arm-linux-gnueabihf.so" to ""
  *** Making wheel...
  *** Created pyzmq-26.0.2-cp311-cp311-linux_armv7l.whl...
  Building wheel for pyzmq (pyproject.toml): finished with status 'done'
  Created wheel for pyzmq: filename=pyzmq-26.0.2-cp311-cp311-linux_armv7l.whl size=303440 sha256=3a4b5c2a29169aabd0cc673ab5ab33a5376a155670a46c45395121cf37f1a5d5
  Stored in directory: /home/testuser/.cache/pip/wheels/3c/04/91/34f7182033fe51d08c78918f20e611b65920ffa91c4594a93e
Successfully built pyzmq
Installing collected packages: pyzmq
Successfully installed pyzmq-26.0.2

minrk · 2024-04-26T14:27:22Z

and is the result still the same for ldd ldd testenv/lib/python3.11/site-packages/zmq/backend/cython/_zmq.*.so?

AlvinSchiller · 2024-04-26T15:06:26Z

Ah sry, yes looks the same

Output ldd

	libzmq.so.5 => /lib/arm-linux-gnueabihf/libzmq.so.5 (0x3ef45000) 
	libc.so.6 => /lib/arm-linux-gnueabihf/libc.so.6 (0x3edcc000) 
	libbsd.so.0 => /lib/arm-linux-gnueabihf/libbsd.so.0 (0x3edbe000) 
	libsodium.so.23 => /lib/arm-linux-gnueabihf/libsodium.so.23 (0x3ed70000) 
	libpgm-5.3.so.0 => /lib/arm-linux-gnueabihf/libpgm-5.3.so.0 (0x3ed10000) 
	libnorm.so.1 => /lib/arm-linux-gnueabihf/libnorm.so.1 (0x3ec00000) 
	libgssapi_krb5.so.2 => /lib/arm-linux-gnueabihf/libgssapi_krb5.so.2 (0x3ebd0000) 
	libstdc++.so.6 => /lib/arm-linux-gnueabihf/libstdc++.so.6 (0x3ea00000) 
	/lib/ld-linux-armhf.so.3 (0x40000000) 
	libgcc_s.so.1 => /lib/arm-linux-gnueabihf/libgcc_s.so.1 (0x3e9c0000) 
	libmd.so.0 => /lib/arm-linux-gnueabihf/libmd.so.0 (0x3e990000) 
	libpthread.so.0 => /lib/arm-linux-gnueabihf/libpthread.so.0 (0x3e98b000) 
	libm.so.6 => /lib/arm-linux-gnueabihf/libm.so.6 (0x3e944000) 
	libkrb5.so.3 => /lib/arm-linux-gnueabihf/libkrb5.so.3 (0x3e8b7000) 
	libk5crypto.so.3 => /lib/arm-linux-gnueabihf/libk5crypto.so.3 (0x3e895000) 
	libcom_err.so.2 => /lib/arm-linux-gnueabihf/libcom_err.so.2 (0x3e891000) 
	libkrb5support.so.0 => /lib/arm-linux-gnueabihf/libkrb5support.so.0 (0x3e888000) 
	libkeyutils.so.1 => /lib/arm-linux-gnueabihf/libkeyutils.so.1 (0x3e860000) 
	libresolv.so.2 => /lib/arm-linux-gnueabihf/libresolv.so.2 (0x3e851000)

AlvinSchiller · 2024-04-26T23:13:19Z

I have created a testrepo running the build commands in an action, to better test this without clutter.
@minrk I have invited you to collaborate, so you can see the build logs, and feel free to test on new branches.

https://github.com/AlvinSchiller/TestPyzmqBuild

minrk · 2024-04-28T07:23:04Z

Thanks! Looks like the rpath and ldconfig does work, as seen here, and since you are using /usr/local (on rpath by default), just ldconfig is enough, as seen here.

Or is there something I'm missing that's still not working?

AlvinSchiller · 2024-04-28T23:31:41Z

Here are the things i observed:

Why is ldconfig needed after all? From the log output of the libzmq build, it looks like this is already done?
But i am not familiar with this build processes or this commands.

...
libtool: finish: PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin:/sbin" ldconfig -n /usr/local/lib
...

Adding the ldconfig call works in this test repo, but not in the original. I have compared the output of ldconfig and it looks like that the linking is done with the first appearence in the paths. This would explain the different outcomes in different systems.

Test repo run

`ldconfig -v` output test repo

    /sbin/ldconfig.real: Can't stat /usr/local/lib/x86_64-linux-gnu: No such file or directory
    /sbin/ldconfig.real: Path `/usr/lib/x86_64-linux-gnu' given more than once
    (from /etc/ld.so.conf.d/x86_64-linux-gnu.conf:4 and /etc/ld.so.conf.d/x86_64-linux-gnu.conf:3)
    /sbin/ldconfig.real: Path `/usr/lib32' given more than once
    (from /etc/ld.so.conf.d/zz_i386-biarch-compat.conf:3 and /etc/ld.so.conf.d/zz_i386-biarch-compat.conf:2)
    /sbin/ldconfig.real: Path `/lib/x86_64-linux-gnu' given more than once
    (from <builtin>:0 and /etc/ld.so.conf.d/x86_64-linux-gnu.conf:3)
    /sbin/ldconfig.real: Path `/usr/lib/x86_64-linux-gnu' given more than once
    (from <builtin>:0 and /etc/ld.so.conf.d/x86_64-linux-gnu.conf:3)
    /sbin/ldconfig.real: Path `/usr/lib' given more than once
    (from <builtin>:0 and <builtin>:0)
    /sbin/ldconfig.real: /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 is the dynamic linker, ignoring
    /usr/lib/x86_64-linux-gnu/libfakeroot: (from /etc/ld.so.conf.d/fakeroot-x86_64-linux-gnu.conf:1)

        libfakeroot-0.so -> libfakeroot-tcp.so
    /usr/local/lib: (from /etc/ld.so.conf.d/libc.conf:2)
>		libzmq.so.5 -> libzmq.so.5.2.5
    /sbin/ldconfig.real: /lib32/ld-linux.so.2 is the dynamic linker, ignoring

    /lib/x86_64-linux-gnu: (from /etc/ld.so.conf.d/x86_64-linux-gnu.conf:3)
        ...
>		libzmq.so.5 -> libzmq.so.5.2.4
        ...
    /lib32: (from /etc/ld.so.conf.d/zz_i386-biarch-compat.conf:2)
        ...
    /lib: (from <builtin>:0)
        ...

Original repo run

`ldconfig -v` output original repo

    ldconfig: Can't stat /usr/local/lib/arm-linux-gnueabihf: No such file or directory
    ldconfig: Path `/usr/lib/arm-linux-gnueabihf' given more than once
    (from /etc/ld.so.conf.d/arm-linux-gnueabihf.conf:4 and /etc/ld.so.conf.d/arm-linux-gnueabihf.conf:3)
    ldconfig: Path `/lib/arm-linux-gnueabihf' given more than once
    (from <builtin>:0 and /etc/ld.so.conf.d/arm-linux-gnueabihf.conf:3)
    ldconfig: Path `/usr/lib/arm-linux-gnueabihf' given more than once
    (from <builtin>:0 and /etc/ld.so.conf.d/arm-linux-gnueabihf.conf:3)
    ldconfig: Path `/usr/lib' given more than once
    (from <builtin>:0 and <builtin>:0)
    /lib/arm-linux-gnueabihf: (from /etc/ld.so.conf.d/arm-linux-gnueabihf.conf:3)
        ...
>		libzmq.so.5 -> libzmq.so.5.2.4
        ...
    ldconfig: /lib/arm-linux-gnueabihf/ld-linux-armhf.so.3 is the dynamic linker, ignoring

        ld-linux-armhf.so.3 -> ld-linux-armhf.so.3
    /usr/lib/arm-linux-gnueabihf/libfakeroot: (from /etc/ld.so.conf.d/fakeroot-arm-linux-gnueabihf.conf:1)
        libfakeroot-0.so -> libfakeroot-tcp.so
    /usr/local/lib: (from /etc/ld.so.conf.d/libc.conf:2)
>		libzmq.so.5 -> libzmq.so.5.2.5
    /lib: (from <builtin>:0)
    /lib/arm-linux-gnueabihf/neon: (hwcap: 0x0000000000001000) (from /etc/ld.so.conf.d/arm-linux-gnueabihf.conf:3)
    /lib/arm-linux-gnueabihf/vfp: (hwcap: 0x0000000000000040) (from /etc/ld.so.conf.d/arm-linux-gnueabihf.conf:3)
    /lib/arm-linux-gnueabihf/neon/vfp: (hwcap: 0x0000000000001040) (from /etc/ld.so.conf.d/arm-linux-gnueabihf.conf:3)
        ...
    /lib/arm-linux-gnueabihf/vfp/neon: (hwcap: 0x0000000000001040) (from /etc/ld.so.conf.d/arm-linux-gnueabihf.conf:3)
        libvpx.so.7 -> libvpx.so.7.1.0

(I have removed the vast majority of other lib to make it readable and marked the `libmzq` entries to show what i mean.)

EDIT: I can reproduce this manually on a Raspberry Pi 2 W with Raspi OS Bookworm lite 32-bit installed.
I have build libzmq, executed the ldconfig command and build pyzmq like described above.
If libzmq5 is not installed via apt, the command python -c 'import zmq; print(f"{zmq.zmq_version()}")' succeeds.
If libzmq5 is installed via apt (even afterwards), the command fails.

Using a "custom" prefix does not work, see the branch test/prefix in the testrepo. Thought the build finds the correct location for libzmq, the linking seems to use another logic for the path and doesn't respect the given prefix.

        ...
        -- Looking for libzmq in /etc/libzmq
>       -- Looking for libzmq in /etc/libzmq - /etc/libzmq/lib/libzmq.so
        -- Using Python Python /opt/hostedtoolcache/Python/3.11.9/x64/bin/python
        -- Building Cython backend
>       -- Linking libzmq target libzmq
        ...

minrk · 2024-04-29T09:27:42Z

Why is ldconfig needed after all?

See this answer.

This is unrelated to pyzmq, but ld itself. ld has a cache, so you may need to run ldconfig to refresh its cache after installing a new library, especially when you already have another version of the same library at a lower priority on ld's path.

Using a "custom" prefix does not work, see the branch test/prefix in the testrepo.

"Custom" prefix usually needs to be combined with setting the runtime path (rpath), e.g -Wl,-rpath,$ZMQ_PREFIX/lib, as demonstrated here, and shown to be working here.

I made a mistake in my suggestion where I specified the prefix (-Wl,-rpath,$ZMQ_PREFIX), which should have been the library dir (-Wl,-rpath,$ZMQ_PREFIX/lib).

It's arguable whether this should be done by pyzmq by default or not, because it doesn't seem to be standard practice for other packages to do this, but I did used to do it in setup.py, so I've put it back in #1983.

AlvinSchiller · 2024-04-29T11:12:03Z

Thanks a lot for the help and clearing this out!

Using the changed prefix path -Wl,-rpath,${ZMQ_PREFIX}/lib finally fixed the situation on bookworm!
I wasn't aware of this commands as i have only little knowledge with building and honestly just followed the documentation on how to build pyzmq with libzmq draft support.

For me personally it would just be fine to add this to the documentation, so c&p gets it working. :)

AlvinSchiller · 2024-04-29T11:40:45Z

The original issue still persists, but is not caused from this project (see comment), so closing this.

Thanks again!

AlvinSchiller mentioned this issue Apr 16, 2024

🐛 | Failed to install 3.5.2 due to pyzmq issue MiczFlor/RPi-Jukebox-RFID#2344

Closed

minrk mentioned this issue Apr 29, 2024

add ZMQ_PREFIX to $RPATH by default #1983

Draft

AlvinSchiller closed this as completed Apr 29, 2024

minrk mentioned this issue May 1, 2024

mention rpath, ldconfig in build docs #1985

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG: pyzmq v26.0.0 installation fails on Bullseye (build from sources) #1976

BUG: pyzmq v26.0.0 installation fails on Bullseye (build from sources) #1976

AlvinSchiller commented Apr 16, 2024 •

edited

minrk commented Apr 16, 2024

minrk commented Apr 16, 2024

gg-mmill commented Apr 19, 2024

minrk commented Apr 19, 2024

minrk commented Apr 20, 2024

AlvinSchiller commented Apr 20, 2024 •

edited

AlvinSchiller commented Apr 20, 2024 •

edited

minrk commented Apr 21, 2024

AlvinSchiller commented Apr 24, 2024

minrk commented Apr 25, 2024

AlvinSchiller commented Apr 26, 2024

AlvinSchiller commented Apr 26, 2024 •

edited

minrk commented Apr 26, 2024 •

edited

AlvinSchiller commented Apr 26, 2024

minrk commented Apr 26, 2024

AlvinSchiller commented Apr 26, 2024 •

edited

AlvinSchiller commented Apr 26, 2024 •

edited

minrk commented Apr 28, 2024

AlvinSchiller commented Apr 28, 2024 •

edited

minrk commented Apr 29, 2024 •

edited

AlvinSchiller commented Apr 29, 2024

AlvinSchiller commented Apr 29, 2024

BUG: pyzmq v26.0.0 installation fails on Bullseye (build from sources) #1976

BUG: pyzmq v26.0.0 installation fails on Bullseye (build from sources) #1976

Comments

AlvinSchiller commented Apr 16, 2024 • edited

What pyzmq version?**

What libzmq version?**

Python version (and how it was installed)

OS

What happened?

Traceback, if applicable

More info

minrk commented Apr 16, 2024

minrk commented Apr 16, 2024

gg-mmill commented Apr 19, 2024

minrk commented Apr 19, 2024

minrk commented Apr 20, 2024

AlvinSchiller commented Apr 20, 2024 • edited

AlvinSchiller commented Apr 20, 2024 • edited

minrk commented Apr 21, 2024

AlvinSchiller commented Apr 24, 2024

minrk commented Apr 25, 2024

AlvinSchiller commented Apr 26, 2024

AlvinSchiller commented Apr 26, 2024 • edited

minrk commented Apr 26, 2024 • edited

AlvinSchiller commented Apr 26, 2024

minrk commented Apr 26, 2024

AlvinSchiller commented Apr 26, 2024 • edited

AlvinSchiller commented Apr 26, 2024 • edited

minrk commented Apr 28, 2024

AlvinSchiller commented Apr 28, 2024 • edited

minrk commented Apr 29, 2024 • edited

AlvinSchiller commented Apr 29, 2024

AlvinSchiller commented Apr 29, 2024

AlvinSchiller commented Apr 16, 2024 •

edited

AlvinSchiller commented Apr 20, 2024 •

edited

AlvinSchiller commented Apr 20, 2024 •

edited

AlvinSchiller commented Apr 26, 2024 •

edited

minrk commented Apr 26, 2024 •

edited

AlvinSchiller commented Apr 26, 2024 •

edited

AlvinSchiller commented Apr 26, 2024 •

edited

AlvinSchiller commented Apr 28, 2024 •

edited

minrk commented Apr 29, 2024 •

edited