patchelf 0.17.0 can lead to broken repairs

[behrisch]

After updating to the specified tag several of our applications started to segfault directly on startup, see eclipse-sumo/sumo#12142. Going back to 2022-11-14-1226cfc solved the problem. The given library is the only one that changed.
To reproduce you can compare running marouter after doing pip install -i https://test.pypi.org/simple eclipse-sumo==1.15.0.post434 (last working version before the change) with the versions post450 (first broken) and post616 (fixed).

[mayeut]

Thanks for the report.
Do you have an un-repaired wheel somewhere ? This is taking taking ages to build on my laptop and a simple usage of libcrypt.so.1 is not showing a segfault (ported a test from auditwheel, using libcrypt.so.1 instead of libcrypt.so.2).

[behrisch]

pip install -i https://test.pypi.org/simple eclipse-sumo==1.15.0.post450 should give you a broken wheel

[mayeut]

yes but it's been processed by auditwheel. I need the wheel before it's been repaired to investigate. It's been building for more than 4 hours now (I just cloned the project & started ./tools/build/build_wheels.sh).

PS: I could reproduce the behavior mentioned in eclipse-sumo/sumo#12142 with the link provided so thanks for that but that's not enough to investigate.

[behrisch]

So you need the wheel before auditwheel runs, correct? I do not have that but I could set it up. But it will take probably an hour to build.

[mayeut]

So you need the wheel before auditwheel runs, correct?

Exactly. I'd like to see if it comes from the libxcrypt update or the patchelf update. Having the wheel before auditwheel runs is the easiest path for that investigation.

[behrisch]

The build is here: https://github.com/behrisch/sumo/actions/runs/3561204237 and it should be in the raw wheels artifact

[mayeut]

@behrisch,

Thanks for the raw wheels.
I could verify that the patchelf update is the culprit for your builds.
Using the following bisect script:

cat ../bisect.sh
#!/bin/bash

set -euxo pipefail

cd $(dirname ${BASH_SOURCE[0]})

pushd patchelf
./bootstrap.sh || exit 125
./configure || exit 125
make || exit 125
make install
make clean
popd
patchelf --version

rm -rf wheelhouse || true
./cp38/bin/pip uninstall -y eclipse_sumo || true
auditwheel repair eclipse_sumo-0.0.1-cp38-cp38-linux_x86_64.whl
./cp38/bin/pip install wheelhouse/eclipse_sumo-0.0.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 
./cp38/bin/marouter --version || exit 1

lead to:

42394e880bc5524122234fe2c2eaa043063ac581 is the first bad commit
commit 42394e880bc5524122234fe2c2eaa043063ac581
Author: Jörg Thalheim <joerg@thalheim.io>
Date:   Sat Nov 5 09:08:17 2022 -0400

    write out replace sections in original order
    
    Libc and other programs sometimes make assumption in which order
    sections.
    
    i.e. glibc expects that the strtab is after the symtab section: https://github.com/bminor/glibc/blob/9cc9d61ee12f2f8620d8e0ea3c42af02bf07fe1e/elf/dl-fptr.c#L179
    
    To decrease the likelyhood of breakages we keep the relative order the
    same when replacing section.

 src/patchelf.cc | 22 ++++++++++++++--------
 1 file changed, 14 insertions(+), 8 deletions(-)
bisect found first bad commit

A workaround for your builds would be to pin patchelf for now using pipx install -f patchelf==0.16.1.0
It still requires some work to report the breakage with more info upstream.

[mayeut]

It seems that only the marouter executable is broken (for this specific test). Using all libraries from patchelf 0.16.1 or 0.17.0 with an 0.16.1 repaired marouter executable works. Using all libraries from patchelf 0.16.1 or 0.17.0 with an 0.17.0 repaired marouter executable doesn't work.

[behrisch]

Thanks a lot! I used your workaround and it seems to work just fine now. Also for the other executables which failed before. I am doing more extensive tests now.

[behrisch]

Our tests are running fine now. Anything I can do to help with the bugreport for upstream?

[mayeut]

@behrisch, it might be related to NixOS/patchelf#446 so checking if NixOS/patchelf#447 solves your issue might be worth a try (& commenting there if that's the case).
Given the bisect returns another commit here than the one mentioned in NixOS/patchelf#447, that might not work in your case in which case opening an issue there with the info found here is probably a good starting point even though the reproducer is not that simple.

[pramodk]

Hello @behrisch @mayeut!

We are having the same issue while building wheels for the NEURON project using latest manylinux2014_x86_64 images (containing patchelf >= 0.17.0). Similar to #1421 (comment), our binary also fails starting from the commit 42394e880bc5524122234fe2c2eaa043063ac581 (i.e. NixOS/patchelf/pull/430).

Given the bisect returns another commit here than the one mentioned in NixOS/patchelf#447, that might not work in your case in which case opening an issue there with the info found here is probably a good starting point even though the reproducer is not that simple.

I tried 0.17.2 containing NixOS/patchelf/pull/447 but that didn't help us. I have provided details in the NixOS/patchelf#446 (comment).

[rdb]

We also hit this on the Panda3D project; of the version 1.10.13 wheels we have on PyPI, several of our binaries segfault on run. Even calling ldd on them fails.

Downgrading to patchelf<0.17 fixes the problem.

I would suggest that manylinux ship an older patchelf for now until these issues are fixed upstream, since it is affecting multiple projects.