jffi9396993929588479283.so: /lib64/libc.so.6: version `GLIBC_2.27' not found #138
Comments
|
Faced the same issue recently. Still investigating, not sure yet if it's tied to a java update or centos update on our side, as it used to work about a month ago. |
|
Not an issue on our end, it seems. Once i downgraded to jnr 2.1.12 the problem disappeared. Sadly will have to exclude this library from auto-updates for the time being. |
|
I am also facing this exact issue. The latest Since then my bamboo builds are failing. When going back to |
|
We have the same error which is prevent us to upgrade to the newest version. |
|
Looking into this now. Anyone have any idea what version this started at? Over the past year we've moved to using GHA to do builds of all the Linux-based jffi binaries. When building for x86_64, I believe we just build directly on GHA, so it's going to be a very recent version of Ubuntu/Debian-ish Linux. If we need to be mindful of the glibc version, that will require some finesse. |
@DManstrator So that narrows it down, thank you. |
|
FWIW this is really a jffi issue so I'm going to move it there. |
|
Ok so we are also building x86_64 binaries using Docker, so there's a path to use an older Linux install that has an earlier glibc version. Unfortunately, we're building based on Debian, and even as far back as debian:8, the closest we get is glibc 2.19. The alternative to this would be having the build fetch and build an older glibc and link against that, but that's quite a bit more work. I will try a branch that uses debian:8 to build and see if it works for those of you on CentOS 7. |
|
Anyone having this issue that's NOT on CentOS 7? |
Building on debian:10 results in a binary that requires glibc 2.27 which is not available on some LTS server Linuxes like CentOS 7. This commit attempts to make a compatible jffi.so using debian:8, which ships with glibc 2.19. This is not as old as 2.17 but may remove enough dependencies on newer glibc to be sufficient. See #138
|
Ok, there it is... can someone on CentOS 7 give the binaries here a try? https://github.com/jnr/jffi/actions/runs/4196057988 Everything seems to have built ok on debian:8. We can try to go even earlier, but we're getting into weird territory. |
|
My PR in #138 did not work unfortunately. Switching to debian:8 did not appear to produce binaries for older glibc. I welcome assistance getting this to build with older glibc in CI. |
|
Let me clarify a bit more. Debian >= 9 and ubuntu >= 20.04 have no problem. This issue is only on CentOS7. Besides the symbol issue, it looks like a directory permission bug. Here is the full logs: |
|
@kingluo That looks more like an SELinux restriction, preventing the usual temp directories from being used for executable code. You would either disable SELinux or set a temp dir that does not have such restrictions (like one that is local to the current user). The main challenge at the moment is that I don't know how to build the Linux jffi extension against an old glibc using Github Actions. We have a rig that uses Docker and qemu to build for a number of different Linux platforms, but setting that to debian:8 (the earliest image available) did not roll back glibc far enough. |
I tried both before, but it does not work. It seems a false log.
I am curious about that did you try to build it in the local environment (VM or container) with Debian8 or Centos7 and test it without a problem? If so, then GitHub action is an insignificant side issue. |
|
@kingluo Ok, maybe open that as a separate bug. |
|
@kingluo ...
I need for anyone to be able to reproduce these builds in any environment, and making it happen in GHA is an easy way to do that (just reproduce the docker setup and it will work). I can certainly build it myself in my CentOS7 VM, but then I'll have to manually do that again next time, and the following times, etc. If this is a serious blocker for folks on this issue, we could go with the CentOS 7 manual build for one release, but I need to automate it somehow. |
|
@headius Have you reproduced the bug and found a bug fix? |
|
Hi, In case of CentOS 7 and RHEL 7 this issue seems to be caused by mismatch of Glibc version. I have the following stack trace from Jenkins (running on RHEL 7): Glibc version on respective Jenkins node is 2.17 (i.e. older than required 2.27): |
|
@kingluo I have reproduced it and the only fix I know of at the moment is to build the extension on CentOS7/RHEL7. |
Building on debian:10 results in a binary that requires glibc 2.27 which is not available on some LTS server Linuxes like CentOS 7. This commit attempts to make a compatible jffi.so using debian:8, which ships with glibc 2.19. This is not as old as 2.17 but may remove enough dependencies on newer glibc to be sufficient. See #138
|
The new PR #140 appears to build a working binary! I will update it to have the new build archived and available and then need someone to confirm it locally before release. |
|
PR #140 now has an updated archived build for Linux x86_64 that appears to work on CentOS 7. Can someone verify that branch works for them? |
|
Hi @headius, I confirm that Linux x86_64 binary from PR #140 fixes this issue. Here is my test I performed on CentOS 7.9:
|
|
@mabrarov This is excellent information and exactly what I was looking for, thank you! I will proceed with putting out a jffi release and start to test it in the rest of the jnr-* family. |
|
Ah there is a wrinkle here I had not noticed: this is in libffi code. It may make more sense for us to explicitly disable HAVE_MEMFD_CREATE during the build since we don't really want to be modifying libffi code directly. |
|
Another approach: call the kernel function directly rather than binding to the libc version: memfd = syscall(SYS_memfd_create, "", 0);https://www.rapid7.com/blog/post/2019/12/24/memory-laundering-is-cleaner-better/ I'm digging around to figure out what the implications are of avoiding |
|
More background from reading code and information about memfd_create... This is being used in libffi to generate callbacks for functions that need them, in order to take two independent functions and make a "closure" so the one can call the other by pointer. In order to get around execution checks, a file is filled with the closure stub code and then mapped into memory as executable. The old approach uses The old approach should work fine, but will require privileges to access a temp location. Luckily, jffi already requires this in order to unpack an appropriate platform binary which is then loaded in as executable code. |
|
Ok, the syscall approach will also not work because the kernel in RHEL/CentOS 7 predates the addition of this function. I want to verify that there's no impact from just dropping its use altogether. |
|
I probably am missing a way to do this, but I just filed libffi/libffi#826 to try to get libffi to add The down sides as I know them currently:
|
|
Disable it thusly: ./configure 'ac_cv_func_memfd_create=no' |
|
I should mention that in the latest version of libffi, the temp file approach is just a backup for when static trampolines aren't available for some reason. Static trampolines are a relatively new way to implement closures so they don't require RWX memory. It's best explained here: https://blog.lazym.io/2021/07/29/Cast-a-Closure-to-a-Function-Pointer-How-libffi-closure-works/ |
|
@atgreen Thank you for your help! That should be sufficient for us to build a binary that works even on older glibc/linux without memfd_create...if we even have to?
Ahh this appears to be (similar to?) the "double mmap" strategy I read about in a few other posts. So are you saying that latest libffi no longer depends on memfd_create? If so, perhaps we can simply update libffi and build a binary that supports those older platforms? I'll push a PR that updates libffi and then @PerfectSlayer @rosti-il @mabrarov and others here can test it out on their environments. |
|
There are now builds of jffi (with updated libffi) for x86_64, arm, and aarch64 that could be tested (I think only the x86_64 is relevant to this issue) here: https://github.com/jnr/jffi/actions/runs/7889009015?pr=153 |
|
I will need to find another OpenJDK distribution that publishes i386 binaries for Java 8 on Debian to get that build working. |
|
@mabrarov @PerfectSlayer @kingluo @RainerGanss @DManstrator @Sobakaa Could you confirm whether the built binaries from #153 work for you under CentOS/RHEL 7? See the binaries at the bottom of this page: https://github.com/jnr/jffi/actions/runs/7889009015?pr=153 |
|
Hi @headius I took a look at the x64 and arm64 binaries (I used readelf, which is in binutils) and the latest change made things worse because there's now a GLIBC 2.27 symbol (memfd_create) exported by the x64 binary: and still a GLIBC 2.27 symbol (memfd_create) exported by the arm64 binary: So the binaries can't be linked on systems with glibc versions older than 2.27. Though you're going to want to update the libffi dependency at some point anyway, I think that leaving libffi and the build images alone and applying the command mentioned above, (assuming |
|
@richardstartin Aha, thank you for that (and for providing your command lines so I can check this myself). I will do a separate PR with existing libffi that configures without memfd_create and we'll see if that looks better |
Workaround for #138 until we can come up with a better solution.
|
Ok I've merged the build fixes to master, rebased libffi update for another day, and pushed #155 to add that configure line to the build. |
|
@richardstartin The new binary looks better on x86_64: |
|
It was the aarch64 and arm binaries causing problems before (and loongarch64 but I think we can ignore that one) #138 (comment) |
|
aarch64 looks ok too: |
|
I don't know how best to update the i386 build but I will pull in the new binaries for x86_64, arm, and aarch64 and start the process of release testing. |
This pulls in binaries for x86_64, aarch64, and arm that have been configured to avoid the use of memfd_create, a glibc 2.27 feature not available on some older (but still-supported) RHEL and CentOS versions. The i386 build is not updated due to issues finding an appropriate Java 8 build. See #138.
Potentially fixes this jnr/jffi#138
The error message is very weird.
In Centos7 with openjdk 11:
It reports
noexecproblem, but the/homeis not mounted withnoexec, is it symbol issue?I found only
memfd_createC symbol comes from `GLIBC_2.27'.Please help, thanks.
The text was updated successfully, but these errors were encountered: