Looking for docs on the loader fragment mechanism

[mikehearn]

Hello,

I work on Hydraulic Conveyor which is a program that can build self-updating app packages for Windows, macOS and Linux. It has deep support for JVM apps, meaning you can just supply it with some JARs, run a local build and it spits out a static website with fully code signed, notarized and auto-updating downloads. It will hopefully make it much easier and more convenient to distribute JVM apps both on client and server, and that will hopefully benefit everyone using JavaCV and JavaCPP. It's free for open source projects.

A customer has reported that an app using JavaCV doesn't package correctly. The error is:

java.lang.UnsatisfiedLinkError: /path/to/package/lib/runtime/lib/libjniopenblas_nolapack.so: libopenblas_nolapack.so.0: cannot open shared object file: No such file or directory

And sure enough, libopenblas_nolapack.so.0 isn't there. This error has cropped up before with Spring Boot but this time Spring isn't involved in any way. Instead what's going on here is different: Conveyor processes JARs as part of the building the app image before packaging it. A part of that processing involves deleting shared libraries from the JARs if they're the wrong CPU architecture for the package being built, and moving libraries of the right arch to the lib directory of the bundled JVM. This is done for several reasons:

1. All native code has to be code signed on some platforms, and it's a good idea to do it on all of them. Rather than rummage through JARs to find signable libraries we just extract them.
2. It ensures apps uninstall cleanly without leaving cache directories behind in the user's home directory.
3. It makes startup faster because there's no extraction step required (assuming your Java code tries to load the library and falls back to extraction only if that fails).
4. It makes downloads smaller because you aren't shipping useless binaries for the wrong CPU or OS.
5. It avoids bloating the user's home directory (which may be roaming, on Windows).

So, a bunch of advantages to doing things this way.

On investigation, it appears that JavaCPP has a rather featureful and complex JNI loader. The libopenblas_nolapack library is indeed missing from the JARs and the JavaCV POM seems to exclude them deliberately. Yet when looking at the .javacpp/cache directory created when not using Conveyor, the library is indeed there on disk.

It seems that this library is special in some way and inability to load it crops up in various bug reports with various different causes. Obviously this time it's a new cause.

From some reading and grepping, my guess is that this library is 'special' due to the fragments, which looks like some sort of aliasing mechanism. In the decompiled class file for this library we can see:

@Properties(
    inherit = {javacpp.class},
    global = "org.bytedeco.openblas.global.openblas_nolapack",
    value = {@Platform(
    define = {"__OPENBLAS 1", "LAPACK_COMPLEX_CPP"},
    include = {"openblas_config.h", "cblas.h"},
    link = {"openblas_nolapack@.0"},
    resource = {"include", "lib"},
    preload = {"gcc_s@.1", "quadmath@.0", "gfortran@.5", "gfortran@.4", "gfortran@.3", "openblas@.0#openblas_nolapack@.0"},
    preloadpath = {"/opt/intel/oneapi/mkl/latest/lib/", "/opt/intel/oneapi/compiler/latest/mac/compiler/lib/"}
)

That looks an awful lot like libopenblas.so.0 is supposed to be aliased to libopenblas_nolapack.so.0 and sure enough if we compare the files in the cache directory they are identical.

Finally (phew!) we reach my actual questions :-)

1. Is JavaCPP actually designed to allow non-cache/extraction based loading of libraries? I mean, is shipping libraries in a packaged app for installation purposes actually supported at all? It feels like it should be because it does look in the right places.
2. Is this aliasing mechanism documented anywhere? I took at look at the loader page on the wiki but didn't see anything.
3. Is this #alias system just a space saving optimization for the JAR or is it deeper than that?
4. Is there any generic way to go from JavaCPP JARs to a set of libraries on disk that will work, or, will fixing this require specific knowledge of JavaCPP metadata in Conveyor itself?

[saudet]

@HGuillemet @johanvos @HannesWell @reckart @timothyjward are facing similar issues with other frameworks like Android and OSGi, so let's bring them in the discussion. The only standard loading mechanism that OpenJDK is willing to offer is jlink, which does not actually fulfill user requirements to any meaningful extent, so we still all need to come up with our own disparate hacks. JavaCPP is no different. However, it does fall back on System.loadLibrary(), so if your framework is able to place all the required libraries such that System.loadLibrary() succeeds, then JavaCPP will be happy. We don't usually need to know anything more than that, so there isn't any detailed documentation about all the corner cases, which would be a huge waste of time to maintain. OpenJDK should be offering something that works without all those ugly workarounds, but they won't, so here we are.

Now, the case of OpenBLAS is a bit special. It provides a standard implementation of BLAS and LAPACK, but to let users switch to a different implementation at runtime, JavaCPP can create links to those offered by MKL, Accelerate, etc that may be installed on the system. Although MKL provides an implementation of LAPACK, Accelerate does not. So applications that do not need LAPACK should use the API exposed by libopenblas_nolapack.so.0 instead of libopenblas.so.0. However, we don't actually need libopenblas_nolapack.so.0, since we can just copy libopenblas.so.0 at runtime, and libjniopenblas_nolapack.so can link with that. Further, if we were to load one library as libopenblas_nolapack.so.0 but then load another library as libopenblas.so.0 because we find out later that the user needs LAPACK, that can result in conflicts since we end up with multiple implementations of BLAS in memory, so that's why they should be the same library for applications that use LAPACK. In any case, if your users are unlikely to want to use an implementation of BLAS and LAPACK other than OpenBLAS, you can simply bundle libopenblas.so.0 along with a symbolic link to it named libopenblas_nolapack.so.0. Does that make sense? To automate that process @HGuillemet started working on extending CacheMojo with pull #510, and he may have a few updates to share about how he's been dealing with this in the case of jlink.

[saudet]

AFAIK, there's no standard way to leave resources in an extracted state with jlink. Everything has to be bundled in an image. I still don't understand how we're supposed to make System.load() do something useful other than load system libraries that are known to be present on all systems.

[mikehearn]

Ah, you mean, JAR resources always get put into the modules file? Right. That's true indeed. Conveyor extracts native libraries before jlinking for that reason.

[saudet]

Which is fine, if System.loadLibrary() could do all the things we can do with System.load(), but that's not always the case. One thing I've already explained is that, if your users want to use libraries installed on the system like CUDA, MKL, etc, they won't be able to, unless you provide multiple different bundles for your application, and then well your users have to figure out the mess. Android "solves" this in the cloud with Google Play by forcing publishers to upload everything, but at least the users don't have to figure out which packages they need to download. Now, that would be OK if your applications are all mobile-like constrained, but the more important issue is that System.loadLibrary() has neither support for library versions (library filenames ending with something else than ".so"), nor the ability to load dependent libraries, see https://bugs.openjdk.org/browse/JDK-8213772. On non-Windows machines that means you'll need to hack manually the builds of all the native libraries your application depends on. On Android, they "solve" that by removing support for library versions from Linux. Obviously, that's not an option for OpenJDK on the desktop.

[mikehearn]

Got it, thanks. The launcher in our case uses an rpath to ensure that the directory the libraries are extracted to is always on the linker search path, so I think that bug won't hit us. WRT versions, indeed you need the symlinks in some cases.

[saudet]

Ah, yes, RPATH does help with that, and JavaCPP does include one for the JNI wrappers, but RPATH is actually deprecated, GCC uses RUNPATH by default, which still means that ideally we need to hack the build of all dependent libraries! It's not impossible to make everything work, but jlink just makes it that much harder, and like I keep telling @HGuillemet I can't do everything on my own! Contributions are welcome though, and it would be nice if OpenJDK would actually give a damn about it, but unfortunately, they don't.

[HGuillemet]

I indeed worked on this problem for the same 5 reasons you mention.
I currently shipped applications by pre extracting the JavaCPP native libraries in the JVM runtime. To simplify this process, I wrote 2 Gradle plugins:

• javacpp-libextract : that uses Javassist and the JavaCPP loader to populate a directory with the needed native libraries, and only the needed ones (the openblas special case should work).
• javacpp-jlink : that uses the previous plugin for jlink (about which I'm much less negative than @saudet).

Have a look at the README in the github repos and see if they can help you.
I haven't advertise them yet, not even talked to much with @saudet of how to publish them, because they would need more tests and they require a JavaCPP feature present in 1.5.8 SNAPSHOT only. There is also a small issue with Gradle daemon mode as explained in the README.

[mikehearn]

OK. It sounds like the way to go is: after jlink but before packaging, run the app in a special mode where it uses this org.bytedeco.javacpp.loadLibraries system property to make JavaCPP unpack and set up the native libraries (symlinking, copying, etc) to a dedicated 'cache' directory inside what will get packaged. This ensures everything gets signed, is immutable at runtime etc whilst still letting JavaCPP do whatever it needs to the libraries. Then at runtime it can just use System.load given an absolute path derived from app.dir (or whatever other tools supply).

Probably the app needs a dedicated entry point as presumably, javacpp does the unpacking on-demand and the app has to drive that.

[saudet]

Yeah, but it's not going to work as easily as you make it sound. Desktop applications are not typically like mobile applications. Sure, we can make them run that way, but because they are intrinsically limited, that's not how JavaCPP is designed to work. We have to do it in the case of Android, so that's why it's possible to hack things a bit so that ultimately we can leave the System.loadLibrary() do its tricks, and that's most easily done with Gradle because Android uses Gradle for its build system. If your framework works better with Maven, we can do stuff in the Maven plugin as well for that. Feel free to try it out and make contributions as necessary.

[mikehearn]

OK. You mentioned mobile vs desktop a few times now. I'm not sure I understand what mobile/desktop execution difference you mean here - maybe my slightly obsolete Android knowledge is limiting me. Could you clarify? Does ART implement loadLibrary/load in different ways to HotSpot?

[saudet]

Android depends on OpenJDK, and they do not modify anything that would break compatibility with OpenJDK, I assume mainly for licensing reasons to make sure they don't get sued by Oracle again, but for technical reasons too I'm sure. What they do, as I explain above #600 (reply in thread) is they work around these issues via cloud services like Google Play and by amputating the Linux kernel to make sure developers can't do things that important APIs like System.loadLibrary() do not support in the first place.

[HGuillemet]

Sure, so @HGuillemet did add a "org.bytedeco.javacpp.loadLibraries" system property for that that you can set to "false", but I think he just gave up on all that because there's no way to make this work well with jlink.

In fact that's a property you added a long time ago to be used by the Builder to prevent library loading. But it prevents library extraction too.
Maybe you meant org.bytedeco.javacpp.findlibraries that we added recently. This one tells JavaCPP not to look for libraries in resources (see below).

OK. It sounds like the way to go is: after jlink but before packaging, run the app in a special mode where it uses this org.bytedeco.javacpp.loadLibraries system property to make JavaCPP unpack and set up the native libraries (symlinking, copying, etc) to a dedicated 'cache' directory inside what will get packaged. This ensures everything gets signed, is immutable at runtime etc whilst still letting JavaCPP do whatever it needs to the libraries. Then at runtime it can just use System.load given an absolute path derived from app.dir (or whatever other tools supply).

Probably the app needs a dedicated entry point as presumably, javacpp does the unpacking on-demand and the app has to drive that.

Yes, that's the goal of the javacpp-libextract Gradle plugin mentioned above : extract the needed libraries in a directory of your choice using the "JavaCPP's loading hacks". With Maven system, the CacheMojo does something similar, but will not extract automatically the minimum set of libraries needed. Instead it extracts the libraries of a presets by Loader.loading either all classes in the preset, or an explicit subset of these classes (as specified with classOrPackageNames property).

Then there are two options: either extract to a dedicated directory (or several dedicated directories if you would like to support different variants like CUDA-enabled and not CUDA-enabled) or direcly to the Java runtime library directory (like it's done when you build the jlink image with jmods containing native libraries, like JavaFX jmods).
After going back and forth, in the javacpp-jlink gradle plugin, I finally opt to extract to the Java runtime, I's much simpler. Using a dedicated directory, you need to pass this directory to the JVM using a java.library.path based on launcher absolute path AND to ensure that all library dependencies are sought in this directory. Both must be dealt with platform-dependent tricks. I believe there is also an issue on MacOS which will ignore java.library.path unless you remove some security constraint. Maybe possible, but I gave up this option.

Extracting to jlink runtime works well for me (including signing), but you also have to:

• set the new property org.bytedeco.javacpp.findlibraries to false in the launcher to prevent JavaCPP to look uselessly for libraries in resources , and also to prevent the creation of the user cache directory.
• on Linux, ensure that the RPATH/RUNPATH of all bundle libraries is set to $ORIGIN/, or the linux linker will fail to link dependencies, or will link with libraries found on user system when you want to link with bundled ones. This can be done with patchelf just after the libextract step. But ideally all native libraries in JavaCPP presets should already have the RUNPATH set to $ORIGIN. Similarly, on MacOS, rpath must be set and all libs should use @rpath in their install names.