JVM Crashing in Dockerized Alpine Environment

[dgmneto]

This is the code I'm using to reproduce the problem:

import com.uber.h3core.H3Core;

import java.io.IOException;

public class Main {
    private static final double[][] coordinates = new double[][] {
            {0.0    , 90.0    },
            {13.5467, -19.2347},
            {13.5467, -30.2347},
            {17.5467, -30.2347}
    };

    public static void main(String[] args) throws IOException {
        H3Core h3Core = H3Core.newInstance();
        for (double[] coordinate: coordinates) {
            long h3Id = h3Core.geoToH3(coordinate[0], coordinate[1], 12);
            System.out.printf("Lat: %s Lng: %s H3: %d%n", coordinate[0], coordinate[1], h3Id);
        }
    }
}

I'm using the gradle docker plugin to build the docker images. The project with the crashing scenario and setup is available here.
To build the image, run ./gradlew dockerBuildImage. Then run docker run -it divinomenezes/h3-issue-test:latest to run the code. You should be able to run normally.
If you change the build.gradle:27 as described in the comment and rerun the steps above, you should see the following error:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x0000000000003b36, pid=1, tid=0x00007fbaf7e76ae8
#
# JRE version: OpenJDK Runtime Environment (8.0_171-b11) (build 1.8.0_171-b11)
# Java VM: OpenJDK 64-Bit Server VM (25.171-b11 mixed mode linux-amd64 compressed oops)
# Derivative: IcedTea 3.8.0
# Distribution: Custom build (Wed Jun 13 18:28:11 UTC 2018)
# Problematic frame:
# C  0x0000000000003b36
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# //hs_err_pid1.log
#
# If you would like to submit a bug report, please include
# instructions on how to reproduce the bug and visit:
#   http://icedtea.classpath.org/bugzilla
#

The hs_err_pid1.log is also available here.

[isaacbrodsky]

Thanks for providing detailed steps for reproducing. I was able to reproduce the issue with the openjdk:8-alpine image.

After taking a look at the core dump, it looks like the issue is inside the function h3ToGeo. I noticed it calls a C core library function isfinite, and I was able to move the crash up in the call stack by adding a call to isfinite in jniapi.c. The issue is that the linux-x64 cross-compile build links to glibc, which is not available on Alpine Linux.

Since installing glibc on Alpine or statically linking with glibc are not recommended, it looks like the best approach here might be to change the linux-* cross-compile builds to statically link to musl.

[dgmneto]

If this is the path we'll follow, I'd really enjoy opening a PR to change that during the weekend.
Are you aware of any dependency from glibc that is not available in musl?

[isaacbrodsky]

No, we shouldn't have any dependency on glibc itself. The core library has been tested with Alpine Linux and musl itself, and with MSVC.

cc @dfellis @isaachier @nrabinowitz in case there's something I'm not considering about statically linking to a different libc.

[isaachier]

@isaacbrodsky statically linking libc is generally a bad idea, and I think affects the licensing of the resulting binary (https://lwn.net/Articles/117972/), so H3 doesn't do that.

[dgmneto]

@isaachier I believe the idea is to statically link musl, actualy. musl has a MIT license

[dfellis]

I don't really like the idea of statically linking Musl to support one uncommon Linux variant. If we're going to support this, I'd rather we do some autoconf-like detection of which library, glibc or musl, to link to, and to use the OS-provided one. Otherwise we're opening ourselves to an upgrade treadmill for all security updates to musl.

[isaachier]

@dgmneto I think @isaacbrodsky helped me find the underlying problem. Here is a quote from the musl FAQ:

At present, some glibc-linked shared libraries can be loaded with musl, but all but the simplest glibc-linked applications will fail if musl is dropped-in in place of /lib/ld-linux.so.2.

https://www.musl-libc.org/faq.html

[dgmneto]

I'll have to disagree with @dfellis on how uncommon Alpine is: in the docker world Alpine is usually used to reduce layer/image sizes. Even so, I get your point.
We are using H3 in a few Java applications rn, so I'll try to make a dockcross that uses musl-gcc and then I'll share here the results.

[isaachier]

@isaacbrodsky somewhat related question: Would it make sense to use -DCMAKE_BUILD_TYPE=RelWithDebInfo so the stacktraces aren't hex values?

[dfellis]

I don't want to get into a Linux distro flamewar (Linux is my daily driver, I've had enough of those over the years), I wanted to point out that we could easily tangle ourselves into a support nightmare when we look beyond the top 10 Linux distros (Alpine is currently 65th according to Distrowatch, and even in the world of Docker, Alpine has half the stars of Ubuntu).

[isaachier]

@dfellis I understand your position, but you'd be surprised about how many docker images use Alpine as a base. Stars aside, I wonder if there is a way to search for base image "spinoffs."

[isaachier]

See here for known issues with libraries on Alpine Linux: https://github.com/gliderlabs/docker-alpine/blob/master/docs/caveats.md.

[isaacbrodsky]

A good solution would be to package H3 itself as a package in Alpine, then H3-Java and other bindings can use the system provided core library.

edit: this would be limited by the need for JNI adapter functions.

The other option would be for Alpine users to build H3-Java on Alpine Linux and use the built artifact from there. That should be fully compatible with Alpine.

[isaachier]

Yes that would work but you will have a lot of package systems to work with (debian, rpm, etc.).