Research GraalVM native image compatability #91
Comments
vitaliihonta
added
enhancement
|
While the The biggest quirk is that when I generate this metadata by using an e2e test, a mock temporal server is used so the agent apparently doesn't hit the GRPC classes not including them in the metadata so the generated binary fails. When the metadata is generated with a real run of the worker and a real workflow execution, the binary works (supposedly all classes are found). Some refs:
Binary error when running with the e2e generated metadata: |
|
There are two points on this IMO, one is the proxy/reflection/jni metadata, the other is initialization flags. Reflection/proxy/etc metadata could be handled by the tracing agent as I described above. For the initialization, most flags from the My big gripe is where all this initialization flags should be?
@Spikhalskiy, would you take a look at this since you worked on GraalVM in Temporal SDK? |
|
On talks with GraalVM guys (https://twitter.com/carlosedp/status/1675945560119865350), I found that all classes are initialized at run-time by default so looking at GraalVM PRs, I found https://github.com/oracle/graal/pull/3179/files that added a flag to exclude configs from regexes. Since a lot of libs bundle initialization rules, I've added to my native-image build the flag regex'ing out all In the end I've successfully built the native-image tackling one of the issues listed above. The other one is how to properly generate the json metadata automatically. Since zio-temporal also uses reflection, I don't think bundling this into the library solves since the user's workflows and activities should also be added as it can be seen in https://github.com/carlosedp/zio-temporal-hello/blob/3f33c09d6cf2cec201eec5b3bf8d3e33fc172481/shared/resources/META-INF/native-image/proxy-config.json#L18-L21. Here's the PR removing the flags and excluding lib configs: carlosedp/zio-temporal-hello#4 |
|
One thought I had is that we could bundle the reflection/proxy/etc json metadata for the zio-temporal and temporal classes (like the GRPC ones) in zio-temporal and kinda automate the generation of the additional metadata required by the user application (like the activities and workflows) at build time. In talks with Alex Archambault, he mentioned they do that in scala-cli... I searched and it's here: https://github.com/VirtusLab/scala-cli/blob/a7b887a45fadc158cafa37a14dcbac8dd0ea36ca/project/settings.sc#L414-L435 At native-image generation, it creates the metadata beforehand. Some docs are here: https://github.com/VirtusLab/scala-cli/blob/a7b887a45fadc158cafa37a14dcbac8dd0ea36ca/DEV.md?plain=1#L254 |
|
Last commit to my sample app added some missing reflection/proxy classes: carlosedp/zio-temporal-hello@793d3ec |
|
Kinda got a false-positive on #91 (comment) where I've removed all initialization flags replacing by a config to exclude the properties from JARs. The image was built successfully but once I've tried to access the app's HTTP server (based on zio-http which uses netty) I got the error: And the app doesn't respond of return the HTTP request. Weird because the temporal requests are processed and the issue only happens once http is used |
|
Are you using the default metadata for Netty as available at https://github.com/oracle/graalvm-reachability-metadata/tree/master/metadata/io.netty/netty-common? What Netty version are you using? The error indicates that there is a missing reflection entry for: Netty uses a lot of complex reflection in static constructors. If you initialize those classes at build time, the configuration is not required. It is still recommended to rather configure than initialize at build time. |
|
Oh that was it... the issue is that zio-http uses netty-common version 4.1.93 and the metadata has 4.1.80. I've manually generated the metadata using the native-image-agent updating the existing metadata and it worked! Another thing I should add to my end-to-end tests to generate http requests so the metadata is generated automatically. Thanks Thomas! |
|
Tried to consolidate the steps I use to generate all metadata for GraalVM in my project: carlosedp/zio-temporal-hello#5 |
|
OK, cool! Starting from netty 5, the metadata is directly included in the netty JAR file itself to avoid that problem. Can you maybe create a pull request at https://github.com/oracle/graalvm-reachability-metadata with the 4.1.93 netty data? |
|
FYI, the Agent way is not working for me, nor the I only have 2 dependencies: Netty ( What is "working" is to use these $ ./my-cli say-hello
00:52:29.427 [ZScheduler-Worker-10] ERROR zio-slf4j-logger -
java.lang.ExceptionInInitializerError: null
at java.base@20.0.1/java.lang.Class.ensureInitialized(DynamicHub.java:579)
at java.base@20.0.1/java.lang.Class.ensureInitialized(DynamicHub.java:579)
at java.base@20.0.1/java.lang.Class.ensureInitialized(DynamicHub.java:579)
at java.base@20.0.1/java.lang.Class.ensureInitialized(DynamicHub.java:579)
at java.base@20.0.1/java.lang.Class.ensureInitialized(DynamicHub.java:579)
at java.base@20.0.1/java.lang.Class.ensureInitialized(DynamicHub.java:579)
at io.netty.util.internal.PlatformDependent$Mpsc.newMpscQueue(PlatformDependent.java:1029)
at io.netty.util.internal.PlatformDependent.newMpscQueue(PlatformDependent.java:1040)
at io.netty.channel.nio.NioEventLoop.newTaskQueue0(NioEventLoop.java:283)
at io.netty.channel.nio.NioEventLoop.newTaskQueue(NioEventLoop.java:154)
Caused by: java.lang.RuntimeException: java.lang.NoSuchFieldException: producerIndex
at io.netty.util.internal.shaded.org.jctools.util.UnsafeAccess.fieldOffset(UnsafeAccess.java:111)
at io.netty.util.internal.shaded.org.jctools.queues.BaseMpscLinkedArrayQueueProducerFields.<clinit>(BaseMpscLinkedArrayQueue.java:55)
at java.base@20.0.1/java.lang.Class.ensureInitialized(DynamicHub.java:579)
at java.base@20.0.1/java.lang.Class.ensureInitialized(DynamicHub.java:579)
at java.base@20.0.1/java.lang.Class.ensureInitialized(DynamicHub.java:579)
at java.base@20.0.1/java.lang.Class.ensureInitialized(DynamicHub.java:579)
at java.base@20.0.1/java.lang.Class.ensureInitialized(DynamicHub.java:579)
at java.base@20.0.1/java.lang.Class.ensureInitialized(DynamicHub.java:579)
at io.netty.util.internal.PlatformDependent$Mpsc.newMpscQueue(PlatformDependent.java:1029)
at io.netty.util.internal.PlatformDependent.newMpscQueue(PlatformDependent.java:1040)
Caused by: java.lang.NoSuchFieldException: producerIndex
at java.base@20.0.1/java.lang.Class.checkField(DynamicHub.java:1004)
at java.base@20.0.1/java.lang.Class.getDeclaredField(DynamicHub.java:1119)
at io.netty.util.internal.shaded.org.jctools.util.UnsafeAccess.fieldOffset(UnsafeAccess.java:107)
at io.netty.util.internal.shaded.org.jctools.queues.BaseMpscLinkedArrayQueueProducerFields.<clinit>(BaseMpscLinkedArrayQueue.java:55)
at java.base@20.0.1/java.lang.Class.ensureInitialized(DynamicHub.java:579)
at java.base@20.0.1/java.lang.Class.ensureInitialized(DynamicHub.java:579)
at java.base@20.0.1/java.lang.Class.ensureInitialized(DynamicHub.java:579)
at java.base@20.0.1/java.lang.Class.ensureInitialized(DynamicHub.java:579)
at java.base@20.0.1/java.lang.Class.ensureInitialized(DynamicHub.java:579)
at java.base@20.0.1/java.lang.Class.ensureInitialized(DynamicHub.java:579)Any idea how to fix this @thomaswue @carlosedp maybe? 🤔 |
|
The usage of The crash at runtime is because the reflection/proxy metadata is missing and netty requires them. You need to generate with the agent. Now on to the initialization, what happens is that netty/logback changes some initialization to build-time in it's own metadata so you might need to override or ignore it like I did in my app here: https://github.com/carlosedp/zio-temporal-hello/blob/32adb3de5f0e724833e3a1bb28089d9f50fb8b0c/build.sc#L74C7-L74C7 If you have the link to the sample app, I could look at it to help :) |
|
There you go @guizmaii ... a sample app with zio-http and logback which can generate a native-image... I added all configuration needed in the scala file itself so it can be run with scala-cli. https://gist.github.com/carlosedp/6547bc730899995683442916bb225876 |
|
@carlosedp what is this configuration for/about Edit: [info] /Users/jules/.sdkman/candidates/java/20.0.1-graal/bin/native-image --exclude-config .*.jar,.*.properties -cp /Users/jules/workspace/myproject/modules/cli/target/native-image-internal/manifest.jar --no-fallback --enable-http --enable-https --enable-url-protocols=http,https --enable-sbom --install-exit-handlers --diagnostics-mode -H:+BuildReport -H:ConfigurationFileDirectories=/Users/jules/workspace/myproject/modules/cli/src/main/resources/META-INF/native-image io.myproject.cli.CLI /Users/jules/workspace/myproject/cli
# Diagnostics mode enabled: image-build reports are saved to reports/diagnostics_20230718_110258
Error: Unrecognized option: --exclude-config .*.jar,.*.properties
[error] native-image command failed with exit code '20'
[error] (cli / nativeImage) native-image command failed with exit code '20'Edit 2: Edit 3: [info] /Users/jules/.sdkman/candidates/java/20.0.1-graal/bin/native-image -cp /Users/jules/workspace/myproj/modules/cli/target/native-image-internal/manifest.jar --no-fallback --enable-http --enable-https --enable-url-protocols=http,https --enable-sbom --install-exit-handlers --diagnostics-mode -H:+BuildReport -H:ExcludeResources=.*.jar,.*.properties -H:ConfigurationFileDirectories=/Users/jules/workspace/myproj/modules/cli/src/main/resources/META-INF/native-image io.myproj.cli.CLI /Users/jules/workspace/myproj/clifails with: Error: Classes that should be initialized at run time got initialized during image building:
ch.qos.logback.classic.Logger was unintentionally initialized at build time. To see why ch.qos.logback.classic.Logger got initialized use --trace-class-initialization=ch.qos.logback.classic.Logger
ch.qos.logback.core.hook.DefaultShutdownHook was unintentionally initialized at build time. To see why ch.qos.logback.core.hook.DefaultShutdownHook got initialized use --trace-class-initialization=ch.qos.logback.core.hook.DefaultShutdownHook
io.netty.channel.DefaultFileRegion the class was requested to be initialized at run time (subtype of io.netty.util.AbstractReferenceCounted). To see why io.netty.channel.DefaultFileRegion got initialized use --trace-class-initialization=io.netty.channel.DefaultFileRegion
org.slf4j.LoggerFactory was unintentionally initialized at build time. To see why org.slf4j.LoggerFactory got initialized use --trace-class-initialization=org.slf4j.LoggerFactory
ch.qos.logback.core.status.StatusBase was unintentionally initialized at build time. To see why ch.qos.logback.core.status.StatusBase got initialized use --trace-class-initialization=ch.qos.logback.core.status.StatusBase
ch.qos.logback.core.util.Duration was unintentionally initialized at build time. To see why ch.qos.logback.core.util.Duration got initialized use --trace-class-initialization=ch.qos.logback.core.util.Duration
io.netty.util.AbstractReferenceCounted the class was requested to be initialized at run time (from 'META-INF/native-image/io.netty/netty-common/native-image.properties' in 'file:///Users/jules/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/io/netty/netty-common/4.1.94.Final/netty-common-4.1.94.Final.jar' with 'io.netty.util.AbstractReferenceCounted'). To see why io.netty.util.AbstractReferenceCounted got initialized use --trace-class-initialization=io.netty.util.AbstractReferenceCounted
ch.qos.logback.core.status.InfoStatus was unintentionally initialized at build time. To see why ch.qos.logback.core.status.InfoStatus got initialized use --trace-class-initialization=ch.qos.logback.core.status.InfoStatus
To see how the classes got initialized, use --trace-class-initialization=ch.qos.logback.classic.Logger,ch.qos.logback.core.hook.DefaultShutdownHook,io.netty.channel.DefaultFileRegion,org.slf4j.LoggerFactory,ch.qos.logback.core.status.StatusBase,ch.qos.logback.core.util.Duration,io.netty.util.AbstractReferenceCounted,ch.qos.logback.core.status.InfoStatus
com.oracle.svm.core.util.UserError$UserException: Classes that should be initialized at run time got initialized during image building:
ch.qos.logback.classic.Logger was unintentionally initialized at build time. To see why ch.qos.logback.classic.Logger got initialized use --trace-class-initialization=ch.qos.logback.classic.Logger
ch.qos.logback.core.hook.DefaultShutdownHook was unintentionally initialized at build time. To see why ch.qos.logback.core.hook.DefaultShutdownHook got initialized use --trace-class-initialization=ch.qos.logback.core.hook.DefaultShutdownHook
io.netty.channel.DefaultFileRegion the class was requested to be initialized at run time (subtype of io.netty.util.AbstractReferenceCounted). To see why io.netty.channel.DefaultFileRegion got initialized use --trace-class-initialization=io.netty.channel.DefaultFileRegion
org.slf4j.LoggerFactory was unintentionally initialized at build time. To see why org.slf4j.LoggerFactory got initialized use --trace-class-initialization=org.slf4j.LoggerFactory
ch.qos.logback.core.status.StatusBase was unintentionally initialized at build time. To see why ch.qos.logback.core.status.StatusBase got initialized use --trace-class-initialization=ch.qos.logback.core.status.StatusBase
ch.qos.logback.core.util.Duration was unintentionally initialized at build time. To see why ch.qos.logback.core.util.Duration got initialized use --trace-class-initialization=ch.qos.logback.core.util.Duration
io.netty.util.AbstractReferenceCounted the class was requested to be initialized at run time (from 'META-INF/native-image/io.netty/netty-common/native-image.properties' in 'file:///Users/jules/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/io/netty/netty-common/4.1.94.Final/netty-common-4.1.94.Final.jar' with 'io.netty.util.AbstractReferenceCounted'). To see why io.netty.util.AbstractReferenceCounted got initialized use --trace-class-initialization=io.netty.util.AbstractReferenceCounted
ch.qos.logback.core.status.InfoStatus was unintentionally initialized at build time. To see why ch.qos.logback.core.status.InfoStatus got initialized use --trace-class-initialization=ch.qos.logback.core.status.InfoStatus
To see how the classes got initialized, use --trace-class-initialization=ch.qos.logback.classic.Logger,ch.qos.logback.core.hook.DefaultShutdownHook,io.netty.channel.DefaultFileRegion,org.slf4j.LoggerFactory,ch.qos.logback.core.status.StatusBase,ch.qos.logback.core.util.Duration,io.netty.util.AbstractReferenceCounted,ch.qos.logback.core.status.InfoStatus
at org.graalvm.nativeimage.builder/com.oracle.svm.core.util.UserError.abort(UserError.java:73)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.classinitialization.ProvenSafeClassInitializationSupport.checkDelayedInitialization(ProvenSafeClassInitializationSupport.java:277)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.classinitialization.ClassInitializationFeature.duringAnalysis(ClassInitializationFeature.java:164)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.lambda$runPointsToAnalysis$10(NativeImageGenerator.java:770)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.FeatureHandler.forEachFeature(FeatureHandler.java:86)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.lambda$runPointsToAnalysis$11(NativeImageGenerator.java:770)
at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.AbstractAnalysisEngine.runAnalysis(AbstractAnalysisEngine.java:179)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.runPointsToAnalysis(NativeImageGenerator.java:767)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.doRun(NativeImageGenerator.java:582)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.run(NativeImageGenerator.java:539)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.buildImage(NativeImageGeneratorRunner.java:408)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.build(NativeImageGeneratorRunner.java:612)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.start(NativeImageGeneratorRunner.java:134)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.main(NativeImageGeneratorRunner.java:94)Edit 4: Edit 5: |
|
Apparently you are using Graal 20 and the The "Classes that should be initialized at run time got initialized during image building" error is because netty and/or logback are initializing some classes at build-time with their own metadata, the exclude-config should override this. Also you don't need the According to Oracle docs, the
Check my Gist, it contains the basics to make it all work with both logback and zio-http (which uses netty). |
Take a look at @carlosedp example here https://github.com/carlosedp/zio-temporal-hello/tree/main/shared/resources/META-INF/native-image
I believe it could be handled somehow
The text was updated successfully, but these errors were encountered: