Java 17 support

[clairemcginty]

Java 17 has been supported in Dataflow as of Beam 2.37.0. Currently several of Scio's unit tests fail when run with java 17, mostly Kryo related.

[regadas]

Hi @clairemcginty looked into this recently on other projects. This can probably be useful? twitter/chill#692 (comment)

[clairemcginty]

@regadas ha, I just found that link-hopping through the Kryo Java17 ticket! the jvm opt workaround seems to fix about 90% of Kryo-related errors, I just need to track down the remaining few :) ty!

[clairemcginty]

Started a branch for Scio+Java 17: https://github.com/spotify/scio/compare/java_17

Adding the java opts seems to solve about 75% of the errors, but there are still a few Kryo-related error that need to be tracked down.

[clairemcginty]

It's odd, even with the java 17 javaOptions some tests are still failing related to Kryo, i.e.:

[info] ParquetEndToEndTest:
[info] Parquet SMB
[info] - should support writing Avro reading typed *** FAILED *** (793 milliseconds)
[info]   java.lang.reflect.InaccessibleObjectException: Unable to make field protected transient int java.util.AbstractList.modCount accessible: module java.base does not "opens java.util" to unnamed module @28f4e455
[info]   at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:354)
[info]   at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297)
[info]   at java.base/java.lang.reflect.Field.checkCanSetAccessible(Field.java:178)
[info]   at java.base/java.lang.reflect.Field.setAccessible(Field.java:172)
[info]   at com.esotericsoftware.kryo.serializers.FieldSerializer.buildValidFields(FieldSerializer.java:283)
[info]   at com.esotericsoftware.kryo.serializers.FieldSerializer.rebuildCachedFields(FieldSerializer.java:218)
[info]   at com.esotericsoftware.kryo.serializers.FieldSerializer.rebuildCachedFields(FieldSerializer.java:157)
[info]   at com.esotericsoftware.kryo.serializers.FieldSerializer.<init>(FieldSerializer.java:150)
[info]   at com.esotericsoftware.kryo.serializers.FieldSerializer.<init>(FieldSerializer.java:134)
[info]   at com.esotericsoftware.kryo.serializers.FieldSerializer.<init>(FieldSerializer.java:130)

I'm wondering if it's because we're on a lower version of Kryo (kryo-shaded stopped being published after 4.x, recommendation seems to be to use the versioned jar). Unfortunately, kryo 5.x has some breaking changes that conflict with the version used in Chill... 🤷‍♀️

[clairemcginty]

it looks like the culprit here is JavaOptions being ignored when fork := false 🤔

So there are three potential solutions I think:

1. Set Test / fork := true for all projects if java.version is 17.
2. Add the java opts to .jvmopts file -- but then we can't condition them on java.version == 17
3. Modify the GHA config to run all Java17 tests with an exported JAVA_OPTS="--add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED" env variable. If you want to run java 17 tests locally you'd have to remember to export those options with is a little inconvenient.

Solution 3 might be fine in the short term since we still do most active development with Java 11.