-
Notifications
You must be signed in to change notification settings - Fork 28k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-45144][BUILD] Downgrade scala-maven-plugin
to 4.7.1
#42899
Conversation
I am writing the PR description, will be updated later |
updated |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1, LGTM.
Is the idea that we can update this again, along with Scala to 2.13.11, after we drop Java 8 support? |
@srowen After this downgrade, we can upgrade to Scala 2.13.11 again without waiting for dropping Java 8, because the lower version of the plugin will not automatically adding the |
OK that's useful if so. |
Ya, I agree with @LuciferYang and hope we can have Scala 2.13.11 back after this PR. |
The UT failure is irrelevant one which is tracked here (#42908). |
Thanks @dongjoon-hyun @HyukjinKwon @srowen , I will re-upgrade Scala 2.13.11 later |
What changes were proposed in this pull request?
This pr downgrade
scala-maven-plugin
to version 4.7.1 to avoid it automatically adding the-release
option as a Scala compilation argument.Why are the changes needed?
The
scala-maven-plugin
versions 4.7.2 and later will try to automatically append the-release
option as a Scala compilation argument when it is not specified by the user:-release
option for Scala versions 2.13.9 and higher.-release
option for Scala versions 2.12.x/2.13.x/3.1.1, and append-java-output-version
for Scala 3.1.2.The addition of the
-release
option has caused issues mentioned in SPARK-44376 | #41943 and #40442 (comment). This is because the-release
option has stronger compilation restrictions than-target
, ensuring not only bytecode format, but also that the API used in the code is compatible with the specified version of Java. However, many APIs in thesun.*
package are notexports
in Java 11, 17, and 21, such assun.nio.ch.DirectBuffer
,sun.util.calendar.ZoneInfo
, andsun.nio.cs.StreamDecoder
, making them invisible when compiling across different versions.For discussions within the Scala community, see scala/bug#12643, scala/bug#12824, scala/bug#12866, but this is not a bug.
I have also submitted an issue to the
scala-maven-plugin
community to discuss the possibility of adding additional settings to control the addition of the-release
option: davidB/scala-maven-plugin#722.For Apache Spark 4.0, in the short term, I suggest downgrading
scala-maven-plugin
to version 4.7.1 to avoid it automatic adding the-release
option as a Scala compilation argument. In the long term, we should reduce use of APIs that are notexports
for compatibility with the-release
compilation option due to-target
already deprecated after Scala 2.13.9.Does this PR introduce any user-facing change?
No
How was this patch tested?
run
git revert 656bf36363c466b60d00452399994ccaaa654ed8
to revert to using Scala 2.13.11 and rundev/change-scala-version.sh 2.13
to change Scala to 2.13build/mvn clean install -DskipTests -Pscala-2.13 -X
to check the Scala compilation arguments.Before
After
After downgrading the version, the
-release
option should no longer appear in the compilation arguments.Was this patch authored or co-authored using generative AI tooling?
No