Problems with java 17 and mvn build.

[bjornjorgensen]

Hi, we at Apache Spark have merged version 4.8.1. We have some problems with this version.

with <scala-maven-plugin.version>4.8.0</scala-maven-plugin.version>
./build/mvn -DskipTests clean package

[INFO] Reactor Summary for Spark Project Parent POM 3.5.0-SNAPSHOT:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [ 2.993 s]
[INFO] Spark Project Tags ................................. SUCCESS [ 5.352 s]
[INFO] Spark Project Sketch ............................... SUCCESS [ 6.497 s]
[INFO] Spark Project Local DB ............................. SUCCESS [ 6.266 s]
[INFO] Spark Project Networking ........................... SUCCESS [ 8.869 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 7.967 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [ 9.179 s]
[INFO] Spark Project Launcher ............................. SUCCESS [ 4.749 s]
[INFO] Spark Project Core ................................. SUCCESS [01:55 min]
[INFO] Spark Project ML Local Library ..................... SUCCESS [ 22.125 s]
[INFO] Spark Project GraphX ............................... SUCCESS [ 26.436 s]
[INFO] Spark Project Streaming ............................ SUCCESS [ 39.594 s]
[INFO] Spark Project Catalyst ............................. SUCCESS [02:00 min]
[INFO] Spark Project SQL .................................. SUCCESS [02:39 min]
[INFO] Spark Project ML Common ............................ SUCCESS [ 19.772 s]
[INFO] Spark Project ML Library ........................... SUCCESS [01:46 min]
[INFO] Spark Project Tools ................................ SUCCESS [ 4.027 s]
[INFO] Spark Project Hive ................................. SUCCESS [ 58.614 s]
[INFO] Spark Project REPL ................................. SUCCESS [ 15.526 s]
[INFO] Spark Project Assembly ............................. SUCCESS [ 5.192 s]
[INFO] Kafka 0.10+ Token Provider for Streaming ........... SUCCESS [ 13.213 s]
[INFO] Spark Integration for Kafka 0.10 ................... SUCCESS [ 19.317 s]
[INFO] Kafka 0.10+ Source for Structured Streaming ........ SUCCESS [ 28.580 s]
[INFO] Spark Project Examples ............................. SUCCESS [ 33.875 s]
[INFO] Spark Integration for Kafka 0.10 Assembly .......... SUCCESS [ 11.099 s]
[INFO] Spark Avro ......................................... SUCCESS [ 25.772 s]
[INFO] Spark Project Connect Common ....................... SUCCESS [ 33.313 s]
[INFO] Spark Project Connect Server ....................... SUCCESS [ 27.918 s]
[INFO] Spark Project Connect Client ....................... SUCCESS [ 30.611 s]
[INFO] Spark Protobuf ..................................... SUCCESS [ 28.972 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 16:38 min
[INFO] Finished at: 2023-03-17T18:09:50+01:00
[INFO] ------------------------------------------------------------------------

with <scala-maven-plugin.version>4.8.1</scala-maven-plugin.version>
./build/mvn -DskipTests clean package

[INFO] --- maven-compiler-plugin:3.11.0:compile (default-compile) @ spark-core_2.12 ---
[INFO] Not compiling main sources
[INFO]
[INFO] --- scala-maven-plugin:4.8.1:compile (scala-compile-first) @ spark-core_2.12 ---
[INFO] Compiler bridge file: /home/bjorn/.sbt/1.0/zinc/org.scala-sbt/org.scala-sbt-compiler-bridge_2.12-1.8.0-bin_2.12.17__55.0-1.8.0_20221110T195421.jar
[INFO] compiler plugin: BasicArtifact(com.github.ghik,silencer-plugin_2.12.17,1.7.10,null)
[INFO] compiling 597 Scala sources and 103 Java sources to /home/bjorn/github/spark/core/target/scala-2.12/classes ...
[ERROR] [Error] /home/bjorn/github/spark/core/src/main/scala/org/apache/spark/serializer/SerializationDebugger.scala:71: not found: value sun
[ERROR] [Error] /home/bjorn/github/spark/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:26: not found: object sun
[ERROR] [Error] /home/bjorn/github/spark/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:27: not found: object sun
[ERROR] [Error] /home/bjorn/github/spark/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:206: not found: type DirectBuffer
[ERROR] [Error] /home/bjorn/github/spark/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:210: not found: type Unsafe
[ERROR] [Error] /home/bjorn/github/spark/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:212: not found: type Unsafe
[ERROR] [Error] /home/bjorn/github/spark/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:213: not found: type DirectBuffer
[ERROR] [Error] /home/bjorn/github/spark/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:216: not found: type DirectBuffer
[ERROR] [Error] /home/bjorn/github/spark/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:236: not found: type DirectBuffer
[ERROR] [Error] /home/bjorn/github/spark/core/src/main/scala/org/apache/spark/util/ClosureCleaner.scala:452: not found: value sun
[ERROR] [Error] /home/bjorn/github/spark/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:26: not found: object sun
[ERROR] [Error] /home/bjorn/github/spark/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:99: not found: type SignalHandler
[ERROR] [Error] /home/bjorn/github/spark/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:99: not found: type Signal
[ERROR] [Error] /home/bjorn/github/spark/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:83: not found: type Signal
[ERROR] [Error] /home/bjorn/github/spark/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:108: not found: type SignalHandler
[ERROR] [Error] /home/bjorn/github/spark/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:108: not found: value Signal
[ERROR] [Error] /home/bjorn/github/spark/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:114: not found: type Signal
[ERROR] [Error] /home/bjorn/github/spark/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:116: not found: value Signal
[ERROR] [Error] /home/bjorn/github/spark/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:128: not found: value Signal
[ERROR] 19 errors found
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Spark Project Parent POM 3.5.0-SNAPSHOT:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [ 3.848 s]
[INFO] Spark Project Tags ................................. SUCCESS [ 12.106 s]
[INFO] Spark Project Sketch ............................... SUCCESS [ 10.685 s]
[INFO] Spark Project Local DB ............................. SUCCESS [ 8.743 s]
[INFO] Spark Project Networking ........................... SUCCESS [ 9.362 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 7.828 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [ 9.071 s]
[INFO] Spark Project Launcher ............................. SUCCESS [ 4.776 s]
[INFO] Spark Project Core ................................. FAILURE [ 17.228 s]
[INFO] Spark Project ML Local Library ..................... SKIPPED
[INFO] Spark Project GraphX ............................... SKIPPED
[INFO] Spark Project Streaming ............................ SKIPPED
[INFO] Spark Project Catalyst ............................. SKIPPED
[INFO] Spark Project SQL .................................. SKIPPED
[INFO] Spark Project ML Common ............................ SKIPPED
[INFO] Spark Project ML Library ........................... SKIPPED
[INFO] Spark Project Tools ................................ SKIPPED
[INFO] Spark Project Hive ................................. SKIPPED
[INFO] Spark Project REPL ................................. SKIPPED
[INFO] Spark Project Assembly ............................. SKIPPED
[INFO] Kafka 0.10+ Token Provider for Streaming ........... SKIPPED
[INFO] Spark Integration for Kafka 0.10 ................... SKIPPED
[INFO] Kafka 0.10+ Source for Structured Streaming ........ SKIPPED
[INFO] Spark Project Examples ............................. SKIPPED
[INFO] Spark Integration for Kafka 0.10 Assembly .......... SKIPPED
[INFO] Spark Avro ......................................... SKIPPED
[INFO] Spark Project Connect Common ....................... SKIPPED
[INFO] Spark Project Connect Server ....................... SKIPPED
[INFO] Spark Project Connect Client ....................... SKIPPED
[INFO] Spark Protobuf ..................................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:24 min
[INFO] Finished at: 2023-03-17T18:13:26+01:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:4.8.1:compile (scala-compile-first) on project spark-core_2.12: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:4.8.1:compile failed: org.apache.commons.exec.ExecuteException: Process exited with an error: 255 (Exit value: 255) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :spark-core_2.12

However when using ./build/mvn -DskipTests -Djava.version=17 package without java 11 on my system it works.

For more info apache/spark#40442

[slandelle]

Hi,

The following comments only engage myself (I'm the most active maintainer here atm, not the original author).

This is a general advice to all Apache Spark users/maintainers, as I've always observed the same behavior over the years.

How to report a bug

The proper way to report a bug is to provide a good reproducer.
There are several good definitions for what a good reproducer is, including Wikipedia's and Stack Overflow's .

Spark's build is the exact opposite of a minimal reproducer.
Only someone with an interest in Spark would be willing to spend time figuring out how things work there, which I'm not.

Contributing

IMO, when some dependency becomes critical for an organization with decent means, it's the duty of this organization to contribute, not just consume.

So far, the total amount of contributions here from the Spark community has been zero.

Spark is hundreds of contributors and a super rich backing company: Databricks.

In comparison, this project has been for years a one-man on-personal-time army.

Making things very clear: I'm not asking for money, I'm asking for manpower.

[bjornjorgensen]

@slandelle I did ask chatgpt about the code in 7ce1e01 and the problem her..

this is what it return

private static String computeTargetOption(String target) {
  if (target.equals("1.5") || target.equals("5")) {
    return "jvm-1.5";
  } else if (target.equals("1.6") || target.equals("6")) {
    return "jvm-1.6";
  } else if (target.equals("1.7") || target.equals("7")) {
    return "jvm-1.7";
  } else if (target.equals("1.8") || target.equals("8")) {
    return "jvm-1.8";
  }
  return null;
}

private static String computeReleaseOptionFromTarget(String target) {
  if (target.equals("1.5")) {
    return "5";
  } else if (target.equals("1.6")) {
    return "6";
  } else if (target.equals("1.7")) {
    return "7";
  } else if (target.equals("1.8")) {
    return "8";
  }
  return target;
}

// visible for tests
static List<String> computeBytecodeVersionOptions(
    String target, String release, VersionNumber scalaVersion) {
  List<String> options = new ArrayList<>();
  boolean targetIsDefined = StringUtils.isNotEmpty(target);
  boolean releaseIsDefined = StringUtils.isNotEmpty(release);
  boolean releaseIsSupported = scalaVersion.compareTo(new VersionNumber("2.12.0")) >= 0;
  String releaseOrJavaOutputVersionOptionName =
      scalaVersion.compareTo(new VersionNumber("3.1.2")) >= 0
          ? "-java-output-version"
          : "-release";

  if (releaseIsDefined && releaseIsSupported) {
    // release's default is "maven.compiler.release"'s default, which is null
    options.add(releaseOrJavaOutputVersionOptionName);
    options.add(release);
  } else if (targetIsDefined) {
    // target's default is "maven.compiler.target"'s default, which is 1.8
    if (releaseIsSupported) {
      options.add(releaseOrJavaOutputVersionOptionName);
      options.add(computeReleaseOptionFromTarget(target));
    } else {
      String correctTarget = computeTargetOption(target);
      if (correctTarget != null) {
        options.add("-target:" + correctTarget);
      }
    }
  }
  return options;
}

protected List<String> getScalacOptions() throws Exception {
  List<String> options = new ArrayList<>();
  if (args != null) Collections.addAll(options, args);
  if (StringUtils.isNotEmpty(addScalacArgs)) {
    Collections.addAll(options, StringUtils.split(addScalacArgs, "|"));
  }
  options.addAll(getCompilerPluginOptions());
  options.addAll(computeBytecodeVersionOptions(target, release, findScalaVersion()));

  return options;
}

Can this code help? If so I can try to open a PR with the code.

[slandelle]

I did ask chatgpt about the code in 7ce1e01 and the problem her..

Our profession is doomed.

Again, the first step is to provide a minimal reproducer of your problem.

[eejbyfeldt]

I looked into this a bit and I think the underlying issue is scala/bug#12824