Gluten 1.1.1 does not work on Spark 3.3.1 on EMR

[astarrr]

Backend

VL (Velox)

Bug description

When running a simple query like df.count() the spark job stops working with NoSuchMethodError:org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec.isSubquery

I traced the error and this is to be address to the implementation of AdaptiveSparkPlanExec in spark-sql_2.12-3.3.1-amzn-0.1.jar which does not provide the isSubquery() method/field (I decompiled the JAR).

The method is called at GlutenExplainUtils.scala:93.
Switching to Gluten 1.1.0, there is no callback to AdaptiveSparkPlanExec.isSubquery and Gluten does work also in EMR.

Spark version

Spark-3.3.x

Spark configurations

No response

System information

No response

Relevant logs

java.lang.NoSuchMethodError: org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec.isSubquery()Z
  at org.apache.spark.sql.execution.GlutenExplainUtils$.$anonfun$collectFallbackNodes$1(GlutenExplainUtils.scala:93)
  at org.apache.spark.sql.execution.GlutenExplainUtils$.$anonfun$collectFallbackNodes$1$adapted(GlutenExplainUtils.scala:80)
  at org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:410)
  at org.apache.spark.sql.execution.GlutenExplainUtils$.collect$1(GlutenExplainUtils.scala:80)
  at org.apache.spark.sql.execution.GlutenExplainUtils$.collectFallbackNodes(GlutenExplainUtils.scala:112)
  at org.apache.spark.sql.execution.GlutenExplainUtils$.$anonfun$processPlan$8(GlutenExplainUtils.scala:223)
  at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
  at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
  at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
  at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
  at scala.collection.TraversableLike.map(TraversableLike.scala:286)
  at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
  at scala.collection.AbstractTraversable.map(Traversable.scala:108)
  at org.apache.spark.sql.execution.GlutenExplainUtils$.processPlan(GlutenExplainUtils.scala:220)
  at org.apache.spark.sql.execution.GlutenFallbackReporter.postFallbackReason(GlutenFallbackReporter.scala:108)
  at org.apache.spark.sql.execution.GlutenFallbackReporter.apply(GlutenFallbackReporter.scala:45)
  at org.apache.spark.sql.execution.GlutenFallbackReporter.apply(GlutenFallbackReporter.scala:36)
  at io.glutenproject.extension.ColumnarOverrideRules.$anonfun$transformPlan$3(ColumnarOverrides.scala:725)
  at scala.collection.LinearSeqOptimized.foldLeft(LinearSeqOptimized.scala:126)
  at scala.collection.LinearSeqOptimized.foldLeft$(LinearSeqOptimized.scala:122)
  at scala.collection.immutable.List.foldLeft(List.scala:91)
  at io.glutenproject.extension.ColumnarOverrideRules.$anonfun$transformPlan$1(ColumnarOverrides.scala:723)
  at io.glutenproject.metrics.GlutenTimeMetric$.withNanoTime(GlutenTimeMetric.scala:41)
  at io.glutenproject.metrics.GlutenTimeMetric$.withMillisTime(GlutenTimeMetric.scala:46)
  at io.glutenproject.extension.ColumnarOverrideRules.transformPlan(ColumnarOverrides.scala:733)
  at io.glutenproject.extension.ColumnarOverrideRules.$anonfun$postColumnarTransitions$2(ColumnarOverrides.scala:712)
  at io.glutenproject.utils.QueryPlanSelector.maybe(QueryPlanSelector.scala:74)
  at io.glutenproject.extension.ColumnarOverrideRules.io$glutenproject$extension$ColumnarOverrideRules$$$anonfun$postColumnarTransitions$1(ColumnarOverrides.scala:699)

[astarrr]

I upload some screenshots.
Here with the Amazon jar there is no method isSubquery.

Here with the classic Spark:

[FelixYBW]

Do you mean AWS EMR? I don't think Gluten can work with AWS EMR. Gluten relies on many Spark APIs and hacked several Spark files, AWS EMR have much optimizations to Spark which may change APIs and conflict with files we hacked, which needs to be solved by AWS EMR team. As far as I know, there is no such effort in AWS.