You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Exception in thread "main" com.ibm.sparktc.sparkbench.utils.SparkBenchException: REST call returned non-sucess code: 400
at com.ibm.sparktc.sparkbench.sparklaunch.submission.livy.LivySubmit$$anonfun$2.apply(LivySubmit.scala:37)
at com.ibm.sparktc.sparkbench.sparklaunch.submission.livy.LivySubmit$$anonfun$2.apply(LivySubmit.scala:37)
at com.ibm.sparktc.sparkbench.sparklaunch.submission.livy.LivySubmit$.sendPostBatchRequest(LivySubmit.scala:61)
at com.ibm.sparktc.sparkbench.sparklaunch.submission.livy.LivySubmit.launch(LivySubmit.scala:102)
at com.ibm.sparktc.sparkbench.sparklaunch.SparkLaunch$.com$ibm$sparktc$sparkbench$sparklaunch$SparkLaunch$$launch$1(SparkLaunch.scala:57)
at com.ibm.sparktc.sparkbench.sparklaunch.SparkLaunch$$anonfun$launchJobs$2.apply(SparkLaunch.scala:65)
at com.ibm.sparktc.sparkbench.sparklaunch.SparkLaunch$$anonfun$launchJobs$2.apply(SparkLaunch.scala:65)
at scala.collection.immutable.List.foreach(List.scala:381)
at com.ibm.sparktc.sparkbench.sparklaunch.SparkLaunch$.launchJobs(SparkLaunch.scala:65)
at com.ibm.sparktc.sparkbench.sparklaunch.SparkLaunch$.main(SparkLaunch.scala:38)
at com.ibm.sparktc.sparkbench.sparklaunch.SparkLaunch.main(SparkLaunch.scala)
Caused by: java.lang.Exception
at com.ibm.sparktc.sparkbench.utils.SparkBenchException$.apply$default$2(SparkBenchException.scala:22)
... 11 more
Description of your problem and any other relevant info
I can't run example(e.g. local-livy-example.conf) using "spark-bench", detailed error stack is as above.
Not sure if it's a compatibility issue between livy server and speark-bench.
The text was updated successfully, but these errors were encountered:
Spark-Bench version (version number, tag, or git commit hash)
2.3.0
Livy Server version
0.5
Details of your cluster setup (Spark version, Standalone/Yarn/Local/Etc)
Spark version: 2.4.0 (installed from "spark-2.4.0-bin-hadoop2.7.tgz")
type: Single node, no hadoop, just installed Livy server
Scala version on your cluster
2.11.12
Your exact configuration file (with system details anonymized for security)
The content of "local-livy-example.conf":
spark-bench = {
spark-submit-config = [{
livy = {
url = "<my_spark_server_ip>:8998"
poll-seconds = 1
}
spark-bench-jar = "/root/packages/apache/spark/spark-bench_2.3.0_0.4.0-RELEASE/lib/spark-bench-2.3.0_0.4.0-RELEASE.jar"
workload-suites = [
{
descr = "One run of SparkPi and that's it!"
benchmark-output = "console"
workloads = [
{
name = "sparkpi"
slices = 10
}
]
}
]
}]
}
Relevant stacktrace
Exception in thread "main" com.ibm.sparktc.sparkbench.utils.SparkBenchException: REST call returned non-sucess code: 400
at com.ibm.sparktc.sparkbench.sparklaunch.submission.livy.LivySubmit$$anonfun$2.apply(LivySubmit.scala:37)
at com.ibm.sparktc.sparkbench.sparklaunch.submission.livy.LivySubmit$$anonfun$2.apply(LivySubmit.scala:37)
at com.ibm.sparktc.sparkbench.sparklaunch.submission.livy.LivySubmit$.sendPostBatchRequest(LivySubmit.scala:61)
at com.ibm.sparktc.sparkbench.sparklaunch.submission.livy.LivySubmit.launch(LivySubmit.scala:102)
at com.ibm.sparktc.sparkbench.sparklaunch.SparkLaunch$.com$ibm$sparktc$sparkbench$sparklaunch$SparkLaunch$$launch$1(SparkLaunch.scala:57)
at com.ibm.sparktc.sparkbench.sparklaunch.SparkLaunch$$anonfun$launchJobs$2.apply(SparkLaunch.scala:65)
at com.ibm.sparktc.sparkbench.sparklaunch.SparkLaunch$$anonfun$launchJobs$2.apply(SparkLaunch.scala:65)
at scala.collection.immutable.List.foreach(List.scala:381)
at com.ibm.sparktc.sparkbench.sparklaunch.SparkLaunch$.launchJobs(SparkLaunch.scala:65)
at com.ibm.sparktc.sparkbench.sparklaunch.SparkLaunch$.main(SparkLaunch.scala:38)
at com.ibm.sparktc.sparkbench.sparklaunch.SparkLaunch.main(SparkLaunch.scala)
Caused by: java.lang.Exception
at com.ibm.sparktc.sparkbench.utils.SparkBenchException$.apply$default$2(SparkBenchException.scala:22)
... 11 more
Description of your problem and any other relevant info
I can't run example(e.g. local-livy-example.conf) using "spark-bench", detailed error stack is as above.
Not sure if it's a compatibility issue between livy server and speark-bench.
The text was updated successfully, but these errors were encountered: