Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can not run examples under Spark 2.4 #186

Open
GangLiCN opened this issue Jan 16, 2019 · 0 comments
Open

Can not run examples under Spark 2.4 #186

GangLiCN opened this issue Jan 16, 2019 · 0 comments

Comments

@GangLiCN
Copy link

GangLiCN commented Jan 16, 2019

Spark-Bench version (version number, tag, or git commit hash)

2.3.0

Livy Server version

0.5

Details of your cluster setup (Spark version, Standalone/Yarn/Local/Etc)

Spark version: 2.4.0 (installed from "spark-2.4.0-bin-hadoop2.7.tgz")
type: Single node, no hadoop, just installed Livy server

Scala version on your cluster

2.11.12

Your exact configuration file (with system details anonymized for security)

The content of "local-livy-example.conf":

spark-bench = {
spark-submit-config = [{
livy = {
url = "<my_spark_server_ip>:8998"
poll-seconds = 1
}
spark-bench-jar = "/root/packages/apache/spark/spark-bench_2.3.0_0.4.0-RELEASE/lib/spark-bench-2.3.0_0.4.0-RELEASE.jar"
workload-suites = [
{
descr = "One run of SparkPi and that's it!"
benchmark-output = "console"
workloads = [
{
name = "sparkpi"
slices = 10
}
]
}
]
}]
}

Relevant stacktrace

Exception in thread "main" com.ibm.sparktc.sparkbench.utils.SparkBenchException: REST call returned non-sucess code: 400
at com.ibm.sparktc.sparkbench.sparklaunch.submission.livy.LivySubmit$$anonfun$2.apply(LivySubmit.scala:37)
at com.ibm.sparktc.sparkbench.sparklaunch.submission.livy.LivySubmit$$anonfun$2.apply(LivySubmit.scala:37)
at com.ibm.sparktc.sparkbench.sparklaunch.submission.livy.LivySubmit$.sendPostBatchRequest(LivySubmit.scala:61)
at com.ibm.sparktc.sparkbench.sparklaunch.submission.livy.LivySubmit.launch(LivySubmit.scala:102)
at com.ibm.sparktc.sparkbench.sparklaunch.SparkLaunch$.com$ibm$sparktc$sparkbench$sparklaunch$SparkLaunch$$launch$1(SparkLaunch.scala:57)
at com.ibm.sparktc.sparkbench.sparklaunch.SparkLaunch$$anonfun$launchJobs$2.apply(SparkLaunch.scala:65)
at com.ibm.sparktc.sparkbench.sparklaunch.SparkLaunch$$anonfun$launchJobs$2.apply(SparkLaunch.scala:65)
at scala.collection.immutable.List.foreach(List.scala:381)
at com.ibm.sparktc.sparkbench.sparklaunch.SparkLaunch$.launchJobs(SparkLaunch.scala:65)
at com.ibm.sparktc.sparkbench.sparklaunch.SparkLaunch$.main(SparkLaunch.scala:38)
at com.ibm.sparktc.sparkbench.sparklaunch.SparkLaunch.main(SparkLaunch.scala)
Caused by: java.lang.Exception
at com.ibm.sparktc.sparkbench.utils.SparkBenchException$.apply$default$2(SparkBenchException.scala:22)
... 11 more

Description of your problem and any other relevant info

I can't run example(e.g. local-livy-example.conf) using "spark-bench", detailed error stack is as above.
Not sure if it's a compatibility issue between livy server and speark-bench.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant