You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hey guys,
I've got Scala [2.11.8] Spark [2.1.0] Hadoop [2.8.0] {Hive ✓} on Ubuntu 16.04.
It is installed from zip archive. I'm running it from spark-notebook folder using ./bin/spark-notebook
I'm trying to add a simple jar through lib folder under spark-notebook.
And, the problem is it doesn't work.
Indeed, the adding the jar into the lib folder isn't enough, this is because the spark-notebook script/s are listing explicitly all jars added to the classpath.
Also, another problem will happen if you want to use these jars on the executors because they won't be added to the Spark configuration.
The recommended way would be (let me know if make sense, and worth improving):
Add the path in the var env EXTRA_CLASSPATH: it uses : as paths separator and will add the jar into the driver's classpath
Add the path in the var env ADD_JARS: this one uses , as paths separator and will the jar into the executors' classpath
So you could do something like:
# copy the jar
cp my-lib.jar lib
# launch with my-lib in both driver and executors classpaths
EXTRA_CLASSPATH=../lib/my-lib.jar ADD_JARS=../lib/my-lib.jar bin/spark-notebook
Hey guys,
I've got Scala [2.11.8] Spark [2.1.0] Hadoop [2.8.0] {Hive ✓} on Ubuntu 16.04.
It is installed from zip archive. I'm running it from spark-notebook folder using
./bin/spark-notebook
I'm trying to add a simple jar through
lib
folder under spark-notebook.And, the problem is it doesn't work.
Here's my source for jar:
sbt file:
Here's a result of import from jar located under spark-notebook/lib folder using
sbt console
And here's spark-notebook one
Interestingly it works for spark-notebook 0.6.3 and spark 1.6.3 version under Windows 10 with same notebook setup. (0.7.0+ wont work on windows :( )
Hope for your help.
Thanks in advance!
The text was updated successfully, but these errors were encountered: