-
Notifications
You must be signed in to change notification settings - Fork 654
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Yarn Support On HDP 2.6.0.3-8 with Spark 2.xx #905
Comments
I was able to solve the problem by adding the hdp.version property through Ambari.
These steps solved the problem for me, hope they help you as well! |
Oh and a side note, if you are having problems deploying on yarn you should check your environment variables with I manually add the environment variables before starting spark-notebook (at the moment) with a bash script that I name env.sh and run using You can find these environment variables by looking in your configs via Ambari or you can go into the
directory (this is an example, yours may be different) and search through the various directories for your specific paths. Here is the env.sh script I created (I added a few non-essential environment variables just to be safe)
To recap, you must have these environment variables loaded before running spark-notebooks or you must add them into the initialize script in spark-notebooks so that they auto-load upon deployment. If you are running spark on yarn in a hadoop cluster you should be familiar with everything I am saying and if not I suggest doing research on your configs, their locations, and spark on yarn deployment. |
feel free to improve the docs by making a pull request, e.g. https://github.com/spark-notebook/spark-notebook/blob/master/docs/clusters_clouds.md P.S. to run apps on YARN cluster you also need if you wanna force this for ALL notebooks you may set it like this: |
It seems that using spark-notebook on a newer versions of hortonworks hadoop and spark will cause an error to be thrown when attempting to use with the yarn-manager in cluster mode. I was getting a "bad substitution" error (viewed from the yarn scheduler portal).
The text was updated successfully, but these errors were encountered: