Apache-spark GPU installation with RAPIDS

[chpiero]

sorry i'm new to the world spark, I should install it on a cluster that has a node with 2 nvidia AMPERE GPUs, I find written in this NVIDIA link in the Install Spark paragraph that: "note that only scala version 2.12 is currently supported by the accelerator."

I would like to install the latest version of Spark 3.3 which supports from what I see scale 2.13, however, I will only work with python. What I don't understand is that if this NVIDIA plugin only supports scale 2.12 but all other versions of other programming languages do. Therefore I have no problem running python on GPUs, I have to use scale and version 2.12 only. to be able to work with GPUs and therefore python is not possible to use?

thanks

[abellina]

@chpiero, thanks for your question.

Spark supports both Scala 2.13 and Scala 2.12, but Scala 2.12 is still the default (https://github.com/apache/spark/blob/master/pom.xml#L163). It is also the default for the official package download here: https://spark.apache.org/downloads.html, but Scala 2.13 is also provided as another option.

From conversation in this jira issue (https://issues.apache.org/jira/browse/SPARK-25075), it seems the move to Scala 2.13 as default is not yet scheduled, it looks like perhaps for Spark 4+.

In terms of python/scala. Scala is still used to compile all the spark core code and pyspark is for the most part calling into this code via JVM hooks. So the version of scala is not very relevant for a pyspark exclusive job. If your program was written in scala instead of python, then I could see the scala version coming into play. Is there something that is forcing you to choose Scala 2.13? If so I'd love to learn more.

Irrespective of language, we do target specific functionality in Spark summarized here: https://nvidia.github.io/spark-rapids/docs/FAQ.html#what-parts-of-apache-spark-are-accelerated, with all the details here: https://nvidia.github.io/spark-rapids/docs/supported_ops.html. In a nutshell, a big focus of the plugin is ETL/SQL operations (e.g. dataframe API), which are accessible from python, scala, java, sql, and R.

Does the above help for you to get started?

[chpiero]

many thanks for you suggestion,
we just bought a dell r650xs and i installed ubuntu 22.04 and i will use it as master instead as worker for now we got a del 750 with 2 nvidia amps inside, i would use spark to check the queues and convinced me i will install all barematel. Thank you

[abellina]

@chpiero I am sorry but I overlooked the "2 gpus in 1 box" part of your setup. We need to have 1 process per GPU (1 executor per GPU), so a local mode setup would only be able to use 1 GPU.

A simple way to allow you to use both GPUs is with a single node standalone cluster, I've put a script together that should help take you through the steps. I went through these instructions https://nvidia.github.io/spark-rapids/docs/get-started/getting-started-on-prem.html#spark-standalone-cluster, and tried to script them. Hope it helps.

# set this to the path where you untar the spark-3.3.0 archive
export SPARK_HOME=$HOME/spark-3.3.0-bin-hadoop3

# set this to the path where you downloaded the plugin jar
export SPARK_RAPIDS_PLUGIN_JAR=$HOME/rapids-4-spark_2.12-22.08.0.jar

# This is to configure the worker to see 2 GPUs, note you really only need to do this once
# added it in this script for convenience
cp $SPARK_HOME/conf/spark-env.sh.template $SPARK_HOME/conf/spark-env.sh

# This script (getGpuResources) allows Spark to interact with `nvidia-smi` to find the two GPUs in your box
export GPU_DISCOVERY_SCRIPT=$SPARK_HOME/examples/src/main/scripts/getGpusResources.sh
echo "SPARK_WORKER_OPTS=\"-Dspark.worker.resource.gpu.amount=2 -Dspark.worker.resource.gpu.discoveryScript=$GPU_DISCOVERY_SCRIPT\"" >> $SPARK_HOME/conf/spark-env.sh

# note that this starts the master in the loopback device
# if you move the master to a different machine, it's ok to just
# know what that machine's hostname is and use that in place of
# "127.0.0.1" below
export SPARK_MASTER_HOST=127.0.0.1
$SPARK_HOME/sbin/start-master.sh -h $SPARK_MASTER_HOST

$SPARK_HOME/sbin/start-worker.sh spark://${SPARK_MASTER_HOST}:7077

# starts a standalone shell that uses 2 gpus and 4 CPU cores per executor.
# Our requirement is 1 executor per GPU, so this launches 2 executors:
# spark.cores.max/spark.executor.cores.
# Refer to https://nvidia.github.io/spark-rapids/docs/tuning-guide.html for more info 
# on some of the configs we recommend for performance. I did add some pinned 
# memory (2GB) which is host memory that makes host to device copies faster, 
# but it is not required.
$SPARK_HOME/bin/spark-shell \
  --master spark://${SPARK_MASTER_HOST}:7077 \
  --driver-memory 2G \
  --conf spark.cores.max=8 \
  --conf spark.executor.memory=4G \
  --conf spark.executor.cores=4 \
  --conf spark.executor.resource.gpu.amount=1 \
  --conf spark.task.resource.gpu.amount=0.25 \
  --conf spark.rapids.memory.pinnedPool.size=2G \
  --conf spark.plugins=com.nvidia.spark.SQLPlugin \
  --jars ${SPARK_RAPIDS_PLUGIN_JAR}

If you run nvidia-smi after the shell is up, you should see within a few seconds the two processes, i.e:

|=============================================================================|
|    0   N/A  N/A    455641      C   ...penjdk-amd64/jre/bin/java    14487MiB |
|    1   N/A  N/A    455642      C   ...penjdk-amd64/jre/bin/java    14487MiB |
+-----------------------------------------------------------------------------+

each attached to a different ampere card.

[chpiero]

thank you for the script, last question.

on the master device, I have no GPU, the GPUs are present only on the worker device. Should the script run on all devices (master and worker) or only on the master? Instead Apache-Spark I have to install it on both right?

this command, do I need to run to the worker or do I have to launch everything from the master? in this case, how does it know where the worker is??

$SPARK_HOME/sbin/start-worker.sh spark://${SPARK_MASTER_HOST}:7077

Thanks again

[abellina]

Here are some basic definitions, but there is a lot more over in Spark's documentation (https://spark.apache.org/docs/latest/spark-standalone.html).

For the type of simple cluster we have here (standalone):

• Master: is a process that is in charge of keeping track of all the workers. It is also the point of entry for your application, where the master decides which workers your application will run on. In this case we have a very simple setup, 1 Master and 1 Worker, so there is only one choice.

• Worker: this process connects to the master, and there can be several workers. This is why when we started the worker we gave it the spark master URL spark://{SPARK_MASTER_HOST}:7077, so there is a persistent connection between the worker(s) and master processes. When the worker is told it needs to be part of an application it starts another process, called the "executor".

There is also such a thing as the "driver". The driver is a process and for simplicity in this setup it lives in the machine where you started the spark shell. Spark starts the driver and it glues everything together because we told spark to use a master at --master. So the driver connects to the master, the master asks the workers to start executors, and the executors are told how to connect to your driver by Spark. You don't have to worry about these connections.

Now onto your question: in the machine where you run the shell (driver) you don't need a GPU. The GPUs are only needed in the worker hosts, they are used by the executors only.

The command you pasted can be started from the worker node pretty easily. It knows where the master is because you need to set SPARK_MASTER_HOST to the appropriate hostname, the rest is handled by the Spark. The Spark documentation mentions a start-workers.sh script here (https://spark.apache.org/docs/latest/spark-standalone.html#cluster-launch-scripts), this you could run from the master node if you edit the conf/workers file (first copy conf/workers.template to conf/workers).

Also note that my script invoked spark-shell. You just want to change that to pyspark so you get a python shell.

[chpiero]

ok,

I will install the apache-spark and rapids plugin only on the master and modify workers so that it finds it automatically.

thank you very very much