This is a general guide line about how to setup RAPIDS in RI2 cluster.
OS:CentOS 7.8
Java:1.8
Due to the oracle policy, one of the easy way to install java is downloading java on the local and upload it to Ri2 cluster via sftp or other methods. Notice that scala have a good support on java 8 and java 11, so it is recommend to download java 8 instead of the newest version of Java. Once the java binary uploaded and uncompressed to the cluster, it is recommended to create a folder in order to manage different java versions:
mkdir java
mkdir maven
mkdir scala
mkdir sbt
mkdir javalib
#you need to upload downloaded jdk tar to the cluster and unzip it
#get maven
wget http://apache.spinellicreations.com/maven/maven-3/3.6.3/binaries/apache-maven-3.6.3-bin.tar.gz ./maven
#get scala
wget -q http://downloads.lightbend.com/scala/2.12.1/scala-2.12.1.tgz ./scala
#get sbt
wget -q https://github.com/sbt/sbt/releases/download/v1.4.0/sbt-1.4.0.tgz ./sbt
#get spark
wget https://mirror.olnevhost.net/pub/apache/spark/spark-3.0.1/spark-3.0.1-bin-hadoop2.7.tgz ./javalib
#unzip compressed files
tar -xvzf ./maven/apache-maven-3.6.3-bin.tar.gz
tar -xvzf ./scala/scala-2.12.1.tgz
tar -xvzf ./sbt/sbt-1.4.0.tgz
tar -xvzf ./javalib/spark-3.0.1-bin-hadoop2.7.tgz
in .bashrc
or .bash_profile
append these lines:
#set environment variables
export SPARK_HOME=$HOME/javalib/spark-3.0.1-bin-hadoop2.7
export JAVA_HOME=$HOME/java/jdk1.8.0_261
export SCALA_HOME=$HOME/scala/scala-2.12.1
export SBT_HOME=$HOME/sbt/sbt
export MAVEN_HOME=$HOME/maven/apache-maven-3.6.3
export PATH=$PATH:$SPARK_HOME/bin:$JAVA_HOME/bin:$MAVEN_HOME/bin:$SCALA_HOME/bin:$SBT_HOME/bin
pull code from github repo:
git clone https://github.com/NVIDIA/spark-rapids.git
go to the directory of the repo and build the repo
mvn clean install -DSkipTest #some test will fail but it doesn't bother
the built jar for RAPIDS
and cuDF
can be also get by wget:
wget https://repo1.maven.org/maven2/ai/rapids/cudf/0.15/cudf-0.15.jar
wget https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/0.2.0/rapids-4-spark_2.12-0.2.0.jar
spark-shell --jars '/%path to jar%/rapids-4-spark_2.12-0.2.0.jar,/%path to jar%/cudf-0.15.jar' --conf spark.plugins=com.nvidia.spark.SQLPlugin --conf spark.rapids.sql.incompatibleOps.enabled=true
for more detailed configuration, please refer to the official guide of NVIDIA RAPIDS guide
for TPCxBB benchmarking, please refer to this guide
Allocate nodes for master and slaves, in this step, the Spark frame work is not yet started:
#run this command as many times as you want for the total number of nodes
srun --partition=batch --mem=16000 -n1 -N1 --pty --preserve-env $SHELL
mapping the localhost:8080 to spark-webui port:
ssh -N -L localhost:8080:10.3.1.XX:8080 [email protected]
configure $SPARK_HOME/conf/slaves
, for the initial installation, just create one file called slaves
we will using $SPARK_HOME/sbin/start-slaves.sh
to start all slave node at once:
# put the ip address of slaves
10.3.1.XX
10.3.1.XX
10.3.1.XX
10.3.1.XX
create spark-env.sh
#copy from the spark-env.sh.template
cp $SPARK_HOME/conf/spark-env.sh.template $SPARK_HOME/conf/spark-env.sh
add slave launch configuration, you may just find the corresponding lines and change the line, the amount should not exceed the total number of gpu that the cluster has no the worker node which allocated:
SPARK_WORKER_OPTS="-Dspark.worker.resource.gpu.amount=1 -Dspark.worker.resource.gpu.discoveryScript=$SPARK_HOME/conf/getGpusResources.sh"
copy GPU discovery script into $SPARK_HOME/conf/
:
cp $SPARK_HOME/examples/src/main/scripts/getGpusResources.sh $SPARK_HOME/conf/getGpusResources.sh
Start master node, master node don't have anything to do with GPU, but it is recommended to allocate the master node exclude header node:
$SPARK_HOME/sbin/start-master.sh
Start slave nodes, if GPU access is desired, each slave node should have at least one GPU:
$SPARK_HOME/sbin/start-slaves.sh
One important thing to check before submitting jobs is the log of each node, make sure everything is successfully connected. If any node fails, just use the stop-all.sh
to stop them and restart again, for more detailed guide, please refer to this guide.
Once you have the successful connection, then you can go to the localhost:8080
to see the status of the cluster, you should get a web page that is similar like this.
Two things important are here: 1) make sure all worker are show up on the webpage, 2) on the resource slot, the GPU should show up.
Till now, the setup part is done. in order to submit the job, refer this guide.