spark-on-mesos wordcount-example
Simple word count example using spark and running it on mesos. For basic configuration to run spark on mesos refer
Clone the repo
git clone https://github.com/mahesh2492/spark-on-mesos.git
cd spark-on-mesos
Build the code
If this is your first time running SBT, you have to download the used dependencies.
cd spark-on-mesos
sbt clean compile
To run application locally, do following things:
First change the spark master in application.conf and run following command
sbt run
To run it on mesos, first create the jar to deploy. For that run following command
sbt assembly
It will create the fat jar of wordcount application.
Run mesos master and slave
cd build
./bin/mesos-master.sh --ip=127.0.0.1 --work_dir=/tmp/mesos
./bin/mesos-slave.sh --master=127.0.0.1:5050 --work_dir=/tmp/mesos --no-systemd_enable_support
Deploy the jar using spark-submit
cd spark-2.2.0-bin-hadoop2.7
./bin/spark-submit --class "rdd.WordCount" --master mesos://127.0.0.1:5050 file://path/target/scala-2.11/spark-on-mesos-assembly-1.0.jar