Data analytics and machine learning are one of the most important applications of distributed computing. spark.ml is a new package introduced in Spark 1.2+, which aims to provide a uniform set of high-level APIs that help users create and tune practical machine learning pipelines.We wanted to learn and explore SparkML in detail. To better demonstrate the vast capabilities in spark.ml, three tutorials were created that cover both supervised and unsupervised techniques. Algorithms included are Random Forest, Bisecting k-means and LDA. Our project contained topics within both Categories A and B.

To demonstrate the power of Spark and AWS together and show how they leverage big data sets, the project completed customer segmentation of a Sun Country Airlines dataset. In concert with analyzing the customer segments and to provide a more complete business solution to Sun Country, topic modeling and sentiment analysis were conducted after scraping airline reviews from the web.

• Access project report

You can go through the following to access the tutorials

• Bisecting K-means
• Latent Dirichlet Allocation
• Random Forest

• Overview of the tutorial
• Setting up AWS on Qubole
• Setting up Spark & Jupyter on Google Cloud