Workshops for Cloud Systems Coursework

Build a 2 stage data-pipeline that imports MovieLens rating dataset to produce list of worst rated movies of all time

Download MovieLens DataSet ml-latest-small.zip from https://grouplens.org/datasets/movielens

wget https://grouplens.org/datasets/movielens/ml-latest-small.zip
unzip ml-latest-small.zip

Input: data/ratings.csv via stdin Output: Sequence of tuples (movieId, rating) for all users

Input: Output of Stage 1 Output: Increasing order of sequence of inverse tuples (rating, movieId)

Steps to follow: https://github.com/MicrosoftDocs/azure-docs/blob/981f98cd0fd7ac267142f3d43836cbc275805f5e/articles/hdinsight/hdinsight-hadoop-streaming-python.md

Stage1: STAGEID=1 yarn hadoop-streaming.jar -files mapper_stage$STAGEID.py,reducer_stage$STAGEID.py -mapper mapper_stage$STAGEID.py -reducer reducer_stage$STAGEID.py -input data/ratings.csv -output data/avg_rating

Stage2: STAGEID=2 yarn hadoop-streaming.jar -files mapper_stage$STAGEID.py,reducer_stage$STAGEID.py -mapper mapper_stage$STAGEID.py -reducer reducer_stage$STAGEID.py -input data/avg_rating -output data/sort_rating