Ireland Marine Data with Spark - Project for 521283S Big Data Processing and Applications, Spring 2023, @unioulu

Irish Marine Institute

• Digital Ocean
• Dataset Source

To run locally in a python environment:

• Create and activate environment

conda create -n spark
conda activate spark

OR

python -m venv venv
source ./venv/bin/activate # linux
.\venv\Scripts\activate # windows

• Install requirements and run

pip install -r requirements.xt
jupyter lab

To run locally in a single node docker instance:

docker run -it --rm -p 8888:8888 -v "<repo-location>:/home/jovyan/work" jupyter/pyspark-notebook:spark-3.4.0