This repo serves as reference and knowledgebase for the work completed during the Fall of 2019 by Clive Unger, Ishank Arora, and Andy Chang in the UT Computational Media Lab

Setting up Python 3, Jupyter Notebook, and Spark 2.4 on TACC

Running a batch job on TACC (This documentation is for Stampede2, but the same process applies for any TACC resource)

The dataset is large enough that it can't be fit into memory, so there are many challenges that come with that. We have written up a comprehensive log of the various challenges we faced and how they were resolved. In addition, we also had various logistical challenges that we thought should be mentioned.

Technical Challenges

Logistical Challenges

Reference to the data locations in TACC since data cannot be uploaded to GitHub

The Data

• Loading the Data in Spark
• Flattening the Data
• Creating a Graph Representation of the Data
• Basic Graph Manipulation with Spark and Graphframes
• Clustering Coefficient example (which turned out to be incorrect)
• Splitting the data per month
• SNAP
• NetworKit

• TACC Wrangler User Guide
• Apache Spark Documentation
• GraphFrames Documentation
• NetworkX Documention
• SNAP Documentation
• NetworKit Documentation