This repo serves as reference and knowledgebase for the work completed during the Fall of 2019 by Clive Unger, Ishank Arora, and Andy Chang in the UT Computational Media Lab
Setting up Python 3, Jupyter Notebook, and Spark 2.4 on TACC
Running a batch job on TACC (This documentation is for Stampede2, but the same process applies for any TACC resource)
The dataset is large enough that it can't be fit into memory, so there are many challenges that come with that. We have written up a comprehensive log of the various challenges we faced and how they were resolved. In addition, we also had various logistical challenges that we thought should be mentioned.
Reference to the data locations in TACC since data cannot be uploaded to GitHub
- Loading the Data in Spark
- Flattening the Data
- Creating a Graph Representation of the Data
- Basic Graph Manipulation with Spark and Graphframes
- Clustering Coefficient example (which turned out to be incorrect)
- Splitting the data per month
- SNAP
- NetworKit