Reddit Source data query on Google BigQuery about graphsage HOT 3 CLOSED

@williamleif : Do you have any smaller preprocessed data file for reddit data.
Or can you provide sources of data containing rich attribute information similar to reddit.

Thanks,

from graphsage.

Hi!

I actually used the raw json Reddit data dump (see links in the appendix) and did not query the BigQuery data. In our lab, we maintain an internal version of the Reddit data (identical to the data in BigQuery) but we store it in our own Hadoop filesystem. Thus, it is hard to say exactly what query you would need to fire, since I was not using BigQuery and was using Hadoop + Python.

However, the basic workflow was the following:

1. Count how many comments were in each subreddit in 2014.
2. Rank communities by # comments in 2014.
3. Select the communities with ranks [11,50] inclusive.
4. Retrieve all posts for these communities.
5. For each pair of posts, check if at least one user commented on both.

Step 5 might be expensive without Hadoop or a powerful server though..

from graphsage.

Thanks @williamleif . Its very clear now.

from graphsage.