Scrape FB group info from AITD site about ati-broadcastapp HOT 9 CLOSED

The basics of this turned out to be pretty easy, since Indivisible has a JSON file of all the groups https://d1icy8tq23460g.cloudfront.net/output/indivisible.json on the map that includes a latitude and longitude for each group. After filtering the JSON to limit it to groups (it includes meetings and events too), I was able to run it through the US Census Bureau's Geocoder to identify the state. About 30 ended up requiring manual intervention because the geocoder couldn't process them. I did this with some quick and dirty Python as a proof of concept.

It seems like it should be fairly straight-forward to write something up in Node that can be run periodically to update groups. We should also have a way to edit groups in the app for those cases where the states cannot be identified programmatically.

I've attached the resulting JSON file which has all the info from the Indivisible JSON plus state info for each group: indivisible_groups_with_state.json.txt

from ati-broadcastapp.

On the data structure, currently it's an array where each group is an object, so what state each group is in would have to be checked group by group. It would probably be better to have an array with states as the highest level object and each group as a child, so [{'Alabama': [{group 1}, {group2}, ...]}, {'Alaska': [{group 3}, {group 4} ,...]} ...] instead of [{group 1}, {group 2}, ...].

from ati-broadcastapp.

OK, I've cleaned up and reorganized the data a little bit. The previous JSON I uploaded included "Group Meetings," which I thought I'd filtered out. This new file has those meetings removed and I also filtered out all the groups that didn't have a Facebook page listed for the value of either the 'facebook' or 'url' keys. Additionally, each group is now a nested document within a state document, so that for each state, we only need to find one document in the database and return it rather than checking every group to see if they match.

Here's the new version pretty-printed, so it should be a little more readable: indivisible_groups2_human.json.txt

from ati-broadcastapp.

It looks nice. It might be good (once a scaffolding is merged so that you have a place to put the file) if you write a file with a function that automatically retrieves this from the Indivisible website.
On a slightly different note, I wonder how we will store this in the Mongo backend. In particular, will we split it by state? If we only ever retrieve the data as the whole chunk for a given state, that works fine; however, if we ever need to do anything else, that will probably not be best.

from ati-broadcastapp.

Yep. Code to automatically fetch and update the information is what I'm working on currently.

It's probably best to amend the structure I proposed to be: { state: 'Alabama', groups: [ ...] } instead of the current {'Alabama': []}, to make querying cleaner, so I'll do that.

As far as storage in Mongo, I'm open to different options. I think grouping them by state in some way is the best approach, since our primary use case of broadcasting a message to Facebook groups within a specific state. By having all the groups clustered by state, we can fetch all the groups we need with a single, simple query. If we set up each state as a collection, the query would be something like db.collection('Oklahoma').find(), which would return all the group documents in the specified collection. If we want to have each state be a document within a collection (which might be better for if there's ever a desire to do national broadcasts), it could be db.collection('States').find( {state: 'Oklahoma'}), which would return the Oklahoma document and the group documents nested within.

Grouping them in state-level collections seems like the cleaner, but possibly less flexible approach. It would probably also make queries to access individual groups slightly cleaner. Grouping them as nested documents may offer more flexibility for adding national-level functionality down the road and, judging from the documentation, Mongo easily supports querying nested documents if we need to make changes to individual groups.

from ati-broadcastapp.

I don't like the idea of having the states be actual collections. I think we can start by having them be one collection, with the the groups contained within state containers (i.e. like db.collection('States').find( {state: 'Oklahoma'})). I agree that nesting the documents won't be a problem. I don't think this will be set in stone anyway.

from ati-broadcastapp.

from ati-broadcastapp.

Sounds like a plan.

from ati-broadcastapp.

Closed in favor of #20

from ati-broadcastapp.