data4democracy / quorum Goto Github PK

Current state: only postgresql storage in implemented in storage

Add functionality to store data on aws

Modify start.sh to make it more user friendly!
There are a ton of sudo's, this seems to be a problem because of how many times you'd have to enter your password.

Get a yelp scraper going on here:

• https://github.com/Yelp/yelp-python

• https://github.com/gfairchild/yelpapi

See:

• pre-implemented algrithms.
• non-negative matrix factorization

Incorporate this functionality into quorum.

create a social media analytics dashboard to keep track of trends, mentions, shared urls, etc.

Right now the way to get this thing running is by using the start.sh get rid of this and make a proper makefile.

Jobs are scheduled by adding an entry in scheduler/run_tasks.py. It would be nice to make this more flexible (i.e., I only want the twitter scraper to run).

Consider creating a docker-compose in the base directory that spins up all the docker containers instead of a shell script that calls docker-compose in each folder.

centralize our nlp efforts:

• assemble: data parsing and cleaning

Work on building models for all scrapers.

Improve the documentation

• How to use this app
• how to plug in new modules using kafka-python

Incorporate this work https://github.com/justforwhimsy/d4d_scripts
done for just-politics

Instead of Pip install in the dockerfiles, have each dockerfile do a pip install -r requirements.txt. In the requirements.txt, make sure the package has a version specified
Example:
facebook/dockerfile has the following lines
RUN pip install facebook-sdk
RUN pip install facepy

those packages should be in the requirements.txt as
facebook-sdk== version
facepy== version

There are a lot more things that can be done with twitter

• get past the 3200 tweet limit

• scrape hashtags using the REST API

• Tweet streaming

gitignore_global, when set, goes across all projects/repos on a users computer. This will probably causes issues with anyone who wants to contribute to this project. Suggest removing it and keeping the .gitignore

Moving this Issues Construct word2vec model with tweets here with the hopes of adding the twitter functionality and expanding the coverage to all other scrapers.