alexanderjfink / miner Goto Github PK

Q - for Python querying SQL style of CSV files

http://pandas.pydata.org/

Need the ability to mark a specific data set by the location it provides coverage of... this will allow building features in the future in such a way as to allow downloading of various data depending on location.

https://github.com/noamraph/tqdm

Most datasets have a data dictionary that accompanies them in order to properly understand and interpret the data. There should be a way (included in the "maps" models) to download, view a summary of, etc., data dictionaries for datasets. They will also probably need to be stored outside of the storage databases since they are not relational data.

Couldn't figure out "load local data infile" in a cross-compatible and reliable way, so moved to individual sql inserts which is a pain and very slow comparatively. I'd like a faster way to do this.

Requires US census data to come back online...

Can use Stem to connect to and download through the Tor architecture https://stem.torproject.org/index.html. This should be developed after Issue #15, the plugin architecture

This is relatively complicated.

Take example data:

• Enron Emails (all separate email files in folders)
• Official Hospital data from Medicare (multiple csvs all in single folder)
• Form 990 data from CitizenAudit (csv manifest and separate set of files from 2012)
• US Census data (separate files for each state, sql sets up database and needs to be repaired before inserts can work)

Should install on different systems and pull necessary dependencies mdbtools and so forth. Should also have basic setup process for database used.

A basic plugin architecture should cover at least the map processing tools -- download, unpack, install, and clean up. A plugin, for example, could do of more ethnic to every downloaded file, could replace how the files are downloaded. It would also need to be able to add a CLI option for a plugin. Maybe even the database driver use could be managed by plugins?

The first potential plugin is in Issue #14. I'd like to do the downloading of data through the Tor network for anonymous and encrypted data. It would be an optional to use CLI option that would vastly slow down the downloading (its gonna take a while to download big data sets through Tor) but would make it secure and anonymous. For people downloading datasets that are in some way political, it might be best that it remain anonymous. Since it isn't part of the core mission of this project, I think it would be better to have it plug in but be separate.

I may try to use this basic plugin architecture:
http://martyalchin.com/2008/jan/10/simple-plugin-framework/

Or this one: http://yapsy.sourceforge.net/

More information that I found about how to build these (since I've never built a plugin architecture before) was here: http://stackoverflow.com/questions/932069/building-a-minimal-plugin-architecture-in-python