timothyrenner / profanitypowerindex Goto Github PK

The readme needs local mode instructions.

Spark streaming is packaged with an older version of jets3t to write to S3. I think the bundled Hadoop distro might have or require a newer version, which is causing a runtime error whenever the program tries to write to S3.

The file system write was nice in a pinch, but Cassandra is a better way to store the data.

Right now I'm just linking the images, but they keep getting deleted and I have to go find new ones.
There's probably a more maintainable way of doing this (yanking them into a hosting service or something).

Currently SQL post processing for aggregating the collected events and merging with the jump started data is ad-hoc. This could easily be done with a shell script and a couple of command line arguments.

Mesosphere requires the fat jar to be publicly accessible. Credentials shouldn't be in it, so twitter4j.properties has to go.

Hillary Clinton, Lincoln Chafee, Bernie Sanders, Martin O'Malley, Rand Paul, Ben Carson need updated images. Wait for Rand Paul and Ben Carson to switch to sane profile pics, if possible.

Add an accumulator to Spark and local modes to count the total number of tweets processed.

Why didn't I do this?

Right now it defaults to my local time, Central. It would be nice to display this in the time local to the debate.

Briefly discuss running in Spark mode in the project's README. Include configuration options not present in local mode.

The json file used to generate the graphic template contains almost all of the information needed for the "jump starter" Python script.
Add the start/end times for the event to the json config file, then have the Python script read from that.

Bar plots don't line up with pictures too well.

Seriously that's not a good name. Change to ProfanityPowerIndex.js

I'm pretty sure the local mode accumulator is worthless unless there's a time limit. Checkpoint the value to STDERR every 100 tweets or so.

Looks like Ben Carson's picture got deleted from Twitter. Swap it out with the new one in the site's JSON config file.