Break it!

Would be great to reduce the links for the open data portal to something not so ugly.

We want to know if the public are using/visiting the site. inclucivics.com

We have two options:

• Google Analytics
• Piwik

This issue is in regards to the following:

• Hook up this repo to Sophicware hosted jenkins instance
• [ ] Create jenkins job for building AMIs within C4N AWS account
• Setup continuous deployment

Not sure what the deal is with this, but a simple sorting of values either alphabetically or by numeric value would be helpful.

@colbydehart You think you could handle this one?

This is currently loaded from a static json file. I'll need to preseed the database first and have the parser create an object from all available timestamps.

@jchapin @JamesNix

Based on the discussions we had it seemed like the basic approach was to use mongo for the db and host the app using Heroku. I don't think MQL will be too difficult to pick up.

I started playing with Heroku, but am not particularly familar with it. Does either one of you want to try and setup a project that I can tag along into at the next hack night? I need a few python libraries, but from what I read Heroku should support it.

Also, if you want to begin working on an interface and testing rendering images using vega. Checkout vega. The vega editor has examples of JSON specifications for graphs that you could use for testing purposes.

Please add this to our website's projects page.

The major problem with switching out the backend here is that we've got to maintain the data we had when we were importing from text files, and put it in a format identical that which is expected by socrata. I'm thinking here that I'm just going to manually create a database that has that data, complete with timestamps, and will provide a rethinkdb backup to be imported if the new database does not exit. This way we should be able to continue on as if we had been importing data all along.

I could pretty easily use celery to handle the periodic update request... seems like a bit of overkill for this, but ... I can set it up pretty fast. The real issue will be deployment.

This makes calculating observed/expected values considerably easier.

@mshenfield

Potential metrics...:

Chi Square. Compare the Observed/Expected ratios for Metro as a whole and plot the resulting chisquare value overtime.

Advantages: Statistical backing, appropriate for this type of data, creates a single metric that can be plotted

Disadvantages: May be to sophisticated for average user, how do we handle multiple income levels?

Some metric that relates the number of departments at or near the values predicted by census ?

Currently does machine sorting 1, 10, 100, 2, 20, etc.

@bdfinlayson

We need your help here. The mobile version of this project looks absolutely atrocious. We can't have public attention on this platform when it looks like this. It will reflect poorly on Code for Nashville and what we are able to do as an organization.

If you can rally some people to fix this, or handle it yourself, it would be deeply appreciated.

look at www.inclucivics.com to get an idea

the python face of this is pretty rusty at this point. Going to go through and do some refactoring of that whole piece.

@jcockhren

http://52.1.163.16/

Content to Use

In January 2015, the Metro Human Relations Commission (MHRC) released the IncluCivics Report, analyzing the demographic makeup of 50 Metro Nashville departments. The data in the original report was provided by Metro Human Resources (Metro HR) in August 2014. Since then, Metro HR has provided more recent data (captured April 1, 2015) and has announced that updated data will be released quarterly. The original IncluCivics Report, and a recent and more robust Data Update are at https://www.nashville.gov/Human-Relations-Commission/IncluCivics.aspx.

This platform, graciously created and maintained free of charge by Code for Nashville, exists for two reasons. First, it is imperative to establish a baseline from which to assess our collective efforts at attaining a more diverse workforce in the future. Second, to further encourage transparency and public education, this platform will capture the demographic data provided quarterly by Metro HR, render it in user-friendly charts and graphs, and will track changes in the data over time.

The raw data used on this platform is available to the public and can be found at https://data.nashville.gov/Metro-Government/General-Government-Employees-Demographics/4ibi-mxs4

I think this is an excellent use case to begin playing with our own open data portal.

I downloaded the CSV from data.nashville.gov only to find that the keynames had changed (surprise!) between the first dataset I got and the next. I resolved this temporarily by just editing the header myself, but obviously this is not a process

We need to handle basic ETL issues (type serialization, missing values, key_names). I'd like to be pulling this data from a staging area rather than from CSVs.

Makes more sense to just have batch jobs output static json files.

@mshenfield and I discussed this out, it would be best to put this chart on the initial load underneath the text snippet describing the intention of the site.

Should use highcharts and be a line graph of some sort. Even a preliminary with fake data would be progress at this point.

This is probably just going to be a simple extension of the simple import job we're using to bring in data right now.

Eventually we'll want to figure out how to integrate with Metro's Open Data Portal and perhaps utilize our own Socrata portal as a staging ground for cleaned data. If we can do a JSON request from our own portal we may be able to eliminate rethinkdb as a dependency all together (which is to our advantage as I believe I am the only person who uses it. Shame though, great NoSQL db)

For some reason, calling an import statement like from ntp.{module or modules} import {something} fails on the deployment server. However, these same statements run wtihout error on the VM from our Vagrantfile.

Removing all such statements in commit b755eee allowed the build to be deployed on the server.

The server and local versions are both running Python 2.7.6, and a fresh install is done each deployment.

Here are some example stack traces from the server deployment for commit 0d23ae7 where the ntp.* imports have not been removed. Haven't been able to recreate on the local VM running the sudo python {relative path e.g. /vagrant/ntp/}run_server.py from different directories.

Traceback (most recent call last):
File "/opt/inclucivics/ntp/run_server.py", line 8, in
from app import app
File "/opt/inclucivics/ntp/app/init.py", line 7, in
from api import departments, data
File "/opt/inclucivics/ntp/app/api.py", line 3, in
from include.functions import rdb_get_data_by_department, rdb_get_department_names#, rdb_get_temporal_values
File "/opt/inclucivics/ntp/app/include/functions.py", line 1, in
from ntp.data.include.rethinkdb.tables import RdbMostRecent, RdbChiMerged
ImportError: cannot import name RdbMostRecent

ImportError: cannot import name filter_str
Traceback (most recent call last):
File "/opt/inclucivics/ntp/run_server.py", line 2, in
from data.load import table_check
File "/opt/inclucivics/ntp/data/load.py", line 2, in
import sanitize as clean
File "/opt/inclucivics/ntp/data/sanitize.py", line 6, in

Traceback (most recent call last):
File "/opt/inclucivics/ntp/run_server.py", line 2, in
from data.load import table_check
File "/opt/inclucivics/ntp/data/load.py", line 1, in
import aggregate as agg
File "/opt/inclucivics/ntp/data/aggregate.py", line 1, in
from ntp.project.common.helpers import merge_json_like, sortDict
ImportError: cannot import name merge_json_like

Need to figure out an easy dependency for this or roll my own

Hey @beck410

I've added the graphs in, but I think you've modded the primary div container so that it only has a single column, but there are two divs of graphs that need to be compared side by side. Could you check the add_graphs branch and change it so that the graphs are aligned left and right?

For some reason the result graphs are not displaying any resuts! It is getting 0 back for all the expected values. See http://52.1.163.16/

Handle data upload using python

Update README to say:

'launch the python webserver available on localhost:8082'

See PR #75

Apologies for pushing to my master branch...

I don't care which one. Either will work fine, simple automated task doesn't necessarily require all of what rdis can offer.

The pie chart looks quite bad when your have big ranges of data. Bar charts are more appropriate for this data anyway. Rather than adjust the existing endpoints I just want to add multiple visualization options.

This project is now part of "The Nashville Transparency Project" and needs an index to introduce the ntp and provide a link to the hr reporting app

Add data for what census based demographics would look like probably best to calculate that on the back-end and expose through api.

The table reference RdbMostRecent, used to pull department data, is defined as None, causing an internal server error when /api/departments is called.

The table reference RdbMostRecent is defined in data.include.rethinkdb.tables.py. This module is imported within the initial run_server.py->load.py->aggregate.py, causing the reference to be defined before relevant tables have been created.

From Mark:

Need to combine or delete the departments that Jon and I previously discussed.

Dates on the temporal graphs should be: 2014-August, 2015-January, and 2015-April

Known problems are mostly associated with type serialization.

Not uncommon to find the salary field with mixed string and floating types.

I.e.

"$37,000.00"
and "37000.00"

Simple answer is to split on "." filter non numeric values, and convert index 0 to int

I'm going to wait until Beck adds in the logo before making this change.

Need a simple static page for NTP that looks good and can be deployed to link to IncluCivics itself.

• Change pie charts headings from "Actual Demographics" to "Metro Demographics."
• Remove the income brackets from the "Census Predicted" pie charts
• Correct the dates on the temporal graphs: Dates on the temporal graphs should be: 2014-August, 2015-January, and 2015-April