codeisland / ri-lobbying Goto Github PK

It looks like this might be our first major technical task!

Here's the problem:

• In 2017, the Secretary of State's office started using a brand-new MySQL database for gathering lobbyist data, with a new schema
• There is an entirely separate MS SQL database that contains all the lobbyist data from the mid-2000s through 2016
• It would be really great if we could migrate the data from the old MS SQL database and add it to the new MySQL database

Here are some details:

• To the best of our knowledge, the new database schema is backwards compatible with the old schema (e.g. they added new fields in 2017, but they didn't change any existing fields, we think)
• The old DB is likely to have malformed data. Almost all fields are unvalidated strings. Any import process will need to look out for bad data and have a process to correct it.
• I haven't been able to get access to actual data yet, but the old DB is roughly 3-4 GB in size

Any initial thoughts? It's worth noting that this is essentially a one-time task – once we successfully convert the pre-2017 data into the 2017 format, we'll never need to work with the old data again.

I'll let y'all know as soon as we get access to actual backups or schema information.

Code Island appears to have an existing email address for reporting harassment: [email protected]. Is that email address still actively monitored?

If so, we should update the email address in our CoC to point to that address. (For now our contact address is "[email protected]", which forwards directly to me, but ideally I think that CoC-related emails should go to more than just one person.)

At our Nov. 30 meeting, someone mentioned that Code Island has an existing AWS account, possibly with free resources. @goldenmeanie How would we get access to that account?

It looks like past Code Island projects have used the MIT license, which seems like a reasonable choice.

In order to do anything with RI Open Government data, first we need access to the data! I've just contacted Terrance from the Secretary of State's office to ask for access to the state data, and I'll update this issue as I learn more.

In the meantime, if we're curious about what kind of data the DB includes, there's a web interface to the data on this page.

What languages/technologies should we use? Here's one possible proposal, but please feel free to propose alternatives, too.

For a project like this, I'd love to keep things small at the start. I think we can do a great job with just a few core components:

• A database. We have data, so we have to store it somewhere. I've had good success with Amazon RDS + MySQL, but other suggestions are welcome!
• A server-side web application framework. I'm most familiar with Ruby on Rails, so I'd vote for that, but our choice should probably depend on the existing technical expertise of the group.
• A cloud hosting platform. I'd vote for Heroku for ease-of-use, as their free tier is more than enough for most low-traffic projects.

Notably, here are a few things that I think we should leave out in v1.0:

• A front-end JavaScript framework. My gut says this is overkill to start. You can accomplish a lot with static web pages, or with just a sprinkling of jQuery! It may make sense in the long-term if this site grows to be more powerful and complex.
• An API. Again, I believe this is overkill to start, especially if we're not using any front-end framework. It could make sense as a later update, though!

Thoughts? I've used this approach to good success in the past – ripta.info and RIBIT both use this stack, and it's made them easy to maintain and update.

We're not entirely sure what we're building yet, other than a basic app to view lobbying data. We need other people to help shape our intuitions – people who have more experience working with lobbying data than we do.

We should talk to people who already look at lobbying data, and ask them:

• What do you currently do with lobbying data?
• What do you wish you could do with lobbying data?
• What obstacles cost you the most time/effort?

Types of people to talk to include:

• Journalists
• Activists
• Lobbyists themselves
• Others?

Anyone have any leads here? I know some people at Common Cause who might have good thoughts about this.

Hi there! If you'd like to get involved with this project, please comment below and tell us a bit about yourself. As we get started, hopefully this will help us to self-organize, and help us better understand what kind of team we're working with 🙂

Who are you?
I'm Ryan/@rofreg! I helped create Splitwise, and I build a lot of web-based side projects in my spare time. I'm a full-stack developer and designer, and most of my projects are built with Ruby on Rails + MySQL + Heroku.

How can you help?
I'm our team leader! At the start, I think I can be helpful for pretty much anything (user research, UI/UX design, front-or-back-end development, project management, etc), so I'd love to help wherever is most needed. As time goes on, I hope to see other people step up and claim responsibility for some of these areas, too.