opencleveland / drocer Goto Github PK

Scraper and parser of Cleveland City Council's records and the produced text.

Python 2.34% HTML 95.95% Java 1.71%

drocer's Introduction

drocer

Cleveland City Council's legislative records are available as PDFs from 1996-present.

As PDFs, there's many things that you cannot to do with them: search in multiple files at one time; easily extract data from them for further analysis; browse legislation by a particular topic; and more.

In short, PDFs make it very difficult for programmers, journalists, researchers, and residents to use the data inside them.

Our goal is to remove those barriers.

What's here:

The year folders (e.g. 2017) consist of plain text files extracted from PDFs of The City Record, Cleveland City Council's legislative notes.

The files are not recommended for reading. Small errors (misspellings and improper spacing, especially for lists) were created during the text extraction. Tables and graphics in the City Record are also missing from these plain text files.

Check the original PDFs or download them by year in bulk to verify if any information is missing.

Learn more about our goals, project roadmap, and how you can help at:

http://www.opencleveland.org/projects/drocer/

https://github.com/opencleveland/drocer/wiki

Additional documentation and plans is scattered at: city-record-contents-formatting.md - boilerplate text that is used in the city record and what it means.

drocer's People

Contributors

Stargazers

Watchers

Forkers

mbramson mef79 calarrick randolphlacey ysakthi danielhelfand

drocer's Issues

update PDF archive with ISO601 file names

Thanks to Meaghan, the PDFs are now named in a standardized format (ISO 8601) (i.e. 2015-08-03).

Need to update documentation and our google drive with these files.

graph/visualization of how something becomes a law

nice to have.

A brief guide to city record and city council.

this may be a separate project from the website, but:

a basic guide for people for people to ask:

A one-two pager for average residents with a 9th grade reading level to understand
what type of information is in city council, here, what do terms like resolution and ordinance mean; explaining the concept of 'emergency ordinance'

http://www.clevelandcitycouncil.org/about-city-council/legislative-process
(this page above is new, and a welcome addition to the city council website).

However, if you search "854-17" (without parenthesis) in your browser's search function; (I copy and pasted "854-17"and pasted into my browser - FF57 - search).
on
https://github.com/opencleveland/drocer/blob/master/2017/2017-07-19.txt ; several results will be returned in

Finish parsing departments

And clean up department-related code

Dump all the PDFS to text

convert the pdfs to plain text.

After this is done, we can begin finding patterns for particular data that we're looking for.

For some inspiration or already existing projects, see:

https://github.com/18F/doc_processing_toolkit
http://textract.readthedocs.org/en/latest/
http://www.unixuser.org/~euske/python/pdfminer

Improve documentation

our documentation is a hot mess.

Goals:

[ provide brief overview of what project does;
how someone can get involved; (audience: people who never heard of project before] -
done at:
[more thorough] https://github.com/opencleveland/drocer/wiki