iamwilhelm / charyb Goto Github PK

h2. What is Charyb?

Charyb is a tool to help suck down data and shove them into a simple datawarehouse.

There are two parts, the web interface and the crawler.  

The web interface makes it easier to suck in data from different mime types.  
Because data scraping isn't yet fully automated, there needs to be a little bit 
of human intervention. 

The crawler makes use of the data humans entered to figure out how to scrape the 
data.  It will keep checking those data sources as updates.

h2. Installation

if you haven't added the github source, do it by doing:

  sudo gem source -a http://gems.github.com
  sudo gem source -a http://gems.rubyforge.org

The install the following packages:

  libopenssl-ruby

Then install the following gems:

  # for web application
  sudo gem install -d sinatra -v 0.9.4
  sudo gem install -d hpricot -v 0.8.1
  # sudo gem install -d couchrest -v 0.33
  sudo gem install -d sqlite3-ruby -v 1.2.5  # only for sqlite3

  # for testing
  sudo gem install -d rack-test -v 0.4.2
  sudo gem install -d webrat -v 0.5.3
  sudo gem install -d thoughtbot-shoulda -v 2.10.2

  # for mocks and stubs
  sudo gem install -d mocha -v 0.9.8

  # for fixtures
  sudo gem install -d notahat-machinist -v 1.0.3
  sudo gem install -d faker -v 0.3.1

h2. Setup

You need to pull in the git submodules first

  git submodule init
  git submodule update

If it tells you that you can access it, ping the owner of the repo to add you as 
a collaborator

Then you must set up the database.

  rake db:schema:load

h2. To run it

To run the web interface:

  rake web:run

And then go to http://localhost:4567

To run the crawler:

  rake crawler:run


h2. Architecture

There are two major components:  The web interface and the crawler.  The web 
interface is used to for the human in the loop to more easily say how to suck 
in data for a datasource.  Right now, it only does html tables.  Eventually, 
it'll also do CSV files as well. It stores this information in the 
SourceTracker, which is a wrapper around an SQLite database.

The crawler is run separately and uses the SourceTracker's information about 
where a datasource can be found and how to suck it in.

When the crawler sucks in the data and cleaned it properly, it then
pushes it to the redis datawarehouse.  Then teabag queries the
datawarehouse when it gets web requests from the outside world.

The web part starts with src/application.rb  it requires 'config/initialize' 
at the top, which sets all the global constants and paths.  you can find it 
in config/initialize.

Then the rest of that code is in sinatra's DSL.  I suggest reading 
http://www.sinatrarb.com/intro.html first, and then using 
http://www.sinatrarb.com/book.html as a reference.

The crawler starts with crawler.rb, but I haven't finished it yet.  Right 
now, the source tracker isn't finalized yet either, so the web interface 
directly uses the activerecord objects.  activerecord is an ORM object.  
http://api.rubyonrails.org/classes/ActiveRecord/Base.html

If you have specific questions, don't hesitate to ask either email or by 
phone.

h2. Import Table Use Case

1. user starts the web interface (rake web:run) from a console.
2. webserver starts, messages scroll by in the console.
3. user opens a browser to http://localhost:4567
4. browser opens showing the charyb front page.
5. user enters the url of a page containing an html table into the
   textfield and clicks "submit."
6. browser shows requested page in left div and list of tables
   imported from that page (none) and a button to add a new table in
   the right div.
7. user clicks "Add a table."
8. a form is displayed in the right div.
9. user fills out the fields from "Table heading" to "Converter."
   Some fields have special formatting rules:
   - "col heading" or "row heading" can be "Category" if no word
     describes the collection of headings.  Most of the time "row
     heading" will be "Year" or "State."
   - "other dims" should be a csv list of "dimension name, dimension
     value" pairs.  For example, "Year, 2006"
   - "default" should match the value of the "row header" "col header"
     or one of the "other dims."
   - "published at" should be a date, eg. "01/01/2009."
   - "units" can be a single value or can be a csv list for each column.
   - "converter" can be blank, or a single value, or a csv list for each
     column.  values must match a converter name. (not implemented yet)
10. user clicks a cell at one corner of the column headers in the table
    on the left div.
11. the cell is colored red.
12. user shift-clicks a cell at the opposite corner of the column headers.
13. the column headers are all colored red.
14. the user clicks the "set" button next to "Col labels."
15. the column headers are all colored green, their values are listed
    in the "Col labels" textarea, one per line.
16. the user does the same for row headers and data.
17. row headers are colored brown; data cells are colored purple.  if
    there are n row headers and m col headers there must be nxm data
    values.
18. user clicks the "Create" button.
19. right div removes the form and lists the table as a link.


h2. Edit Table Use Case
1. user starts the web interface (rake web:run) from a console.
2. webserver starts, messages scroll by in the console.
3. user opens a browser to http://localhost:4567
4. browser opens showing the charyb front page.
5. user clicks on a datasource title link.
6. browser shows requested page in left div and list of tables
   imported from that page with edit buttons, and a button to add a
   new table in the right div.
7. user clicks edit.
8. browser shows partially completed form used to import the table.
9. user clicks the "Finish Loading" button.
10. browser selects cells in the table in the right div, coloring the
   table as it was when it was imported, and fills in the textareas
   with selected values in the form.
11. user makes changes and clicks the "Update" button.
12. right div removes the form and lists the table as a link.


h2. cli commands to look into redis
In the following commands, all spaces are replaced by underscores.
This is only for input.  The result can contain spaces.

> redis-cli smembers datasets
lists datasets by name

> redis-cli smembers "[dsname]||dimensions"
lists the dimensions of dataset dsname.

> redis-cli smembers "[dsname]||[dimname]"
lists the values of the dimension dimname.

> redis-cli smembers "[dsname]||meta"
lists metadata for a dataset.

> redis-cli get "[dsname]||[dimvalue]||[nextdimvalue]"
returns a value.