This repository is the database and API underlying the EDGI Web Monitoring Project.
It’s a Rails app that:
- Acts as a database of monitored pages and revisions that have been made to them
- Allows other services to add new tracked pages/versions (we are currently focused on Versionista, but this database will soon host data from other sources, such as the Internet Archive)
- Provides an API to get that version data and allow analysts or other automated tools to annotate those versions with metadata
-
Ensure you have Ruby 2.4.1+
You can use rbenv to manage multiple Ruby versions
-
Ensure you have PostgreSQL 9.5+
-
Ensure you have Redis
On OSX:
$ brew install redis
On Debian Linux:
$ apt-get install redis
-
Clone this repo
-
If you don’t have the
bundler
Ruby gem, install it:$ gem install bundler
-
Wherever you cloned the repo, go to that directory and install dependencies:
$ bundle install --without production
-
Set up your database. The simple way to do this is:
$ bundle exec rake db:setup
That will create a database, set up all the tables, create an admin user, and add some sample data. Make note of the admin user e-mail and password that are shown; you’ll need them to log in and create more users, import more data, or make annotations.
If you’d like to do the setup manually see manual postgres setup below.
If you're getting error such as
FATAL: role "user" doesn't exist. Couldn't create database.
check troubleshooting below. -
Start the server!
$ bundle exec rails server
You should now have a server running and can visit it at http://localhost:3000/. Open that up in a browser and go to town!
-
Bulk importing (and, in the future, potentially other features) make use of a Redis queue. If you plan to use any of these features, you must also start a Redis server and worker.
Start redis:
$ redis-server
Start a worker:
$ QUEUE=* VERBOSE=1 bundle exec rake environment resque:work
If you don’t want to populate your DB with seed data, want to manage creation of the database yourself, or otherwise manually do database setup, run any of the following commands as desired instead of rake db:setup
:
$ bundle exec rake db:create # Connects to Postgres and creates a new database
$ bundle exec rake db:schema:load # Populates the database with the current schema
$ bundle exec rake db:seed # Adds an admin user and sample data
If you skip rake db:seed
, you’ll still need to create an Admin user. You should not do this through the database since the password will need to be properly encrypted. Instead, open the rails console with rails console
and run the following:
User.create(
email: '[your email address]',
password: '[the password you want]',
admin: true,
confirmed_at: Time.now
)
If you are getting errors such as FATAL: role "user" doesn't exist. Couldn't create database.
while running rake db:setup
or rake db:create
then it may mean that your database is password protected. There are two ways to setup required databases:
-
(Recommended) Create users and databases manually
sudo -u postgres psql -c "CREATE USER \"web-monitoring-db_development\" WITH PASSWORD 'wmdb';" sudo -u postgres createdb -O web-monitoring-db_development web-monitoring-db_development -E utf-8 sudo -u postgres psql -c 'CREATE EXTENSION IF NOT EXISTS "uuid-ossp";' web-monitoring-db_development sudo -u postgres psql -c 'CREATE EXTENSION IF NOT EXISTS "plpgsql";' web-monitoring-db_development
and then set
DATABASE_URL
environment variable to point to the development database:export DATABASE_URL=postgres://web-monitoring-db_development:wmdb@localhost/web-monitoring-db_development
You can put this line in your
~/.bashrc
or~/.profile
file not to type it each time you open terminal.Required databases exist, now continue with loading schema.
-
Loosen local Postgres database security to allow local users without password
You have to edit pg_hba.conf config file (
/etc/postgresql/9.6/main/pg_hba.conf
on Unix) and add or update authorization line for local logins frommd5
totrust
:# "local" is for Unix domain socket connections only local all all trust # IPv4 local connections: host all all 127.0.0.1/32 trust
Create Postgres superuser that will link to your account:
sudo -u postgres createuser `whoami` -ds
Now
bundle exec rake db:setup
command should work.
This project wouldn’t exist without a lot of amazing people’s help. Thanks to the following for all their contributions!
Contributions | Name |
---|---|
💻 🚇 📖 💬 👀 | Rob Brackett |
📖 👀 | Dan Allan |
📖 📋 | Dawn Walker |
📖 📋 📢 | Matt Price |
📖 | Patrick Connolly |
📋 🔍 | Toly Rinberg |
📋 🔍 | Andrew Bergman |
💻 | Robert Dalin |
💻 📖 | Krzysztof Madejski |
(For a key to the contribution emoji or more info on this format, check out “All Contributors.”)
Copyright (C) 2017 Environmental Data and Governance Initiative (EDGI)
This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3.0.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
See the LICENSE
file for details.