Giter Club home page Giter Club logo

muckrock's Introduction

MuckRock

MuckRock · Squarelet · DocumentCloud · DocumentCloud-Frontend

Codeship Status for MuckRock/muckrock codecov.io

MuckRock is a non-profit collaborative news site that gives you the tools to keep our government transparent and accountable.

Prerequisites

MuckRock depends on Squarelet for user authentication. As the services need to communivate directly, the development environment for MuckRock depends on the development environment for Squarelet - the MuckRock docker containers will join Squarelet's docker network. Please install Squarelet and set up its development environment first.

Install

Software required

  1. docker
  2. python
  3. invoke
  4. git

Installation Steps

  1. Check out the git repository - git clone --recurse-submodules [email protected]:MuckRock/muckrock.git
  2. Enter the directory - cd muckrock
  3. Run the dotenv initialization script - python initialize_dotenvs.py This will create files with the environment variables needed to run the development environment.
  4. Set an environment variable that directs docker-compose to use the local.yml file - export COMPOSE_FILE=local.yml
  5. Set up the javascript run inv npm "install" and inv npm "run build"
  6. Start the docker images - inv up This will build and start all of the docker images using docker-compose.
  7. Set dev.muckrock.com to point to localhost - echo "127.0.0.1 dev.muckrock.com" | sudo tee -a /etc/hosts
  8. Enter dev.muckrock.com into your browser - you should see the MuckRock home page.
  9. In .envs/.local/.django set the following environment variables:
  • SQUARELET_KEY to the value of Client ID from the Squarelet Client
  • SQUARELET_SECRET to the value of Client SECRET from the Squarelet Client
  1. You must restart the Docker Compose session (via the command docker-compose down followed by docker-compose up) each time you change a .django file for it to take effect.

You should now be able to log in to MuckRock using your Squarelet account.

Docker info

The development environment is managed via docker and [docker compose][docker compose]. Please read up on them if you are unfmiliar with them. The docker compose file is local.yml. If you would like to run docker compose commands directly, please run export COMPOSE_FILE=local.yml so you don't need to specify it in every command.

The containers which are run include the following:

  • Django This is the Django application

  • PostgreSQL PostgreSQL is the relational database used to store the data for the Django application

  • Redis Redis is an in-memory datastore, used as a message broker for Celery as well as a cache backend for Django.

  • Celery Worker Celery is a distrubuted task queue for Python, used to run background tasks from Django. The worker is responsible for running the tasks.

  • Celery Beat The celery beat image is responsible for queueing up periodic celery tasks.

All systems can be brought up using inv up. You can rebuild all images using inv build. There are various other invoke commands for common tasks interacting with docker, which you can view in the tasks.py file.

Networking Setup

The MuckRock development environment will join Squarelet's environments docker network, so that the services can coexist. Please see the README file from Squarelet for more information.

Environment Variables

The application is configured with environment variables in order to make it easy to customize behavior in different environments (dev, testing, staging, production, etc). Some of this environment variables may be sensitive information, such as passwords or API tokens to various services. For this reason, they are not to be checked in to version control. In order to assist with the setup of a new development environment, a script called initialize_dotenvs.py is provided which will create the files in the expected places, with the variables included. Those which require external accounts will generally be left blank, and you may sign up for an account to use for development and add your own credentials in. You may also add extra configuration here as necessary for your setup.

Invoke info

Invoke is a task execution library. It is used to allow easy access to common commands used during development. You may look through the file to see the commands being run. I will go through some of the more important ones here.

Release

inv prod will merge your dev branch into master, and push to GitHub, which will trigger CodeShip to release it to Heroku, as long as all code checks pass. The production site is currently hosted at https://www.muckrock.com/. inv staging will push the staging branch to GitHub, which will trigger CodeShip to release it to Heroku, as long as all code checks pass. The staging site is currently hosted at https://muckrock-staging.herokuapp.com/.

Test

inv test will run the test suite. To reuse the database, pass it the -r=1 option. inv coverage will run the test suite and generate a coverage report at htmlcov/index.html.

The test suite will be run on CodeShip prior to releasing new code. Please ensure your code passes all tests before trying to release it. Also please add new tests if you develop new code.

Code Quality

inv pylint will run pylint. It is possible to silence checks, but should only be done in instances where pylint is misinterpreting the code. inv format will format the code using the yapf code formatter.

Both linting and formatting are checked on CodeShip. Please ensure your code is linted and formatted correctly before attempting to release changes.

Run

inv up will start all containers in the background. inv runserver will run the Django server in the foreground. Be careful to not have multiple Django servers running at once. Running the server in the foreground is mainly useful for situations where you would like to use an interactive debugger within your application code. inv shell will run an interactive python shell within the Django environment. inv sh will run a bash shell within the Django docker comtainer. inv dbshell will run a postgresql shell. inv manage will allow you to easily run Django manage.py commands.

  • inv manage migrate to migrate database inv npm will allow you to run NPM commands. inv npm "run build" should be run to rebuild assets if any javascript or CSS is changed. If you will be editing a lot of javascript or CSS, you can run inv npm "run watch". inv heroku will open a python shell on Heroku.

Pip Tools

Python dependencies are managed via pip-tools. This allows us to keep all of the python dependencies (including underling dependencies) pinned, to allow for consistent execution across development and production environments.

The corresponding files are kept in the pip folder. There are requirements and dev-requirements files. requirements will be installed in all environments, while dev-requirements will only be installed for local development environments. It can be used for code only needed during develpoment, such as testing. For each environment there is an .in file and a .txt file. The .in file is the input file - you list your direct dependencies here. You may specify version constraints here, but do not have to.

Running inv pip-compile will compile the .in files to the corresponding .txt files. This will pin all of the dependencies, and their dependencies, to the latest versions that meet any constraints that have been put on them. You should run this command if you need to add any new dependencies to an .in files. Please keep the .in files sorted. After running inv pip-compile, you will need to run inv build to rebuild the docker images with the new dependencies included.

FOIAMachine

FOIAMachine is our free FOIA filing tool, that allows you to track your requests while requiring you to manually handle all of the message sending and receiving. It is run off of the same code base as MuckRock. To access it, set dev.foiamachine.org to point to localhost - sudo echo "127.0.0.1 dev.foiamachine.org" >> /etc/hosts. Then pointing your browser to dev.foiamachine.org will take you to FOIAMachine - the correst page is shown depending on the domain host.

Update search index

MuckRock uses watson for search. The index should stay updated. If a new model is registered with watson, then build the index (fab manage:buildwatson). This command should be run on any staging or production servers when pushing code that updates the registration.

muckrock's People

Contributors

adityajain15 avatar advaydev1 avatar aleenaloves avatar allanlasser avatar anthonyjpesce avatar benjamindoran avatar benlk avatar callajune avatar duckduckgrayduck avatar emilyliu7321 avatar ericliuche avatar erikreyna avatar jugglinmike avatar mitchelljkotler avatar morisy avatar pjsier avatar simongle avatar titanous avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

muckrock's Issues

Remove Epic Editor

We should still use MarkDown processing, but remove EpicEditor from the article page.

Draft page too confirming

screen shot 2014-10-16 at 10 16 38 pm

The draft page is a little too confirming.

  • "Your request has been created." makes it sound like you're all done.
  • The green and check box reinforces that.
  • The message should be more "almost there" and then make it easy to submit.
  • I feel that having the button on the top right kind of makes it invisible if you're kind of scanning. If you're reviewing, you work your way down the screen and there's nothing to click. If you just want it filed, you want to click something in that "Almost there!" message up top. The current location feels very buried.

Task Tracking

We should have a new task management system which is, to start, pretty much a simple model and a way for items to get into the model via Mailgun.

The model should include (all fields optional):

Title
Description
Date Created
Assigned to (Allow multiple users)
Priority (just a float number)
Project (Just a string, ideally allow multiple such as "Spylighter" and "Drone Census")
Completed (Boolean)
Due Date

This system will be, at first, purely in the backend. Admins should be able to get (and regenerated) keys (possibly same as API key?) to email into this system in the following format:

userstring+Spylighter+Drone_Census+due2d+priority30@tasks.muckrock.com

The subject line becomes the title, the body becomes the description. This would be assigned to Spylighter and marked due in two days. All fields are optional.

priority just checks for the word "Priority" and then assigns the value after that. due looks for m (month), d (day), w (week) , or y (year) and marks it due that many d/m/w/y after the task is received.

Everything should be accessible via API.

Instead of completed being a Boolean should be a should be status with options of:

  • Open (default)
  • Completed
  • Declined

Ability to have non-FOIA-able agencies

Some agencies (such as white house) should be in the database, but should not be FOIA-able. If a user tries to file with them, there should be a notification as to why, and brief tips on following up.

Adding ToMails to part of tasks system

To be done as part of overall task system:

I've been thinking a lot about the to mail issues the last few days, in
terms of the delays, tracking, one-click setting, etc.

I'd like to try breaking it out into a dedicated page (probably
/tasks/mail, and move orphans to /tasks/orphans) and have it skip
[email protected] entirely.

On that page it would just include every thing that currently needs to
be mailed, with links associated that go to:

— The request page.
— One click to send it out as mailed today.

  • One click to send it out as mailed tomorrow (for things sent late in
    the day and postmarked the next day).
  • Same as above, but to mark everything currently on the page as mailed
    out today/tomorrow.
  • Notes if a payment needs to be included with a given mailer, including
    how much.
  • A link to an HTML page that is exactly what needs to be printed and
    mailed for each communication, ready to just hit CTRL-P, so the
    processor can just open them all in tabs, print them all out, and mail
    them.
    — POSSIBLY a link to a one page-collation of the above that includes
    page breaks so you just have to hint print once, and can then use the
    one-click. I'm a bit wary of this, particularly at start, but thinking
    about it if it would work.

Eventually, we want to move communication handling out of the requests
inbox and into the site, to make it easier to crowdsource, to undo, and
to make sure stuff doesn't get lost, and I think this is a good first
step.

We can run this in parallel with the emails for a while as a
safety, but I'm concerned at how the emails for tomail stuff tend to get
pushed to late in the day or the next day, and I think this is a chance
to make the process as simple and as fool proof as possible, as well as
a good way of batching like tasks so that they're done more efficiently
and with fewer errors.

Whitelisted emails to orphan page

Emails that are on the various MuckRock agency whitelists (.gov, an email that matches our database of agency emails, etc.) should be able to send to *@requests.muckrock.com and have the message go to orphans if it doesn't match an existing request email, even if it doesn't fit the pattern of ####-#######@requests.muckrock.com

Mail reputation monitoring

Have some kind of regular, independent testing of our mailing domain to ensure we're not being spamblocked.

This is a longer term thing, just noted here to get it on the road map.

Buffer faxes

From Faxaway:

Hello;

Okay, the faxes failed, because there were so many messages addressed to the same fax number, at the same time, that all three fax attempts had a busy signal, because other faxes were being transmitted at the same time.

Please separate the emails (if they are to the same fax number) by 5 or so minutes, so that all the emails for the same fax number, are not received (or attempted to be processed) at the same time.

Thank you,

Kory Miller
[email protected]

Changes to crowdfunding payments

Regarding this page:

https://www.dropbox.com/s/gv6ghwrcjlr8wrs/Screenshot%202014-05-07%2023.30.54.png

After amount, add new checkboxes:

Follow this request for updates on its status:
List my name as a contributor:
Both should be checked by default.

And then after the crowdfund summary it should list donors that submitted with the check mark checked like we do on this request:

https://www.muckrock.com/foi/united-states-of-america-10/fbi-drone-documents-1542/

Contributors

Scott Ainslie
Mark Erickson
Conor Skelding
... plus 1 anonymous supporter

More about this crowdfunding campaign. <- And then at a bottom, a link to the child news article, if applicable.

Scheduled/Recurring Requests

Users should be able to schedule one-time (on a certain date) and recurring requests (the request will be refiled weekly/monthly/yearly). It might be nice to include the ability to do some relative values ("I request any emails sent between [today's date - 1 week] to [today's date]") but for simplicity, just having recurring and scheduled requests would be very useful.

Design process to prevent users from accidentally including their email address

New Users in particular like to include their email address in requests, which is confusing to agencies and ends up breaking shit.

Once the new redesign is launched, we should include some kind of check to make sure users understand and don't put in their email address unless they really want to.

Ideally, it'll also check for addresses but that seems tricky.

Registration flow on bottom of file screen

screen shot 2014-10-16 at 9 59 22 pm

I think this can be better. Instead of asking them to sign up, can we ask for:

  • First Name
  • Last Name
  • Email

and then send them to the next page, with a small text link to let them log in if they have an account?

RSS Feeds for Users

3 RSS feeds on user pages that include:

  • All a users new requests
  • All updates to a user's requests
  • All a user's completed requests.

Allow searching through DocumentCloud documents

We should offer a way for users to search through our documents on DocumentCloud. They offer a really nice API of their own, but I think it'd probably be better for us to pull the raw text data into our backend and allow searching through it on our main search interface.

Crowdfund Display/Article Tweaks

A few tweaks with crowdfunding:

On the "Enable Crowdfunding Page," it should offer a few more fields and explanatory text:
https://www.dropbox.com/s/99netyk85zkrl3c/Screenshot%202014-05-07%2023.10.56.png

Enable Crowdfunding for Request

Dog Registration Data (Saugus Town Clerk)

MuckRock makes it easy to crowdfund payments for requests when agencies ask for money. Explain (briefly) why you think release the information is important, interesting, or otherwise something people should support, and then optionally set how much additional money you'd like and how you would like to use it. Your crowdfunding campaign for the campaign will be begin immediately, while an editor will be in touch with you if you'd like to raise additional funding. Have more questions? Get in touch at [email protected].

and then fields:

Three sentences on why people should chip in:

Total amount you'd like to raise:
(automatically insert cost of request)

If you're asking for additional money, explain what you'd do with it here:

The crowdfunding campaign would then go live trying to raise either the request's costs OR "Total amount you'd like to raise," whichever is lower.

In ADDITION, a new article is saved as non-published with the current date and time and:

Title: Crowdfunding campaign for
Kicker: Help hit the $[total asked amount for campaign] goal

Summary: the three sentences

Body: The three sentences + afterwards.

Author: Whoever submitted the crowdfunding campaign request.

FOIAs: The FOIA the campaign is assigned to.

Also the crowdfund widget of any request tagged in the FOIAs of an article should should show up in article pages that tag that FOIA.

Crowdfund campaigns should also be able to have articles set as children to them, and if an article is the child of a crowdfund request that crowdfund request should pull the summary of it and run that summary under its "Crowdfund this request" header once the article's are approved by an editor and live.

Share tools

Requests and articles should have share tools for Facebook, Twitter, plus easy-to-grab URLs to specific communications. I think we can skip the other social networks for now.

Higher resolution Fav Icon

screen shot 2014-10-16 at 10 09 58 pm

While I appreciate the retro charm, for our new hotness on the site we should have a new favicon to match. Fortunately, it looks what Allan whipped up for the mobile homepage should work nicely?

Fax verification

There should be some sort of fax verification mechanism that scans for incoming fax verification or failure emails, and adds a component that gracefully handles failed requests on site by allowing resubmissions or failures.

Stats background too loud

If you want to do a pattern, that might be ok but should be much more subtle. I'd skip a pattern and go with a solid color altogether to reduce load times however.

screen shot 2014-10-16 at 9 48 10 pm

Fee waiver rejection should not trigger "Payment Required" status

When uploaded, a fee waiver rejection letter currently changes the status to "Payment Required" -- this is typically not actually the case, as many requests remain free even if a fee waiver was declined. A number of users have been confused when their requests are changed to "Payment Required" but see no fee assessed. Furthermore, "Payment Required" stops automatic followups, so there is a risk of these requests drying up unnecessarily.

The status should not be changed to "Payment Required" unless the agency has indicated an amount to be paid.

Ideally, don't change the status at all.

Better "From" fields

We should study how things were between the From field experiments, and come up with a solution that works better for all situations.

Grouping multi-requests

Multi-requests should have preserve the "parent" multi-request and link back to it, so it's easy to see all requests filed as part of a multi-request and analysis can be done on resolutions to related requests.

One-click status setting for bad sender emails

Can we add one-click approval for Bad Sender emails?

Suggested format:

Bad Sender: Receipt of FOIA Request FA-14-0050 (UNCLASSIFIED)

Target email address: [email protected]

Request Admin: http://www.muckrock.com/admin/foia/foiarequest/10944/

Accept email to request: http://j.mp/WHYVfv

Set status as Completed: https://www.muckrock.com/admin/foia/foiarequest/set_status/7706/done/735333/
Set status as Rejected: https://www.muckrock.com/admin/foia/foiarequest/set_status/7706/rejected/735333/
Set status as Partially Completed: https://www.muckrock.com/admin/foia/foiarequest/set_status/7706/partial/
Set status as No Responsive Documents: https://www.muckrock.com/admin/foia/foiarequest/set_status/7706/no_docs/735333/
Set status as Fix Required: https://www.muckrock.com/admin/foia/foiarequest/set_status/7706/fix/
Set status as Payment Required: https://www.muckrock.com/admin/foia/foiarequest/set_status/7706/payment/
Set status as Withdrawn: https://www.muckrock.com/admin/foia/foiarequest/set_status/7706/abandoned/735333/

The "accept" URL would:

  1. Add the email to the request page as a FOIA communication, including all attachments.

  2. Add the email address as the email for

Change email subjects and followup text to reflect each state

Records officers are getting snippy about our blanket use of "Freedom of Information Request" in the subject line of emails and in followups.

Right now all of our requests, federal, state and local, have the following email line:

"Freedom of Information Request: (insert request name)"

And our followup language is as follows:

"I wanted to follow up on the following Freedom of Information request, copied below, and originally submitted on (date)."

This leads to responses like these: "Lastly, the federal freedom of information act applies to federal agencies of which neither the Miami Police nor the Miami-Dade Police Departments are, so you would be better making your request under Florida public records law instead"

Although our request body refers to the appropriate law, these details apparently matter to the people on the other end.

SO, I'd like our email subject lines to be customized to the jurisdiction:

For federal requests: "Freedom of Information Request: (insert request name)"
For state/local requests: "(State public records law) Request: (insert request name)"

And followup language will need to be customized not just for date of submission but also appropriate jurisdiction, as above.

"We can help." screen and link

screen shot 2014-10-16 at 9 56 48 pm

We need to decide what we're promising, and what this will lead to. My instinct would be to a static page, so that we can easily update and tweak it over time. Maybe just muckrock.com/help? Allan, what did you have in mind?

Group Accounts

A system to share both specific requests and a way to set a default sharing of requests.

Grouping cloned requests

It should be easy to see what request a cloned request came from, and follow back it's "lineage" crediting inspiration and helping users explore how FOIAs evolve.

Update invalid address notifications

This needs to be scoped more.


Can we change Invalid Address notifications to make it easier to track down and assign to the appropriate MR #??

Suggested format:

Email title: Invalid Address: Final Response to Request

Target email address: [email protected]

Add email to request: enter MR####

Remember to set the status appropriately before archiving!

The "accept" function would:

  1. Add the email to the request page as a FOIA communication, including all attachments.

  2. Add the email address as the email for the request.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.