alphagov / datagovuk_publish Goto Github PK

View Code? Open in Web Editor NEW

9.0 52.0 5.0 42.83 MB

Beta version of publish data

Home Page: https://docs.publishing.service.gov.uk/apps/datagovuk_publish.html

Ruby 75.58% JavaScript 0.08% HTML 18.99% Shell 1.99% Dockerfile 0.23% SCSS 3.09% Procfile 0.04%

datagovuk govuk

datagovuk_publish's Introduction

data.gov.uk Publish

This repository contains the beta-stage publishing component of data.gov.uk.

Deployment

Continuous Integration has been setup using Github Actions.

Tests are run on pull requests.
Deployments to Integration happen automatically when marging branches into the main branch.
In order to carry out a release to production a developer in the govuk team will need to create a release tag with a leading v and approve of the deployment in Github Actions.

Integration

To deploy to integration merge a PR into main.

Staging & Production

To deploy to staging/production you need to tag the release, the tag needs to be in this format - v9.9.9 where 9 is a number and the leading v is required. E.g. v0.1.11 is valid, 0.1.11 is not.

This will create a PR on govuk-dgu-charts which you should be able to approve and merge into main for testing.

Test that your changes are working in staging by looking at the publish pod logs for evidence of jobs being processed.

Then merge in the production release PR.

Prerequisites

You will need to install the following for development.

rbenv or similar to manage ruby versions
bundler to manage gems
opensearch search engine
postgresql database

Most of these can be installed with Homebrew on a Mac.

How to run this repo locally

There are currently 2 ways to run this repo locally:

Via govuk-dgu-charts - An end to end setup from ckan to opensearch to Find. This is the presently most supported means for running Find and is recommended for local development. Instructions for how to setup and run Find this way available on the linked repo.
Manual installation. Instructions for this below. (needs CKAN running)

Install requirements for this app using Homebrew

## PostgreSQL
brew install postgresql

## Redis
brew install redis

## Opensearch
brew tap caskroom/versions
brew cask install java8
brew install opensearch

Start the services on your machine

brew services start postgresql
brew services start opensearch
brew services start redis

Update config settings

Configure the base URL of your local CKAN in ./config/environments/development.rb:

config.ckan_v26_base_url = "http://localhost:4000"

Install dependencies, initialise the database and search index:

bin/setup

Start the web server

rails s

Then navigate to http://localhost:3000.

Run Sidekiq jobs

These need to be run to sync data from CKAN.

Set up the workers, these sync organisation data and their datasets:

bin/rails runner CKAN::V26::CKANOrgSyncWorker.new.perform
bin/rails runner CKAN::V26::PackageSyncWorker.new.perform

Then run Sidekiq to process the queue:

bundle exec sidekiq

When you create new organisations and datasets in Publish, you will have to run these commands again to trigger the sync. These should then appear in Find.

Clear the database

To completely clear the database:

bin/rails db:drop db:setup

Re-index Opensearch

To re-index Opensearch based on the current database contents, run:

bin/rails search:reindex

Troubleshooting

Flush Redis

This may be necessary if you're having issues trying to completely reset your CKAN stack and start over with no data. See the next section below as an example.

$ redis-cli flushall
OK

Check the database size is 0:

$ redis-cli
127.0.0.1:6379> dbsize
(integer) 0

Delete the index on opensearch

If when attempting to run bin/rails search:reindex you get An alias can not be assigned to index of the same name. Please delete index 'datasets-development' before continuing, then you should run the following command on the publish pod before running the search reindex:

$ curl -XDELETE $ES_HOST/datasets-development

Running the PackageSyncWorker sidekiq job attempts to sync non existent data

When running this sidekiq job it returns errors in the terminal such as:

404 Not Found excluded from capture: DSN not set
{"@timestamp":"2019-06-06T10:03:58Z","@fields":{"pid":43034,"tid":"TID-oxw3pfczg","context":" CKAN::V26::PackageImportWorker JID-3b2dff4c5d230d1d27cc5bea","program_name":null,"worker":"CKAN::V26::PackageImportWorker"},"@type":"sidekiq","@status":"fail","@severity":"INFO","@run_time":0.545,"@message":"fail: 0.545 sec"}

Ensure you have the correct config settings - see Update config settings
Try to flush redis
You will also need to purge SOLR via CKAN
Clear the Publish database
Then re-run sidekiq jobs - see Run sidekiq jobs

Documentation

See here for all of our Architecture Decision Records.

datagovuk_publish's People

Contributors

Stargazers

Watchers

Forkers

waffle-iron idavidmcdonald joshpencheon alphagov-mirror uk-gov-mirror

datagovuk_publish's Issues

Errors on login form

Manage page has no search

Title field description should say 100 char limit (not 65)

From a beta Publisher:

The field description says titles should be less than 65 characters but the field limit is 100, it's a bit confusing.

remove 'remember me' on login page

Not a pattern we commonly use in gov

Tasks and Broken Datasets logic needs implementing

Currently the Tasks page is displaying all tasks and all datasets.

Dataset form flow needs implementing

This includes adding datafiles.

'my datasets' is not populated

JavaScript for Choose-Licence page

Update README for how to load locations data

We need a rake import task that can load in some locations data to pre-populate the database.

Validation errors not showing on publish page

When publishing validation is stricter, if the dataset fails to validate, no errors are shown on the publish page.

Add files page

test waffle integration issue

Extra dataset validations prior to publishing

Broken intermediate table: organisations_users

organisation_users has column called publishing_user_id that breaks queries like current_user.organisations

Overdue tasks

Create a task when a dataset has an overdue datafile

Make application text easy to translate

internationalisation, localisation, you name it. http://guides.rubyonrails.org/i18n.html

CSS styling for some components is broken

Currently .masthead doesn't flow the full-width of the screen,~~and button styling is broken.~~

Generate a data.json file

Publish Data (as it has all of the metadata) should generate a data.json file (on a schedule) that can be hosted somewhere but made available for Find-Data at https://..../data.json

This will allow other catalogues to harvest ours without hammering an API as it does currently.

Re-enable sorting on the manage page.

radio button styles only picked up if you land on the page after pressing skip on the previous page

Fallout of #86

Refactor task generation code

The code for task generation repeats itself quite a lot, and could be cleaned up in such a way as to be re-usable within the app at runtime (if necessary).

Frequency form, start date, end date for datasets

Use slugs instead of IDs for datasets

Fix manage links

The links in the manage pages for Add Data and Edit go nowhere, we need to make them go to the right place.

Heroku banner

My suggestion:

Add a new rails env called nightly or staging (that just inherits from production).
Detect in layout template if this env is active.
Show banner if it is.
Set RAILS_ENV=(nightly|staging) on Heroku

Dataset import

Import legacy datasets into Beta. Need to check models for the extended metadata where the dataset also has INSPIRE metadata attached.

JavaScript on choose-a-licence page

Re-enable sorting on tasks page

Form styles for dataset creation/edit

Titles for unpublished datasets

Javascript on location form

Make datasets links go to find-data

Use an env var to specify where find-data is ( http://find-data-beta.herokuapp.com/ ) and then when we have links in manage data to view them, we can direct users to FIND_DATA_URL/dataset/{{ name }}

install govuk_elements and govuk_template

Properly using https://github.com/alphagov/govuk_elements_rails

Dataset creation specs

Manage My/Organisation datasets

Manage datasets for users where they are assigned as owner to the dataset, and also show datasets for that user's organisation.

Need to support closed organisations....

Some organisations are closed and have been replaced. We need to change the model to account for this and then reload the organisations.

Menu options (in layout) for logged in users.

When users are logged in, they should have more menu options as per Alpha (for now).

Broken datalink task

Create a task when a dataset has a broken datalink
Implement @brokendatasets in the tasks controller

Stop codeclimate complaining

Manage page will require pagination

Find out how best to do pagination and implement it on the manage page.

Validation rules + content

Document validation rules from Alpha so we have a starting point. Validation is different when creating a dataset (for draft) than for publishing, which is much stricter. There are rules around when we can publish without a datafile, or with broken datafiles and we should make sure these are also documented.

Set creator_id when creating a new dataset

When we create a dataset, set creator_id to the ID of the current_user

Soft Deletion + Redirects for duplicate datasets

Sometimes we end up in the situation (because of harvesting) where an organisation publishes duplicates and the old ones need removing.

Rather than deleting the old datasets, where possible we should allow for them to be forwarded to the replacement dataset. Even if this is a manual or API driven task. We don't want to break people's bookmarks.

Location API

/api/locations?q=bl

returns a list of corresponding location names, eg:
["Blaby (local authority)", "Ribble Valley (local authority)", "Blackburn with Darwen (local authority)", "Blackpool (local authority)", "Hambleton (local authority)", "South Ribble (local authority)", "Blaenau Gwent (local authority)", "Blackburn with Darwen (NHS Clinical Commissioning Group area)", "Blackpool (NHS Clinical Commissioning Group area)", "Chorley and South Ribble (NHS Clinical Commissioning Group area)", "Hambleton, Richmondshire and Whitby (NHS Clinical Commissioning Group area)"]

List of locations is at https://github.com/datagovuk/publish_data_alpha/blob/master/src/datasets/fixtures/locations.json

Load data
Make API

check datalinks have valid URLs in the dataset creation flow

Define abilities for permissions check

After the initial /dataset/new, or when editing, we should check the user is either the dataset creator, or is in the same organisation as the dataset.

Might want to wait until after edit is done to do this.