nzherald / nz-covid19-data Goto Github PK

Repackaging the Ministry of Health's Covid-19 data

R 99.88% Shell 0.12%

covid-19 covid19 covid19-data covid19-nz new-zealand

nz-covid19-data's Introduction

New Zealand Covid-19 Case Data

28 March - most of this has been made obsolete by the Ministry's release of spreahsheets - but I'll wait a few days before I clean this up as things are changing very rapidly.

This data has been quickly pulled together by the New Zealand Herald data journalism team - if you have any issues please contact Chris Knox by email [email protected] or via Twitter

Updated to 'Updated 5pm Thursday 26 March 2020'

Daily totals from Feb 28 - NA where data wasn't reported csv and excel
DHB totals starting March 26 csv and excel
Confirmed cases DHB, Age, Gender, Flight details (no information extracted), and Date csv and excel

March 26 update

The Ministry has changed to only reporting the details for confirmed cases - and to providing location data by DHB - which is an improvement on the previous ad-loc locations.

Since the data released previously was not reported by DHB there is no way to link the two. Therefore the final versions of the cases data are now in data/archive I have also copied the could used to produce those files to data/archive/archive.R

A little detail

The data is from https://www.health.govt.nz/our-work/diseases-and-conditions/covid-19-novel-coronavirus/covid-19-current-cases

The data is downloaded and cleaned up using R - all code is in _drake.R

There are a selection of files in the data directory.

I am using drake to process manage the update process - I will turn this into an R data package when I get the time.

Pull requests are very welcome.

The code is a MIT licensed and the data is under the Ministry of Health's creative commons license https://www.health.govt.nz/about-site/copyright

nz-covid19-data's People

Contributors

Stargazers

Watchers

Forkers

petraoleum simontegg finlay jillwithaj leighghunt fogonwater uoa-eresearch kcha193 guptasamarth61

nz-covid19-data's Issues

More direct access to ESR database

Hi @vizowl

I did some poking around the arcgis.com dashboard linked from the MOH webste and found that it is pulling some tables that appear to be obtained from, or maintained by https://www.esr.cri.nz/. This gels with the intel @gizmoguy gathered earlier in the week (I suspect he looked at the author names in the spreadsheet metadata and tied that back to ESR).

I worked out that the right query string dumps these tables in JSON format:

Perhaps using these would be better than working with the spreadsheets?

Regards,
Simeon.

Dataset in CoronaWhy Dataverse

Hello,Your dataset was added to CoronaWhy (https://www.coronawhy.org/) Data Lake on Dataverse as a piece of common COVID-19 dataframe http://datasets.coronawhy.org/dataset.xhtml?persistentId=doi:10.5072/FK2/NC8TLY

Would you be willing to help with maintenance of your dataset in Dataverse, e.g. adding the relevant metadata and keeping the dataset up-to-date? That will help to make the dataset findable and accessible for medical science community.

Running count of tests

Is that published anywhere? A.B. just said 9780 so far, if I heard correctly

South Canterbury?

South Canterbury isn't a Regional Council region (though it's a DHB). Should it be merged with Canterbury in the Region field?

Dates?

I have made a request to MoH (through the feedback button) to add date of likely infection, swab date, and recovery/death date. You may have more away than me though.

Automated scraping of MOH and other sites

Hi there!

I have been scraping the MOH, MSD and covid19.govt.nz sites for a few days now and making the data available via a web API in various formats:

https://nzcovid19api.xerra.nz/

You might find the code that does the scraping useful in an offline workflow (the command line tool can output CSV and JSON): https://github.com/simeonmiteff/nzcovid19cases

Some researchers have asked me for reported dates, and I have spent some time manually joining these with the MOH table in this Google Sheet (which pulls the table from the API's CSV feed using the IMPORTDATA() function that updates every 30 minutes):

https://docs.google.com/spreadsheets/d/1NJiDg2z7Y79RUkQ-a648fFD_EmziztYLLkQW_FOrLGo/edit#gid=576672848

Please let me know if I can help out in some way.

More generally, I know of at least one initiative to build a sensible data format for MOH to do their reporting, but I don't know if there is anyone on the ministry's end to take this on (especially given that I think they have bigger problems right now).

I also found that the MOH site gets overloaded once a day, and at other times has a nasty caching bug where it alternately serves the latest, and outdated versions of the COVID-19 case page. I've been trying to find out who the tech contact is there (to offer help), but no luck yet. Please let me know if you know someone who can connect me at MOH.

Regards,
Simeon.

Consider contributing downstream to this global dataset?

https://github.com/lazd/coronadatascraper

New source of individual case information?

I guess you folks have seen that MOH reworked the cases page now to show only case statistics.

Does someone who MOH takes seriously (NZ Herald?) know if they plan to publish individual case information at all?