act-now-coalition / can-scrapers Goto Github PK

I think we may have a bug in the Delaware scraper

The Date column seems to be set for each csv in the fetch method

Then in normalize we are using dt=self._retrieve_dt("US/Eastern") instead of that Date column

I think the fix should be as simple as renaming Date -> dt in .fetch and then not trying to assign to dt in .normalize

@ChrisBremer does that sound right to you?

Proposal (after call with @cc7768 )

We need to determine category, measurement, unit triplets for capturing vaccine data.

Here is a proposal (in csv form) for how we might do this

category,measurement,unit
moderna_vaccine_allocated,cumulative,people
moderna_vaccine_distributed,cumulative,people
moderna_vaccine_initiated,cumulative,people
moderna_vaccine_completed,cumulative,people
pfizer_vaccine_allocated,cumulative,people
pfizer_vaccine_distributed,cumulative,people
pfizer_vaccine_initiated,cumulative,people
pfizer_vaccine_completed,cumulative,people

https://www.vdh.virginia.gov/coronavirus/

https://covidtracking.com/race/dashboard

http://ww11.doh.state.fl.us/comm/_partners/covid19_report_archive/vaccine/vaccine_report_latest.pdf

Context: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/

We use set-output in some of our CI/CD pipelines, e.g., https://github.com/act-now-coalition/covid-act-now-website/blob/develop/.github/workflows/ci.yml

The dept. of health page here has a link to an xlsx file that they claim is updated daily.

the xlsx file is here: https://www.dshs.state.tx.us/immunize/covid19/COVID-19-Vaccine-Data-by-County.xls

https://healthdata.gov/dataset/covid-19-vaccine-distribution-allocations-jurisdiction-moderna

@jrybacek volunteered for this one 🎉

This may be a starting point to scrape: https://www.cdph.ca.gov/Programs/CID/DCDC/Pages/COVID-19/VaccineDoses.aspx

Context: https://www.prefect.io/guide/blog/freezing-legacy-prefect-cloud-1-accounts-on-starter-and-standard-plans/

We use Prefect Cloud 1 at the moment. I believe this is a separate task from updating our workflows from Prefect 1 to Prefect 2, which requires significant code changes

It looks like the CDC testing data scraper does not include DC. Unsure of why, as DC is included in us.STATES