Add CKAN Org Dataset Count to Registry about catalog-harvest-registry HOT 2 CLOSED

Notes: This is primarily a diagnostic tool with the goal being able to compare the total dataset count for an organization within the registry to the total dataset count from CKAN.

from catalog-harvest-registry.

Reviving and possibly modifying an old issue in relation to ioos/ckanext-ioos-theme#148 and Solr index/db sync issues in CKAN.

Could we add extra columns to the About page on the Registry as follows:

Harvest Type: CS-W or WAF (unrelated to dataset count, but useful anyway)

Records with Errors: (or 'Bad Records' as labeled in each Harvest Source page) a count of the records in the Registry that failed validation in some way at the Registry validation level.

Also, alongside the Total Records count at the top of the page, can we sum the number of 'Records with Errors' from each harvest source to get a Total Records in Error count, and at the same time store in the Registry db somehow the total CKAN record count (available via CKAN API here: https://data.ioos.us/api/3/action/package_search).

This will allow a rough way to compare the total counts in Registry vs CKAN and see to what degree the inconsistency can be explained by Bad Records count.

To go further, we could subtract (Registry Total Records - Bad Records) - CKAN Total Records = Missing Records to show how many records are 'lost' in the CKAN harvest(s) due to any issues/additional validation/ or errors in the CKAN harvest.

I think this would help.

from catalog-harvest-registry.