O+M 2023-10-27 about data.gov HOT 7 CLOSED

@Jin-Sun-tts The error you posted is the same in this issue:

• #4481

from data.gov.

https://github.com/GSA/catalog.data.gov/actions/workflows/snyk.yml

from data.gov.

Friday 10/23

https://github.com/GSA/data.gov/

Check Catalog Auto Tasks

• DB-Solr Sync:
  87 packages need to be removed from Solr
  880 packages need to be updated/added to Solr
  225 packages without harvest_object need to be mannually deleted
  Finished 826s

• Tracking Update:
  2023-10-27T16:40:20.87+0000 [APP/TASK/ckangeodatagovtracking-update-6664271318-2/0] OUT 2023-10-27 16:40:20,875 INFO [ckanext.geodatagov] 6854 package indexes to be rebuilt starting from 2023-10-25 00:00:00

from data.gov.

Monday 10/23

https://github.com/GSA/data.gov/

Check Catalog Auto Tasks

• DB-Solr Sync:
  0 packages need to be removed from Solr
  1173 packages need to be updated/added to Solr
  186 packages without harvest_object need to be mannually deleted
  Finished 588s

• Tracking Update:
  2023-10-23T07:39:17.47+0000 [APP/TASK/ckangeodatagovtracking-update-6610117038-1/0] OUT 2023-10-23 07:39:17,470 INFO [ckanext.geodatagov] 5883 package indexes to be rebuilt starting from 2023-10-20 00:00:00

Check Harvesting Emails

• Catalog:

Other

• New Relic Alerts

Two issues found from New Relic logs:

• catalog web proxy error for url http://www.google-analytics.com/collect#4508
  We have added www.google-analytics.com to the allow list, see the #4509.

  But in the ckanext-googleanalytics code, HTTP instead of HTTPS is used. We need convince ckan upstream to use HTTPS.

• disable new dataset creation on catalog#4161
  anything starts with /dataset/new is blocked as in /dataset/new-hampshire-and-southern-maine-ocean-uses-atlas.

Checked catalog, inventory production, works fine.

Also checked Solr leader and followers, all work as normal.

from data.gov.

Snyk vulnerabilities check failed at synk test

Running setup.py develop for ckanext_googleanalytics

above errors happened since Oct 4th which just after enable ckanext_googleanalytics

from data.gov.

Above snyk error were fixed, will double check tomorrow morning.

Check catalog auto tasks:

• DB-Solr Sync:
  0 packages need to be removed from Solr
  883 packages need to be updated/added to Solr
  190 packages without harvest_object need to be mannually deleted
  Finished 628s

Check Harvesting Emails

• Catalog:

Other

• New Relic Alerts

The two issues found on Monday also be fixed and confirmed from New Relic log.

from data.gov.

harvest run failed : GSA/catalog.data.gov#1121

2023-10-27T13:55:52.02+0000 [APP/TASK/ckanharvesterrun-6668092691-1/0] ERR ckan.lib.search.common.SearchIndexError: Solr returned an error: Solr responded with an error (HTTP 500): [Reason: Server error writing document id 7b08d7b851606bed7c34f955569a891c4c05cff9 to the index]

after restarted solr leader, harvester rebuild index works.

rerun db solr sync and tracking update after above rebuild index issue fixed.

from data.gov.