podaac / hitide Goto Github PK

High-level Tool for Interactive Data Extraction

License: Apache License 2.0

hitide's Introduction

hitide

HiTIDE, the High-level Tool for Interactive Data Extraction, allows users to subset and download popular PO.DAAC level 2 (swath) datasets. Users can search across a wide variety of parameters, such as variables, sensors and platforms, and filter the resulting data based on spatial and temporal boundaries of interest to the user. HiTIDE goes even further, offering instant previews of variable imagery, allowing users to rapidly find data of interest for download and further, rigorous scientific analysis.

Component Repositories

https://github.com/podaac/hitide-ui
https://github.com/podaac/hitide-profile
https://github.com/podaac/l2ss-py
https://github.com/podaac/concise
https://github.com/podaac/forge
https://github.com/podaac/tig
https://github.com/podaac/forge-tig-configuration
https://github.com/podaac/cmr-umm-updater
hitide-backfill-tool (Not published)

hitide's People

Contributors

Stargazers

Watchers

hitide's Issues

Migrate cumulus-postworkflow-normalizer to Github.com

Determine changes needed for DiT

https://wiki.earthdata.nasa.gov/pages/viewpage.action?spaceKey=ENS&title=PI+24.1+Tenant+Action+Items

Needed for repos:

hitide-ui
hitide-profile

Outline process to add new Collections to Hitide

Update EDC Security (OSA)

Make tig easily runnable by DPub to generate image thumbnails

Setup README for DPub to use tig.

Make super easy to run

Update forge-tig-configuration to run correction code on SWOT collections

Improve performance for SWOT collections for DMRPP backfilling

On-premise HiTIDE retirement and remove dependencies in HiTIDE Cloud

Add missing/needed collections to HiTIDE

Details here:

https://docs.google.com/spreadsheets/d/1crT6KK19rPZSxS8r45Wupd1ikTp02Pn-MnnTtcnaYMo/edit?usp=sharing

Release Backfill-Tool v0.4.0

Release Backfill Tool v0.4.0 to also deploy to new forge and tig to Cumulus AWS

Create a "Retiring on-prem HiTIDE" html page. Like done for SOTO.

Migrate hitide-backfill-post-step to Github.com

Reduce cost of tig running in AWS

Reduce cost of tig running in AWS and add tags to services to track hitide costs more precisely

Backfill collections in Phase 8

Run Phase 8:
https://wiki.jpl.nasa.gov/display/PD/Backfilling+Progress+Report

Release Backfill Tool v0.4.1

Release Backfill Tool v0.4.1
Includes new:
forge 0.9.0
tig 0.8.0

I&A: Enable SWOT 1.1 in Reassociation Tool

Backfilling HiTIDE Collections

List of collections to backfill TBD

Redirect hitide on-prem url to html page

Test SWOT_L2_LR_SSH_EXPERT_1.0 and SWOT_L2_LR_SSH_EXPERT_1.1 ssha_karin_2 correction thumbnail

Use the backfill tool to update some SWOT_L2_LR_SSH_EXPERT 1.0 and 1.1 granules and produce the new thumbnail image for ssha_karin_2. This image should be a high resolution image and should be verified with Jinbo that it is correct.

Update CMR (umm-g) to add variable name (umm-v name) in description for each PNG RelatedUrl

Add new SWOT 1.1 associations to Hitide, l2ss-py, concise, etc

Add collections to HiTIDE:

SWOT_L2_LR_SSH_BASIC_1.1
SWOT_L2_LR_SSH_EXPERT_1.1
SWOT_L2_LR_SSH_WINDWAVE_1.1

This includes adding functionality to run subsetting (L2ss-py), merging (l2ss-py with concise), and include footprints and thumbnail images, and Collection Info in HiTIDE fully complete and correct.

Update automated tests for HiTIDE V4.16.0 in OPS and UAT

UI changes in v4.16.0 seemed to have broken the tests. Please update tests to pass again.

Migrate hitide-backfill-sqs-to-step to Github.com

Migrate hitide backfill backend repos to Github public

Move repos from JPL Github to Github.com:

hitide-backfill-sqs-to-step
hitide-backfill-post-step
cumulus-postworkflow-normalizer

Add SWOT L2 Collections to HiTIDE

Here's the official SWOT L2 Collections list to add to HiTIDE:

SWOT_L2_NALT_IGDR_SSHA_1.0
SWOT_L2_NALT_OGDR_SSHA_1.0
SWOT_L2_LR_SSH_BASIC_1.0
SWOT_L2_LR_SSH_EXPERT_1.0
SWOT_L2_LR_SSH_WINDWAVE_1.0

Can NOT do:

SWOT_L2_NALT_GDR_SSHA_1.0 (No data yet)
SWOT_L2_LR_SSH_UNSMOOTHED_1.0 (Footprint issues, will revisit in a future PI)

Update and add gen_dataset_config.py to a repo for DPub

Enable collections in cumulus-configurations

Details defined here:

https://jira.jpl.nasa.gov/browse/PODAAC-5927

Being done by I&A, Stepheny.

Add SWOT 1.0/1.1 Collections to HiTIDE

Acceptance Criteria:

Added the following collections to HiTIDE, and have the ability to subset (l2ss-py) and merge (concise) inside HiTIDE.

Collections include:

SWOT_L2_NALT_IGDR_GDR_1.0
SWOT_L2_NALT_IGDR_SGDR_1.0
SWOT_L2_NALT_OGDR_GDR_1.0
SWOT_L2_NALT_IGDR_1.0 (?)
SWOT_L2_NALT_OGDR_1.0 (?)
SWOT_L2_RAD_GDR_1.0 (?)
SWOT_L2_LR_SSH_1.0 (?)
SWOT_L2_HR_Raster_1.0 (?)
SWOT_L2_LR_SSH_1.1 (?)
SWOT_L2_LR_SSH_BASIC_1.1
SWOT_L2_LR_SSH_EXPERT_1.1
SWOT_L2_LR_SSH_WINDWAVE_1.1
SWOT_L2_LR_SSH_UNSMOOTHED_1.1
SWOT_L2_HR_RiverSP_1.1 (No forge & no tig) (?)
SWOT_L2_HR_RiverSP_node_1.1 (No forge & no tig) (?)
SWOT_L2_HR_RiverSP_reach_1.1 (No forge & no tig) (?)
SWOT_L2_HR_LakeSP_1.1 (No forge & no tig) (?)
SWOT_L2_HR_LakeSP_obs_1.1 (No forge & no tig) (?)
SWOT_L2_HR_LakeSP_prior_1.1 (No forge & no tig) (?)
SWOT_L2_HR_LakeSP_unassigned_1.1 (No forge & no tig) (?)
SWOT_L2_HR_Raster_1.1 (No forge) (?)
SWOT_L2_HR_Raster_100m_1.1
SWOT_L2_HR_Raster_250m_1.1

(?) Should add to HiTIDE?

SWOT 2.0 collections: Fix thumbnail alignment with footprint

The thumbnail should be inside the footprint.

Collections include:

SWOT_L2_LR_SSH_BASIC_2.0
SWOT_L2_LR_SSH_EXPERT_2.0
SWOT_L2_LR_SSH_WINDWAVE_2.0

Add GitLeaks check to some HiTIDE repos in Github Actions

Doing these repos:
podaac/l2ss-py#224
podaac/concise#97
podaac/forge#28
podaac/tig#38

Backfill SWOT NALT collections so all parent and sub collections have footprints

Remove UMM-G bounding box in HiTIDE collections when GPoloygon exists

Collection List details here:

https://jira.jpl.nasa.gov/browse/PODAAC-6117

Work done by Jack, but Jim may try to help to get this done sooner.

New DPub Tool/Process to Add Collections to HiTIDE

Acceptance Criteria: Work with Dpub team to come up with a procedure and deploy software for DPub to run and be able to (re)generate forge-tig-configuration files and generate thumbnail images using TIG, and generate footprints using FORGE. Plan to start simple and then improve the process and "ease of use" over time.

Update backfill-tool post_to_cmr to match Cumulus 16

Update this:
https://github.jpl.nasa.gov/podaac/hitide-backfill-tool/blob/develop/terraform-deploy/post_to_cmr.zip

and this:
https://github.jpl.nasa.gov/podaac/hitide-backfill-tool/blob/develop/terraform-deploy/post_to_cmr.tf

to support Cumulus 16.

Need to check with @skorper to get info on new version.

Remove all "XXXXX-PODAAC" collections from HiTIDE

HiTIDE POCLOUD shouldn't contain any collections that end in -PODAAC.

Verify here:
https://cmr.earthdata.nasa.gov/search/tools.umm_json?name=hitide&pretty=true&provider=pocloud

and in the HiTIDE UI

Add SWOT 2.0 Collections to HiTIDE

Should work according to this JIRA ticket:

https://jira.jpl.nasa.gov/browse/PODAAC-6013

Add configs to forge-tig-configuration for image only:
SWOT_L2_LR_SSH_BASIC_2.0
SWOT_L2_LR_SSH_EXPERT_2.0
SWOT_L2_LR_SSH_WINDWAVE_2.0
SWOT_L2_NALT_GDR_SSHA_2.0
SWOT_L2_NALT_IGDR_SSHA_2.0
SWOT_L2_NALT_OGDR_SSHA_2.0

Add to forge-tig-configuration for footprint only:
SWOT_L2_NALT_GDR_2.0
SWOT_L2_NALT_IGDR_2.0
SWOT_L2_NALT_OGDR_2.0

Associate to l2ss-py service:
SWOT_L2_LR_SSH_BASIC_2.0
SWOT_L2_LR_SSH_EXPERT_2.0
SWOT_L2_LR_SSH_WINDWAVE_2.0
SWOT_L2_NALT_GDR_GDR_2.0
SWOT_L2_NALT_GDR_SGDR_2.0
SWOT_L2_NALT_IGDR_GDR_2.0
SWOT_L2_NALT_IGDR_SGDR_2.0
SWOT_L2_NALT_OGDR_GDR_2.0
SWOT_L2_NALT_GDR_SSHA_2.0
SWOT_L2_NALT_IGDR_SSHA_2.0
SWOT_L2_NALT_OGDR_SSHA_2.0

Associate to HiTIDE Tool:
SWOT_L2_LR_SSH_BASIC_2.0
SWOT_L2_LR_SSH_EXPERT_2.0
SWOT_L2_LR_SSH_WINDWAVE_2.0

Tickets here:

Add correction code to Backfill workflow (inside tig repo)

Associate SWOT 2.0 collections with L2ss-py service

OPS Team shutdown hitide on-prem, including all services

OPS team want to shut down all hitide on-prem services on the week of September 11th.

Fix running full collection MODIS run, cmr times-out or fails after 3 hours.

Verify SWOT 2.0 NALT sub collections have footprints

Automated collection tests have differences, go through each collection and resolve the issues to get the tests passing

What are the differences exactly? Comparing to CMR?

Testrails runs here:
https://cae-testrail.jpl.nasa.gov/testrail/index.php?/runs/view/33983&group_by=cases:section_id&group_order=asc

Get correction code from Jinbo and decide where to put it

THREDDS Cloud evaluation

Talk with Curtis

Ed Armstrong too.

Joe Roberts

Reach out to company (Unidata)

Haley Johnson (Unidata)

Wikio?

Migrate JPL repos to github.com

Next repos to migrate are:

forge
tig

SWOT SSHA Karin 2 Thumbnail Correction

Acceptance Criteria: After the reassociation tool runs the Backfill tool, the "system" (either tig itself or a new step in the AWS Step function) updates and corrects the ssha_karin_2 thumbnail image for all processed granules.

Setup automated tests for SWOT L2 collections in HITIDE UI

Add automated tests for these collections.
Test in both OPS and UAT environments.

SWOT_L2_LR_SSH_BASIC_1.0
SWOT_L2_LR_SSH_EXPERT_1.0
SWOT_L2_LR_SSH_WINDWAVE_1.0
SWOT_L2_NALT_GDR_SSHA_1.0 *** SKIP FOR NOW ***
SWOT_L2_NALT_IGDR_SSHA_1.0
SWOT_L2_NALT_OGDR_SSHA_1.0

Skip SWOT_L2_NALT_GDR_SSHA_1.0 for now though since it has no data.