Replace old sqs-to-step and post-step repos with the new github.com hitide-backfill-lamddas

BBox overlap GPoly Granule Count

Update the backfill-tool to use forge-py for specific collections

• Update forge-tig-configuration for specific collections to use forge-py

AC:

• Backfill Tool will generate DMRPP files using the dmrpp lambda module on a per collection basis
• Performance greatly improved processing SWOT_L1B_HR_SLC_2.0 and other SWOT collections

Tickets:

1. Implement DMRPP Lambda Module (Done by GHRC team)

   GHRCCLOUD-6025
   Publish a lambda of the Cumulus DMRPP Generator Module

2. #11

3. #12

4. #13

Move repos from JPL Github to Github.com:

• hitide-backfill-sqs-to-step
• hitide-backfill-post-step

and merge them into one new repo:

• hitide-backfill-lambdas

Tickets:

1. Create a new hitide-backfill-lambdas repo
2. podaac/hitide-backfill-lambdas#1
3. podaac/hitide-backfill-lambdas#2
4. podaac/hitide-backfill-lambdas#3
5. #19
6. #20

Monitor cloudwatch, update logs for more profile info.

Depends on GHRCCLOUD-6025 being completed first.

AC: Backfill Tool uses new dmrpp lambda module for dmrpp processing

All footprints failed on the last run of the Backfill-Tool.
Please fix this collection generating footprints

AC: Limit resources used for tig on a per collection bases. tig size is big enough for all granules in each collection, but not too big that resources are not used.

Tickets:

1. #16
2. #18

Add new workflow to backend to support cleaning CMR bounding boxes when polygons are present.

This is for running tig in the backfill tool backend.

For each collection, get memory limit, cpu limit, and hd space needed to run tig.

Will be used to limit resources in backfill tool for tig to reduce costs.

Check performance and compare to Dmrpp ECS method. Tweak parameters to make more efficient.

Details here:

https://jira.jpl.nasa.gov/browse/PODAAC-5820

AC: Backfill tool uses dmrpp lambda module on a per collection basis

AC:

• Generate footprints with python code, with performance and accuracy in mind
• Try to use off-the-shelf components
• Go from a raster/image to a "footprinter" module then to wkt (Maybe use gdal_footprint)
• Make it easy to add new footprinter code for new collections very quickly. Want to be able to support DPub when adding new collections to ingest in Cumulus
• Design so it can handle standard footprint strategy types like periodic, linestring, polar, etc

Tickets:

1. Create new forge-py repo (Done)
2. podaac/forge-py#1
3. podaac/forge-py#2
4. podaac/forge-py#3
5. #17

If time permits:

Setup specific strategy for AVHRRMT*_G-NAVO-L2P-v2.0 in python code (DE?) (If we have time)
Setup specific strategy for AVHRR19_G-NAVO-L2P-v1.0 in python code (DE?) (If we have time)

Need to add code to error if image mode should be off.

Like this:

imgVariables = forge_tig_config.get('imgVariables')
if imgVariables is not None and len(imgVariables) > 0:
args.image = 'on'