unity-sds / unity-analytics-sdap Goto Github PK

The following AWS/EKS resources are needed:

• An EKS bastion host that I can ssh to that has access to kubectl and helm.
• 1 node of type m5.xlarge that hosts the SDAP web application, spark application driver, and data ingest components. This one should have a node taint of sdap-head:NoSchedule.
• 3 nodes of type r5.2xlarge that mirror each other with each hosting identical replicas of Solr, Cassandra, Zookeeper, and a few Spark executors.

This task is to develop a script to preprocess data files into a format suitable for SDAP ingest. In general SDAP requires a gridded data product that follows CF conventions (https://cfconventions.org/) and has properly structured time, latitude and longitude variables.

Deploy the development branch of SDAP that supports Cloud Optimized GeoTIFF (COG) as the data store to the Unity-Venue-Dev project account.

Acceptance Criteria:

• SDAP with COG support deployed to Unity-Venue-Dev
• Ingest test data into SDAP
• SDAP time series endpoint tested with test data

Ingest sample of SBG L2 data products in SDAP

Acceptance Criteria:

• Sample of SBG L2 data products registered as SDAP dataset
• New SBG L2 data product displayed in SDAP /list inventory endpoint
• Run SDAP time series on new SBG L2 dataset

Determine which product(s) we should ingest into SDAP for SBG analysis examples. Work with SBG team to confirm that correct product(s).

Configure an API to call SDAP endpoints in Unity-Venue-Dev in the Unity API Gateway in Unity-Venue-Dev. This API will be called from the shared API Gateway in Unity-Dev, but does not need to be compatible with the DAPA standard.

Acceptance Criteria:

• API in the Unity API Gateway in Unity-Venue-Dev forwards calls to SDAP deployed in Unity-Venue-Dev
• Test connectivity with SDAP time series call.

Automate deployment of SDAP to MCP/EKS with Terraform.

Acceptance Criteria:

• EKS Cluster provisioned in Unity-Venue-Dev
• Separate EKS node groups: (i) 1 m5.xlarge for SDAP orchestrator/webapp and 1 (scalable up to 3) r5.2xlarge (memory-optimized) for SDAP workers/executors
• SDAP ingress component deployed
• SDAP application load balancer deployed

as per https://jaas.gsfc.nasa.gov/servicedesk/customer/portal/2/GSD-2527

there are several Ec2 instances and Launch templates that point to expired AMIs:

Unity-Dev i-04cd881f3c9bc8559 uas-dev-sdap-sdap-workerNodeGroup-Node
Unity-Dev i-0397444780dce353f uas-dev-sdap-sdap-driverNodeGroup-Node

and Launch templates that presumably launch the above instances:

Unity-Dev lt-07a978ccf2ac99178 eksctl-uas-dev-sdap-nodegroup-sdap-driverNodeGroup
Unity-Dev lt-0be97bfac448cee6d eksctl-uas-dev-sdap-nodegroup-sdap-workerNodeGroup

Probably best to remove this deployment entirely from unity-dev as that's a shared services account.

What does this look like?

Is this a deployed service in a given tenant? shared across tennants?

Acceptance Criteria

• Deployment models / diagrams
• Cost Models

What else?

Configure an SDAP endpoint that is suitable for a health check (runs quickly and returns 200 code) in the Unity SSM parameters. The SSM endpoint format is /unity/healthCheck/<MARKETPLACE_ITEM>/<COMPONENT_NAME>. The SSM parameters should follow the Unity conventions described at https://app.gitbook.com/o/xZRqGQeQXJ0RP4VMj7Lq/s/UMIRhLdbRQTvMWop8Il9/developer-docs/common-services/docs/users-guide/deployment/deployment-concepts-and-infrastructure/unity-ssm-parameters.

SDAP is deployed to Unity-Venue-Dev environment and a suitable health check endpoint that retrieves an inventory of datasets in SDAP is https://cjezi7waw8.execute-api.us-west-2.amazonaws.com/unity-as-venuedev-sdap/sdap/list. In this ticket we need an API in Unity-Venue-Dev under path /health/sdap. The SSM parameter is /unity/healthCheck/analysis-services/sdap/url.

Acceptance Criteria:

• An SDAP endpoint to be used for a health check returns within 10 seconds and returns 200 code.
• SDAP health check endpoint is added to the venue SSM parameters during deployment.
• Terraform script to automate deployment.
• Test deployment automation