nasa-pds / nucleus Goto Github PK

🧑‍🔬 User Persona(s)

Node Operator

💪 Motivation

...so that I can know

• where in the pipeline is currently executing
• what/if something has failed
• when it has completed successfully
• how to see the failure reports from the specific tools

📖 Additional Details

• Capture validate and other tool logs in a consistent fashion
• Monitoring User Guide

⚖️ Acceptance Criteria

Given
When I perform
Then I expect

⚙️ Engineering Details

Develop baseline pipeline for EN Tools and Services necessary for all nodes along with initial documentation outline for extension and adaptation.

💡 Description

Sub-tasks necessary in order to benchmark and develop cost model for Nucleus deployments per number of apps in the pipeline

💡 Description

• #24
• #27
• #23
• #22

💡 Description

We need a consistent approach to logging and monitoring baseline EN components used in Airflow, including a clear and succinct error handling guide for users to know how to track down failures in the pipeline.

Sub-tasks:

• #23

💡 Description

• Contact @thareUSGS to discuss getting access without downloading:
  https://asc-pds-messenger.s3.us-west-2.amazonaws.com/
  • We should probably just replicate this data to a temporary staging bucket for processing, and then drop it for now until we move to a production MCP with appropriate budget allocated.

> aws s3 ls --no-sign-request s3://asc-pds-messenger
                           PRE MESSDEM_1001/
                           PRE MSGRMDS_1001/
                           PRE MSGRMDS_2001/
                           PRE MSGRMDS_3001/
                           PRE MSGRMDS_4001/
                           PRE MSGRMDS_5001/
                           PRE MSGRMDS_6001/
                           PRE MSGRMDS_7001/
                           PRE MSGRMDS_7101/
                           PRE MSGRMDS_7201/
                           PRE MSGRMDS_7301/
                           PRE MSGRMDS_8001/

• Document how we can replicate "external" S3 buckets into a staging bucket

💡 Description

Publish it in the sphynx documentation.

💡 Description

Initial implementation as POC.

The POC will implement this simple workflow:

• pre-condition: scalable harvest runs as a service
• some data is pushed on s3
• validate is trigger
• if validate is successful, data is ingested with harvest-cli

The candidate solution tested are:

• Apache AirFlow
• CWS and/or Camunda Community Edition (depending on when CWS will be upgraded with latest Camunda)

💡 Description

💡 Description

Need to build out the designs for off-nominal behavior in archive management:

• Acceptance of validation errors
• Acceptance of validation warnings
• "floating" files that do not contain labels, but we want in the archive anyways
• Re-run data that has failed unexpectedly due to system failure
• Re-run data that has been edited in the cloud (includes support for how someone could inspect and update data in AWS)
• Remove data, and reload from the ground

🧑‍🔬 User Persona(s)

Node Manager

💪 Motivation

...so that I can adapt or create new workflows in nucleus and know how much it will cost

📖 Additional Details

• Adaptation Cost Model
• Cost Modeling User Guide: Adaptation

⚖️ Acceptance Criteria

Given
When I perform
Then I expect

⚙️ Engineering Details

💡 Description

💡 Description

CWL or some other definition standard

💡 Description

• Validate documentation for deploying new baseline Nucleus
• Validate documentation for configuring Nucleus to validate and harvest data products using baseline deployment

🧑‍🔬 User Persona(s)

Node Operator

💪 Motivation

...so that I can understand what Nucleus is, the baseline tools and services included in the deployment, and what this is all for.

📖 Additional Details

• Nucleus Intro User Guide

⚖️ Acceptance Criteria

Given
When I perform
Then I expect

⚙️ Engineering Details

It was decided to consider the Amazon Managed Airflow (Amazon MWAA) for short term for Nucleus and later consider a more flexible approach such and self managed Airflow in future. This task is to prepare the cost model document considering Amazon MWAA.

💡 Description

Data should be loaded into https://search-img-prod-tlnl5qlzgk5iknwhiemnc6aogy.us-west-2.es.amazonaws.com/

🧑‍🔬 User Persona(s)

Node Operator

💪 Motivation

...so that I can customize the baseline Nucleus deployment for my needs. e.g. how the data will be delivered, or any particular tool config options

📖 Additional Details

• Provide the ability to configure nucleus baseline components
• Configuration User Guide

⚖️ Acceptance Criteria

Given
When I perform
Then I expect

⚙️ Engineering Details

The Amazon Managed Workflows for Apache Airflow (MWAA) has some ongoing costs due to the always up and running Airflow UI. There is no option available to suspend the MWAA environment when it is not in use.

This task is to evaluate the self-managed airflow on AWS EKS as an alternative to MWAA to reduce the operating cost of Nucleus.

See sub-tasks for details, but this initial phase of PDS Nucleus will look into the overall design of the pipeline as well as an initial trade study to determine existing technologies to accomplish this task.

Deliverables:

• Preliminary SRD
• Preliminary set of user stories
• Trade Study

Here are some background docs / diagrams that were the origins of this pipeline idea.

• Original pipeline from IMG recompete design in 2015: https://lucid.app/lucidchart/edf5ba5a-4e7f-42e9-8216-31647790cfb6/edit?invitationId=inv_96ed635f-0664-4119-832c-f70ac984e347
• This eventually morphed into AIMS, which focused on archive metrics. We should now capture this info via the registry, but the overall architecture may be something to consider for utilizing AWS technologies. AIMS_Briefing_Work_Remaining.pptx
• NOTE: This is tightly coupled with the Tracking Service and Tracking UI, but that is more focused on tracking where a product is within the Nucleus processing. So we should be designing / implementing the pipeline with the understanding that we will an ability to track and report on the status of products as they travel through the pipeline (original design docs have been posted to GDrive)

💡 Description

• #52
• #53
• #54 75% complete. tabled for CSS ingestion. will revisit in B15.0
• #75
• #74

Checked for duplicates

No - I haven't checked

🧑‍🔬 User Persona(s)

data provider

💪 Motivation

...so that I can trigger Nucleus with data delivery by a data provider in the cloud

📖 Additional Details

We want to support this use case in a couple ways

1. PDS node data provider - in place S3 connection and ingestion where we do not need to copy the data into our cloud storage, we just want to execute the pipeline against the data and leave it where it is
2. Mission/other data provider - we will want to replicate the data in PDS cloud storage for long term preservation and trigger the pipeline that way (most likely a new requirement on DUM or some DUM-equivalent tool)

Acceptance Criteria

Given
When I perform
Then I expect

⚙️ Engineering Details

Tentatively adding this towards initial CSS migration, but may need to be punted depending on how those efforts go.

🧑‍🔬 User Persona(s)

Node Operator

💪 Motivation

...so that I can add my own custom data processing tools into the baseline nucleus pipeline

📖 Additional Details

• Adaptation User Guide

⚖️ Acceptance Criteria

Given
When I perform
Then I expect

⚙️ Engineering Details

💡 Description

Complete POCs and Trade Study

🧑‍🔬 User Persona(s)

Node Operator

💪 Motivation

...so that I can use an open workflow definition standard encouraged by NASA to define Nucleus workflows.

⚖️ Acceptance Criteria

Working example of a Nucleus workflow created with CWL

⚙️ Engineering Details

The Amazon Managed Workflows for Apache Airflow (MWAA) has some ongoing costs due to the always up and running Airflow UI. There is no option available to suspend the MWAA environment when it is not in use.

This task is to evaluate options to backup and restore the metadata database and logs of MWAA to reduce the operating cost of Nucleus.

💡 Description

Deliverable: detailed technical architecture diagrams describing underlying technologies and interfaces

💡 Description

🧑‍🔬 User Persona(s)

Node Manager

💪 Motivation

...so that I can accurately measure how much it will cost for a baseline Nucleus deployment

📖 Additional Details

• Baseline Nucleus Cost Model
• Cost Modeling User Guide

⚖️ Acceptance Criteria

Given
When I perform
Then I expect

⚙️ Engineering Details

💡 Description

🧑‍🔬 User Persona(s)

Node Manager

💪 Motivation

...so that I can ensure I am within my budget / cost estimate

📖 Additional Details

• Cost Monitoring User Guide

⚖️ Acceptance Criteria

Given
When I perform
Then I expect

⚙️ Engineering Details

The Amazon Managed Workflows for Apache Airflow (MWAA) has some ongoing costs due to the always up and running Airflow UI. There is no option available to suspend the MWAA environment when it is not in use.

This task is to evaluate the self-managed airflow on AWS EC2 as an alternative to MWAA to reduce the operating cost of Nucleus.

💡 Description

It was decided to evaluate Cumulus as a Candidate for the Nucleus Trade Study.

Cumulus is a native cloud-based data ingest, archive, distribution, and management system used in Earth Observing System Data and Information System (EOSDIS) data streams.

Cumulus documentation is available at: https://nasa.github.io/cumulus/docs/cumulus-docs-readme

💡 Description

• Document benchmark metrics for execution time, cost, ?

The Amazon Managed Workflows for Apache Airflow (MWAA) has some ongoing costs due to the always up and running Airflow UI. There is no option available to suspend the MWAA environment when it is not in use.

This task is to evaluate the self-managed airflow with ECS and Fargate as an alternative to MWAA to reduce the operating cost of Nucleus.

It was identified that race conditions can happen while multiple S3 event notifications invoke lambda to update DynamoDB tables (reference #63 ).

To avoid that, a locking mechanism should be implemented.

Implement queuing service for streaming data into Nucleus based on the following design related inputs:

• #60
• #60 (comment)

💡 Description

Currently if we want to develop a new workflow for Nucleus, we have to complete the following steps.

1. Create and push the docker image to ECR
2. Create an EFS volume to share data/configuration files between docker containers (optional, only if the use case must share data/configuration between tasks)
3. Create an ECS task (This can be done through AWS Console or Terraform)
4. Implement a DAG (workflow) primarily using the Airflow ECS Operator
5. Upload DAGs to S3 bucket

We have to execute above multiple steps because different PDS docker images have different setups. For example, some of the PDS docker images need various environment variables and others need various volumes to be mounted. As a result, we cannot use a common template.

This problem is there regardless of the workflow tool we use (not specific to Airflow). With any workflow tool, we will have to maintain different setups for different docker images (for example, when we used docker compose, we had different settings for each components in docker compose file).

Implementing and uploading the DAG is easy if the PDS components are already available as ECS tasks. If we plan to automatically generate Terraform scripts, then we have to discuss,

1. What are the INPUTS that we get from users to generate these scripts?
2. How should those INPUTS looks like?
3. Should we maintain a catalog of docker components that they have to select from?
4. Should we have a UI to select docker components required for a workflow and after making the the selection a way to generate and execute the scripts?

A Workflow Manager Utility can be solution to address above issues.

Checked for duplicates

Yes - I've already checked

🧑‍🔬 User Persona(s)

Node Manager

💪 Motivation

...so that I can know when our budget is approaching or passing our estimated budget.

📖 Additional Details

• Trigger at actual cost >= 80%
• Trigger at projected cost > 100% budget
• Trigger at actual cost >= 100% budget

Acceptance Criteria

• Open up AWS Costs and Reporting
• Verify Budget Report is setup to trigger on budget guidelines noted above.
• Verify email is being sent to [email protected]

⚙️ Engineering Details

💡 Description

Options are EFS, S3, other databases

That could managed as:

• a constraint on the DAG steps implementation, e.g. must support S3 to read/write interfaces
• generic DAG's component running during the execution and made available, somehow to the other components
• specific DAG's running environments providing these specific component interfaces to the other components
• ...

A good place to start is to get inputs from PODAAC on how they do that in Cumulus.

💡 Description

🧑‍🔬 User Persona(s)

Sys Admin

💪 Motivation

...so that I can automatically deploy a new baseline nucleus instance with minimal manual intervention.

📖 Additional Details

• Terraform scripts for deploying in AWS (deploy.sh?)
• Docker container for local deployment (It was decided to use MWAA on AWS and a local deployment is not applicable with MWAA)
• Document any manual requirements

⚖️ Acceptance Criteria

Given
When I perform
Then I expect

⚙️ Engineering Details