nasa-pds / nucleus Goto Github PK
View Code? Open in Web Editor NEWNucleus is a software platform used to create workflows for the Planetary Data (PDS).
Home Page: https://nasa-pds.github.io/nucleus
License: Apache License 2.0
Nucleus is a software platform used to create workflows for the Planetary Data (PDS).
Home Page: https://nasa-pds.github.io/nucleus
License: Apache License 2.0
Node Operator
...so that I can know
Given
When I perform
Then I expect
Develop baseline pipeline for EN Tools and Services necessary for all nodes along with initial documentation outline for extension and adaptation.
Sub-tasks necessary in order to benchmark and develop cost model for Nucleus deployments per number of apps in the pipeline
We need a consistent approach to logging and monitoring baseline EN components used in Airflow, including a clear and succinct error handling guide for users to know how to track down failures in the pipeline.
Sub-tasks:
> aws s3 ls --no-sign-request s3://asc-pds-messenger
PRE MESSDEM_1001/
PRE MSGRMDS_1001/
PRE MSGRMDS_2001/
PRE MSGRMDS_3001/
PRE MSGRMDS_4001/
PRE MSGRMDS_5001/
PRE MSGRMDS_6001/
PRE MSGRMDS_7001/
PRE MSGRMDS_7101/
PRE MSGRMDS_7201/
PRE MSGRMDS_7301/
PRE MSGRMDS_8001/
Publish it in the sphynx documentation.
Initial implementation as POC.
The POC will implement this simple workflow:
The candidate solution tested are:
Need to build out the designs for off-nominal behavior in archive management:
Node Manager
...so that I can adapt or create new workflows in nucleus and know how much it will cost
Given
When I perform
Then I expect
CWL or some other definition standard
Node Operator
...so that I can understand what Nucleus is, the baseline tools and services included in the deployment, and what this is all for.
Given
When I perform
Then I expect
It was decided to consider the Amazon Managed Airflow (Amazon MWAA) for short term for Nucleus and later consider a more flexible approach such and self managed Airflow in future. This task is to prepare the cost model document considering Amazon MWAA.
Data should be loaded into https://search-img-prod-tlnl5qlzgk5iknwhiemnc6aogy.us-west-2.es.amazonaws.com/
Node Operator
...so that I can customize the baseline Nucleus deployment for my needs. e.g. how the data will be delivered, or any particular tool config options
Given
When I perform
Then I expect
The Amazon Managed Workflows for Apache Airflow (MWAA) has some ongoing costs due to the always up and running Airflow UI. There is no option available to suspend the MWAA environment when it is not in use.
This task is to evaluate the self-managed airflow on AWS EKS as an alternative to MWAA to reduce the operating cost of Nucleus.
See sub-tasks for details, but this initial phase of PDS Nucleus will look into the overall design of the pipeline as well as an initial trade study to determine existing technologies to accomplish this task.
Deliverables:
Here are some background docs / diagrams that were the origins of this pipeline idea.
No - I haven't checked
data provider
...so that I can trigger Nucleus with data delivery by a data provider in the cloud
We want to support this use case in a couple ways
Given
When I perform
Then I expect
Tentatively adding this towards initial CSS migration, but may need to be punted depending on how those efforts go.
Node Operator
...so that I can add my own custom data processing tools into the baseline nucleus pipeline
Given
When I perform
Then I expect
Complete POCs and Trade Study
Node Operator
...so that I can use an open workflow definition standard encouraged by NASA to define Nucleus workflows.
Working example of a Nucleus workflow created with CWL
The Amazon Managed Workflows for Apache Airflow (MWAA) has some ongoing costs due to the always up and running Airflow UI. There is no option available to suspend the MWAA environment when it is not in use.
This task is to evaluate options to backup and restore the metadata database and logs of MWAA to reduce the operating cost of Nucleus.
Deliverable: detailed technical architecture diagrams describing underlying technologies and interfaces
Node Manager
...so that I can accurately measure how much it will cost for a baseline Nucleus deployment
Given
When I perform
Then I expect
Node Manager
...so that I can ensure I am within my budget / cost estimate
Given
When I perform
Then I expect
The Amazon Managed Workflows for Apache Airflow (MWAA) has some ongoing costs due to the always up and running Airflow UI. There is no option available to suspend the MWAA environment when it is not in use.
This task is to evaluate the self-managed airflow on AWS EC2 as an alternative to MWAA to reduce the operating cost of Nucleus.
It was decided to evaluate Cumulus as a Candidate for the Nucleus Trade Study.
Cumulus is a native cloud-based data ingest, archive, distribution, and management system used in Earth Observing System Data and Information System (EOSDIS) data streams.
Cumulus documentation is available at: https://nasa.github.io/cumulus/docs/cumulus-docs-readme
The Amazon Managed Workflows for Apache Airflow (MWAA) has some ongoing costs due to the always up and running Airflow UI. There is no option available to suspend the MWAA environment when it is not in use.
This task is to evaluate the self-managed airflow with ECS and Fargate as an alternative to MWAA to reduce the operating cost of Nucleus.
It was identified that race conditions can happen while multiple S3 event notifications invoke lambda to update DynamoDB tables (reference #63 ).
To avoid that, a locking mechanism should be implemented.
Implement queuing service for streaming data into Nucleus based on the following design related inputs:
Currently if we want to develop a new workflow for Nucleus, we have to complete the following steps.
We have to execute above multiple steps because different PDS docker images have different setups. For example, some of the PDS docker images need various environment variables and others need various volumes to be mounted. As a result, we cannot use a common template.
This problem is there regardless of the workflow tool we use (not specific to Airflow). With any workflow tool, we will have to maintain different setups for different docker images (for example, when we used docker compose, we had different settings for each components in docker compose file).
Implementing and uploading the DAG is easy if the PDS components are already available as ECS tasks. If we plan to automatically generate Terraform scripts, then we have to discuss,
A Workflow Manager Utility can be solution to address above issues.
Yes - I've already checked
Node Manager
...so that I can know when our budget is approaching or passing our estimated budget.
Options are EFS, S3, other databases
That could managed as:
A good place to start is to get inputs from PODAAC on how they do that in Cumulus.
Sys Admin
...so that I can automatically deploy a new baseline nucleus instance with minimal manual intervention.
Given
When I perform
Then I expect
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.