unity-sds / unity-ads-deployment Goto Github PK

Acceptance Criteria

• Creation of an algorithm package from code, configs, etc
• run automated tests agains packaged code
• publish Algorithm Package to the Algorithm Catalog
• Deployment diagram (e.g. what is needed for this to run, is it a per-tenant deployment, or shared across many tenants)
• Deployed to Dev environment
• Deployed to Test Environment

Review and edit Dockstore deployment documentation by going through steps independently

Create Terraform configuration files for deploying Dockstore into MCP on a per tenant basis.

Create a deployment that connects to the L1A PGE entry point. This probably maps to using docker2repo and/or Dockerfiles.

Configure Gitlab to build and deliver to the algorithm catalog an example U-ADS algorithm.

Based on Sounder SIPS L1A and L1B algorithms document connectivity between CWL and PGE

Create Terraform configuration for deploying Jenkins into MCP

Create a Docker container than can run SIPS L1B acceptance tests

Create a placeholder infrastructure placeholder document

OGC CSW Catalog Documentation

Create a deployment that connects to the L1B PGE entry point. This probably maps to using docker2repo and/or Dockerfiles.

Demonstrate usage of Dockstore to store Sounder SIPS L1A and L1B packages.

Manually deploy Jenkins into MCP

Unity Algorithms will generate generic metadata sidecar files for any data product it generates

The sidecar file is used to define required (below) and optional metadata (TBD) for ingest and searchability within the Unity system. Our goal should be to make 'searchable' metadata consistent and simple to fill out by the algorithm developers.

Acceptance Criteria

• Sidecar File has
  • beginningDataTime
  • endingDateTime
  • productionDateTime
  • collectionName (same as product type, e.g. SounderSIPS_L1A) - this needs to "match" what's in the DS catalog - which is a TBD
• format is ... ???
  • Currently a key/value pair in XML for the SSips work; i wonder if that has longevity or not
• L1A/B Algorithms updated to produce these sidecar files.

Software repository containing the scripts and software needed for deployment, consisting but not exclusively of:

• Terraform configuration
• Helm configuration files
• Service specific configuration files

These should be stored in Github

Acceptance Criteria

• Deploys Dockstore to MCP
• Deploys Jupyterlab to MCP
• Deploys Jenkins to MCP
• Able to configure with a tenant tag for multiple independent deployments

Dependencies

• U-ADS Algorithm Catalog
• U-ADS Development Environment
• U-ADS CI/CD

Document Dockstore deployment using Terraform in Infrastructure Deployment Documentation

Create some notes for the SIPS L1A, L1B Docker creation repository for others to use. Test out instructions on a clean system.

Connect Jenkins to U-CS authentication services

Manually deploy Dockstore into the MCP environment to get a feel for deployment.

Go through the steps of deploying Jenkins within the non MCP AWS

Need the algorithm catalog (and dependent Algorithm Packaging formats) to ensure U-SPS knows what information will be included and how to access the algorithm catalog.

Due to the difficulty in getting CMR deployed it was suggested that we evaluate Dockstore (https://dockstore.org/) as an alternative. This open source project looks to have much more alignment with our requirements than CMR.

Connect Dockstore to U-CS authentification services

Manually deploy Jenkins into MCP

Exercise Dockstore to store a dummy algorithm within MCP

Documentation on the APIs used for M2M communication. Points to other documentation such as for OGC CSW and provides ADS specific examples.

Documentation on the APIs used for M2M communication. Points to third party documents with supplemental information.

Acceptance Criteria

• Describes U-ADS APIs meant for external access
• Points to additional third party documentation
• Details Unity specific information
• Provides example usage
• Provides information on software libraries useful for access

Dependencies

• U-ADS Algorithm Catalog
• U-ADS Development Environment
• U-ADS CI/CD

Develop a Dockerfile that compiles the Sounder SIPS PGE software dependencies in a way that mimics the Sounder SIPS computing environment to avoid needing to change their code repositories.

Create a Docker image for the SIPS L1A PGE

The mechanisms and interfaces for a user to begin development work within the Unity System.

Acceptance Criteria

• MCP deployed Jupyter Hub
• User can create a notebook in Jupyter Hub
• no VCS tie-ins required (?)
• Authentication (Work with U-CS) to see how 'simple' this is to set up- they had a version of this.
• Deployment diagram (e.g. what is needed for this to run, is it a per-tenant deployment, or shared across many tenants)
• Deployed in dev environment
• Deployed in 'test' environment for SounderSIPS users

Configure Gitlab to build and package Sounder SIPS algorithms as defined within the user development environment.

Create a placeholder infrastructure placeholder document

U-ADS infrastructure service component that stores and organizes metadata related to applications/algorithms that can be deployed for processing by U-SDS.

Acceptance Criteria

• Deployed in MCP
• Accessible by MCP based Unity Components
• Ability to deploy a packaged algorithm to an algorithm catalog
• Ability to search an algorithm catalog by 'project' (e.g. SounderSIPS)
• Algorithm Catalog can contains link to CWL and Docker container
• Deployment diagram (e.g. what is needed for this to run, is it a per-tenant deployment, or shared across many tenants)

Create a base Dockerfile for compiling Sounder SIPS PGEs, this may end up just becoming the basis for the L1A and L1B PGE Dockerfiles. But should be able to run unit tests and acceptance tests for all PGEs successfully to uncover dependencies and interfaces outside of Docker.

Configure Jenkins using Terraform into MCP with an initial set of U-ADS specific plugins and configurations.

Centralized document that describes how to use the U-ADS Infrastructure Deployment Software

Acceptance Criteria

• Covers all deployable software
• Describes software tools needed
• Describes what steps deployment software takes
• Provides links to useful third party documentation

Dependencies

• U-ADS Infrastructure Deployment Software

A containerized version of the Sounder SIPS L1B algorithm.

Convert existing Sounder SIPS algorithm to work with Unity interface in conjunction with that team from prototype milestones document. This is the L1B processing.

Finished version should be a docker container that can execute. It does not need to be built and registered as part of the algorithm build pipeline.

Ideally this is registered in the Algorithm Catalog

A containerized version of the Sounder SIPS L1A algorithm.

Convert existing Sounder SIPS algorithm to work with Unity interface in conjunction with that team from prototype milestones document. This is the L1A processing.

Finished version should be a docker container that can execute. It does not need to be built and registered as part of the algorithm build pipeline.

Ideally this is registered in the Algorithm Catalog.