unity-sds / unity-analytics-bcdp Goto Github PK

To test the deployment and execution of your BCDP application package on a Mac laptop, you should be able to:

1. Run Docker Desktop with Kubernetes enabled (launches a 1-node Kubernetes cluster on your laptop that can be managed with kubectl).
2. Install the existing ADES/WPS-T Flask app on your laptop
   • git clone https://github.jpl.nasa.gov/SOAMC/flask_ades_wpst.git
   • cd flask_ades_wpst
   • mkdir sqlite
   • Create an Anaconda or other virtual environment for your ADES and activate it.
   • python setup.py install
3. Deploy ADES/K8S on laptop
   • export ADES_PLATFORM="K8s"
   • python -m flask_ades_wpst.flask_wpst
4. The endpoints we are interested in are getLandingPage, getProcesses, deployProcess, getProcessDescription, undeployProcess, getJobList, getJobStatus, dismissJob, getJobResult as in Table 1 of http://docs.opengeospatial.org/per/18-050r1.html#_wps_t_restjson. Demonstrate WPS-T endpoints with curl or wget
   • Landing page: curl -X POST http://127.0.0.1:5000/
   • Get process list: curl http://127.0.0.1:5000/processes
   • Deploy process: curl -X POST http://127.0.0.1:5000/processes?proc=<raw-url-to-your-app_desc.json>
   • Undeploy process: `curl -X DELETE http://127.0.0.1:5000/processes/
   • Execute job: curl -H "Content-Type: application/json" -X POST -d '{"<your-key1>": "<your-val1>", "<your-key2>": "<your-val2>", ... }' http://127.0.0.1:5000/processes/hello-world-1.0.0/jobs
   • Get job list: curl http://127.0.0.1:5000/processes/<your-proc-id>/jobs
   • Get job status: curl http://127.0.0.1:5000/processes/<your-proc-id>/jobs/<your-job-id>
   • Dismiss job: curl -X DELETE http://127.0.0.1:5000/processes/<your-proc-id>/jobs/<your-job-id>
   • Get job result (when implemented): curl http://127.0.0.1:5000/processes/<your-proc-id>/jobs/<your-job-id>/result

This task is to develop a new BCDP notebook with an extended set of Papermilled parameters. The new parameters should include all the settings necessary to run any combination of one, two, or all of the following functions: reproject, resample, and subset.

Current papermilled notebook uses intake-esm catalog and parameters are for querying the catalog. From the point of view of the SDS we have decided it's preferable for processes outside the app to do a data search and instead the app should explicitly know the URL of the data.

What does this look like- is it a simple workflow that can be used against multiple input products in a deployed ADES? Is it a deployed system that is specific to a project/tenant

Acceptance Criteria

• Architectural diagram of how multi-tennant BCDP will work
• Costing models?

What Else?

Task 3.1; We need to identify what you're expecting from U-CS to make this successful. I'm not sure there will be a singular 'tool kit' for all deployment, some tailoring for SDAP/BCDP are required.

Develop demonstration notebook that deploys, runs, and monitors the new extended BCDP application on ADES-HySDS in U-SPS. The WPS-T interfaces should be used in this demonstration (deploy, execute, getStatus, getResults).

This demonstration notebook should be written in a tutorial format and show how to BCDP can be used in 4 scenarios:

1. Regrid by itself
2. Resample by itself
3. Subset by itself
4. A combination of Regrid, Resample, and Subset as a single job.

Check that BCDP CWL files generated using app-pack-generator runs without errors

Consult with the U-ADS team to understand the Unity capabilities to build OGC applications. A key repo for this is https://github.com/unity-sds/unity-app-generator. Use the Unity tools to build an OGC application package out of the extended BCDP notebook, including workflow CWL, 3-step stage-in, process, and stage-out CWL, and associated Docker images.