Observational Products for End-Users from Remote Sensing Analysis (OPERA)
License: Apache License 2.0
Yes - I've already checked
When I did testing hlsl30_query_timer, hlss30_query_timer and data_subscriber_download_timer, I noticed that data_subscriber_download throws exception as below when hlsl30_query_timer and hlss30_query_timer failed to execute.
Traceback (most recent call last):
File "/home/ops/verdi/ops/hysds-1.1.5/hysds/job_worker.py", line 1237, in run_job
raise RuntimeError("Got non-zero exit code: {}".format(status))
RuntimeError: Got non-zero exit code: 1
I expected the data_subscriber_download_timer to execute without any error and reported out that there is no data_subscriber_catalog ES index exists or no data to download.
No response
- PCM branch: issue_75
- Lambda branch: issue_75Yes - I've already checked
Yes - and alternatives don't suffice
No response
PCM needs to have a smoke test script to exercise R1 capabilities.
Terraform install script should be generalized so that it can perform a cluster provisioning with project specific variables terraform config file.
Update Data Accountability UI
Verify the PCM core capabilities with NISAR PCM R2 deployment to OPERA dev venue.
Use NISAR PCM Operator Guide and Use Cases documents for details.
Write a report with steps and results
Yes - I've already checked
Yes - and alternatives don't suffice
Data download jobs were using 'opera-job_worker-small' queue which is used by PGE jobs. Data download was not occurring while the PGE jobs are executing.
Use different queue for the data download and PGE job execution.
Use data_query_timer_trigger_frequency variable to execute daac_data_subscriber_quey.sh as -m option
OPERA SDS needs to process 125 months historical data within 12 months period.
We need to have a tool to automate the processing of bulk historical data in addition to the forward processing.
PGE to trigger: L2_CSLC_S1 PGE
Data region to cover: North America
Input data: S1 A/B SLC
PCM ER2 contains initial integration of L3_DSWx_HLS_PGE and PCM.
PCM can trigger the L3_DSWx_HLS_PGE after staging HLSS30 or HLSL30 data into ISL bucket. However, PCM handles one .tif as output and didn't test for many outputs.
L3_DSWx_HLS_PGE plans to produce multiple .tif files.
PCM should be able to ingest all outputs and send notification for each product.
We also need to verity all following steps are working properly.
OPERA PCM should be able to query DAAC CMR and retrieve data from S3 or https URLs.
Define a trigger rule to execute L3_DSWx_HLS_PGE
In order to run the PGE with S30, following files are required.
For L30 (Landsat) HLS products, we need layers (6 reflectance bands + Q/A mask): B02, B03, B04, B05, B06, B07, and Fmask
For S30 (Sentinel) HLS products, we need layers (6 reflectance bands + Q/A mask): B02, B03, B04, B8A, B11, B12, and Fmask
Yes - I've already checked
Yes - and alternatives don't suffice
hysds_ui was inaccessible with "500 Internal Server Error"
We want to use the new AMIs (dated April 18) that fix the certs changes. AMIs info is available at https://wiki.jpl.nasa.gov/display/operasds/AMIs.
SDS needs to be able to query various DAACs (ASF DAAC, LP DAAC, or PO DAAC) and retrieve data from them to OPERA ISL location
Following information should be passed to query.
Note: we can use NISAR daac retrieval script as reference.
https://github.jpl.nasa.gov/IEMS-SDS/nisar-pcm/blob/develop/tools/sds_daac_retrieval.py
This script will be used by a PCM worker to automatically query and retrieve data into OPERA ISL S3 bucket.
Yes - I've already checked
Yes - and alternatives don't suffice
No response
Current data-subscriber script queries DAAC CMR and retrieves the data into our S3 bucket within one script.
We should split the querying and retrieving functions in separate scripts so that a querying script can be executed by a factotum worker and a retrieving script can be executed by many verdi workers.
Yes - I've already checked
Yes - and alternatives don't suffice
No response
It would be great to add an ingested time field in the data_subscriber_catalog index and then use the ingested time field to download after sorting by asc. Then, we will be able to get a set of files for each granule (tile based) faster to trigger L3_DSWx_HLS pge.
Write a smoke test script with L3_DSWx_HLS_PGE integration and add to terraform installation.
The data_query_timer queries both HLS S30 and HLS L30 types in a single script and it always ingested HLS L30 types first, so the download timer won't download HLS S30 until the HLS L30 data download is done. The query timer needs to be split to two separate times for S30 and L30 types.
Rename data_subscriber timer and shell script
Add following log messages into the end of query log
Yes - I've already checked
Yes - and alternatives don't suffice
No response
PCM currently generates outputs from L3_DSWx_HLS PGE with product_counter as 1. Therefore, we can't re-submit a PGE job with same inputs unless we clear out L3 outputs. We should be able to increment the product_counter when there are existing L3 outputs.
Yes - I've already checked
Yes - and alternatives don't suffice
No response
The PCM should be able to use the Universal AMIs used by all projects (not specific to OPERA)
Details are available at https://github.jpl.nasa.gov/IEMS-SDS/nisar-pcm/pull/468#issuecomment-457901
There now exists a Jenkins job that determines the latest patched AMIs and runs a smoke test against it:
https://opera-pcm-ci.jpl.nasa.gov/job/opera-pcm-develop-latest-patched-amis/
Yes - I've already checked
Yes - and alternatives don't suffice
No response
We need to generate a CNM-S message with multiple tif files after executing L3_DSWx_HLS PGE.
We also need to be able to set data_version and schema_version from the cnm notify trigger rule.
We need to set different product type for DAAC rather than using ProductType metadata.
Deliver the outputs from OPERA PGEs to appropriate DAAC.
Write recovery procedures when there are system down/failure.
Use https://wiki.jpl.nasa.gov/display/operasds/PCM+Subsystem?preview=/483781733/577281017/NISAR%20PCM%20Disaster%20Recovery%20Guide_R2.docx as reference and modify for OPERA requirements/needs.
Yes - I've already checked
Yes - and alternatives don't suffice
No response
Current DAAC data download timer submits a job using only one worker to download data from DAAC to our ISL bucket. We need to be able to launch many workers to download data simultaneously.
Yes - I've already checked
Yes - and alternatives don't suffice
Repeat of #22 with ASF DAAC data
PCM needs to test how fast we can query and retrieve the data from ASF DAAC for preparation of bulk historical processing.
Modify terraform script to install PCM system on DEV/INT/OPS venues.
Yes - I've already checked
When I tested data-subscriber-timer, I noticed it only querys HLSS30 type.
We should also query HLSL30 type as well.
I also noticed that ISL bucket name is always 'opera-dev-isl-fwd-mplotkin'. ISL bucket name should be entered by the user.
I expected to query DAAC for both HLSS30 and HLSL30 by using a user's ISL bucket.
Log message should include the parameters that was passed to execute the script.
It will be better to combine stdout.txt msgs into the log.
Log filename should contain the datatype and log creation day.
ex) HLSS30_daac_data_subscriber.log
1.- opera-pcm v1.0.0-er.2.0
- HySDS v4.0.1-beta.8 (CentOS 7) AMIsA worker should query and retrieve data from DAACs (LP DAAC, ASF DAAC, or PO DAAC) to OPERA ISL automatically. Duration of query and retrieval should be configurable.
Generates a RunConfig for L3_DSWx_HLS PGE. A sample RunConfig will be provided by PGE team.
Input: L2_HLS (HLSS30 or HLSL30)
Anc: DEM. Global Landcover, Global built up land layer
Should be able to test with mock-up L3_DSWx_HLS PGE.
Test data and sample RunConfig file is located at
https://artifactory-fn.jpl.nasa.gov/artifactory/webapp/#/artifacts/browse/tree/General/general/gov/nasa/jpl/opera/adt/r1/interface/test_datasets.tar.gz
Required files:
Ingest input and output data for L3_DSWx_HLS_PGE. PCM should be able to extract metadata and ingest into PCM catalog (ElasticSearch).
The input dataset is the HLS product version 2.0. HLS products provide surface reflectance (SR) data from the Operational Land Imager (OLI) aboard the Landsat 8 remote sensing satellite and the Multi-Spectral Instrument (MSI) aboard the Sentinel-2 A/B remote sensing satellite.
Input File Naming Convention:
HLS.<product type>.<tile id>.<data acquisition time>.<version>.<band/mask>.tif
product type: S30 or L30
data acquisition time: YYYYDOYTHHMMSS
version: v2.0
band or mask: B01 - B08, B8A, B09, B10, B11, B12, Fmask, SZA, SAA, VZA, VAA
ex) HLS.S30.T17SLU.2020117T160901.v2.0.B01.tif
Output File Naming Convention:
<Project>_<Level>_<ProductType>_<Source>_<Sensor>_<TileID>_<DateTime>_<ProductVersion>.<ext>
If the input is HLS.S30, the Sensor is 'Sentinel2'
If the input is HLS.L30, the Sensor is 'Landsat8'
production date time: YYYYMODDTHHMMSS
ex) OPERA_L3_DSWx_HLS_Sentinel2_T15SXR_20210211T163901_01.tiff
Format: GeoTIFF
Initial design of data accountability page
Setup initial CI/CD for PCM Dev on https://opera-pcm-ci.jpl.nasa.gov/
Yes - I've already checked
No - I haven't considered
I'm frustrated when testing PCM ER2 release. I couldn't find any log for the PGE execution, after the PGE executed successfully and the verdi worker is no longer up and running.
I want to include a PGE log as output dataset and stage into RS location as other output files do.
Yes - I've already checked
Yes - and alternatives don't suffice
No response
We need to store spatial information for HLSL3 and HLSS30 after querying CMR. The spatial information is required to find ancillary and DEM files for triggering L3_DSWx_HLS PGE.
We want to store more metadata into our ES index with the spatial information.
hls_spatial_catalog: granuleid, bbox, shortname, production_datatime, index_create_time
Yes - I've already checked
Yes - and alternatives don't suffice
No response
I noticed that most log message stored in _stderr.txt rather than _stdout.txt. The logging level was DEBUG, how can we change to lower logging level?
Yes - I've already checked
No - I haven't considered
No response
PCM should use Tosca UI to submit a job to query and download products from DAACs by entering geographic information and/or time range.
Following fields are needed to fill
Yes - I've already checked
Traceback (most recent call last):
File "/export/home/hysdsops/verdi/ops/hysds-1.1.5/hysds/job_worker.py", line 1193, in run_job
monitoredRunner.join()
File "/export/home/hysdsops/verdi/lib/python3.9/site-packages/billiard/process.py", line 148, in join
res = self._popen.wait(timeout)
File "/export/home/hysdsops/verdi/lib/python3.9/site-packages/billiard/popen_fork.py", line 57, in wait
return self.poll(os.WNOHANG if timeout == 0.0 else 0)
File "/export/home/hysdsops/verdi/lib/python3.9/site-packages/billiard/popen_fork.py", line 33, in poll
pid, sts = os.waitpid(self.pid, flag)
File "/export/home/hysdsops/verdi/lib/python3.9/site-packages/billiard/pool.py", line 229, in soft_timeout_sighandler
raise SoftTimeLimitExceeded()
billiard.exceptions.SoftTimeLimitExceeded: SoftTimeLimitExceeded()
s3://opera-dev-triage-fwd-hyunlee/triaged_job-data-subscriber-timer-20220308T032000_20220308T033000-20220308T033020.402154Z_task-c768baf6-9194-46a6-aea8-c2cd7900ac8d/
I expected that the data-subscriber continued to query and download HLS data into the ISL bucket.
It should log # of results from the CMR query so we know how many products are available to download. We should download data when it's larger than 2000.
No response
PCM cluster using HySDS v4.0.1-beta.8-oraclelinuxDAAC query and retrieval script should be able to read a file containing list of inputs (URIs) and download the data and stage into the ISL. This capability is for when there is an issue connecting to CMR.
PCM should be able to resubmit jobs that is executed previously.
Add a capability to ingest ancillary/auxiliary products
Download source information is documented in OPERA DSWX HLS SAS Design Doc.
Verify the PCM core capabilities with NISAR PCM R2 deployment to OPERA dev venue.
Use NISAR PCM Operator Guide and Use Cases documents for details.
PCM should be able to download data using S3 URIs into our ISL.
For Release1, we won't have enough time to have direct DAAC S3 -> OPERA SDS S3 setup.
We will proceed to have following data copy setup.
DAAC S3 -> local disk -> OPERA SDS S3
This capability will ensure that PCM can use many verdi workers to retrieve data.
We should only download required bands instead of all available bands and masks.
For L30 (Landsat) HLS products, we need layers (6 reflectance bands + Q/A mask): B02, B03, B04, B05, B06, B07, and Fmask
For S30 (Sentinel) HLS products, we need layers (6 reflectance bands + Q/A mask): B02, B03, B04, B8A, B11, B12, and Fmask
Verify that the PCM can be deployed on Dev and I&T venues.
PCM needs to test how fast we can query and retrieve the data from DAAC (specially from ASF DAAC) for preparation of bulk historical processing.
Generate CNM-S json for L3_DSWx_HLS outputs and notify PO DAAC for data availability
Sample CNM-S json: https://github.com/podaac/cloud-notification-message-schema/blob/master/samples/cumulus_sns_v1.4_notification.json
Schema: https://github.com/podaac/cloud-notification-message-schema
For OPERA:
collection - DSWX_S1
dataversion - 1.0
Create a handler to update delivery status upon receipt of CNM-R from DAAC. For R1, the DAAC is PO DAAC.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.