Per discussions with team, looking at producing a POC for a few different architectures.
Some options:
Data provider stages data in their own S3 bucket, publish CNM to DAAC SNS topic
Data provider stages data in DAAC S3 bucket, publishes CNM to DAAC SNS topic
Data provider stages data in DAAC S3 bucket, DAAC ingest themselves using some crawler tool one time (for one-off collections)
Data provider stages data in DAAC S3 bucket, DAAC continuously ingest data as it comes in (new mechanism that is not yet completed)
Per discussion with the team, going to pursue an approach similar to what @collinss-jpl has proposed:
The approach I had been thinking about utilizes AWS API gateway connected to a Lambda (similar to TEA) to allow a client application on the SBN host to request an upload/sync of a local file or files. The Lambda uses information from the request to determine where in our S3 bucket hierarchy the requested files should get uploaded to (based on product type, PDS submitter node, or whatever other criteria we derive). The S3 URI(s) are then returned back through the API gateway to the client. The SBN client application then uses the returned URI(s) to perform the sync using the CLI or boto library. Eventually we could work in a job queue on the SBN client app so the uploads can be performed asynchronously from the upload requests. We would also be able to use the built-in throttling capability on API Gateway to control how much data or how many requests weโll allow within a window of time etcโฆ
...so that I can know when a directory has completed reading, and we can fully evaluate all the products (XML + data files) in the directory + sub-directories.
๐ Additional Details
Crawl file system
Each directory you come across, write a dum.lock file with TBD information in it
Continue to crawl and write data, as you complete a directory, and all it's sub-directories, remove the dum.lock file
Develop the necessary documentation, terraform scripts and/or other definitions/scripts needed to deploy the app on a user system and deploying to the cloud.
The authentication token returned from Cognito has a default expiration of 1 hour, which is typically shorter than what is expected for large file transfers. Cognito authentication tokens can be refreshed by providing the "refresh" token that is supplied after initial authentication.
The DUM client script needs to be updated to support an automatic refresh of the authentication token based on when the token is expected to expire. This should allow long running transfers to complete without interruption.
...so that I can overwrite data that has already been loaded into the PDC
๐ Additional Details
No response
Acceptance Criteria
Given a file that has already been loaded into the registry When I perform a DUM upload with the --overwrite flag enabled Then I expect the data to overwrite the existing file in the system, and DUM to note that this occurred in the logs
Currently, execution is taking too long to upload data. We need to figure out ways we can improve throughput. TBD on this task. This may be dependent on system CPUs available.
The current behavior of the pds-ingress-client.py script is to ignore an symbolic links encountered when traversing paths to be uploaded. Going forward, it could be useful to add a command-line option to instruct the client to follow encountered symlinks, rather than ignore them by default.
๐ช Motivation
Would allow pds-ingress-client.py to be used with datasets that are compiled from pre-existing data via symlink to avoid data duplication.
The Ingress Lambda function can log messages directly to AWS CloudWatch via the built-in logging library. This will likely be the primary mechanism for tracking incoming requests, so we need to define exactly what we would like to see logged for each request.
Right now data is being pushed to buckets like for SBN, /SBN/my/data/here. the /SBN seems a bit redundant, but we can leave it for now. However, definitely want this lowercase.
Cost model for data upload manager components. Should go hand-in-hand with Design Doc but this will be managed and maintained in a secure location. This Epic will also include consideration of deployment strategies, as needed.
...so that I can include a checksum with the files being uploaded to ensure data integrity as the files flow through the system
๐ Additional Details
No response
Acceptance Criteria
Given a file to be uploaded to S3 When I perform data upload manager execution on that file Then I expect data upload manager to generate a checksum and add to the object metadata for the S3 object
The current command-line interface for the Ingress client script only allows a user to provide a single file path, as well as an (arbitrary) node ID. This interface needs to be developed to allow at a minimum:
Validation of the provided node ID against the standard set of PDS identifiers
Support for specifying multiple input paths
Support for distinguishing paths to files vs. paths to directories and performing the appropriate S3 sync logic
One potential issue was raised for awareness: currently it seems to take about 5-10 seconds per css product to upload. That's going to need to be improved somewhere in the chain, because at that rate it will take more than 24 hours to upload the number of products that are generated every 24 hours.
๐ช Motivation
...so that I can [why do you want to do this?]
๐ Additional Details
No response
Acceptance Criteria
Given When I perform Then I expect
โ๏ธ Engineering Details
Variables we need to take into account:
Bandwidth to AWS - is there a way we could improve upload throughput?
...so that I can have a summary of how things were uploaded
๐ Additional Details
files uploaded
files skipped
files overwritten
Acceptance Criteria
Given a set of files to be uploaded to S3 When I perform a nominal upload Then I expect a final report to be output show metrics of files read, files successfully uploaded, files skipped, and files overwritten
The current Ingress client script contains a number of hardcoded constants related to AWS configuration (such as API Gateway ID and region) that should be refactored into an external .ini config (or similar) to allow easy customization without requiring code redeployment.
The current prototype of the Data ingress lambda function contains some dummy logic for determining the product type from the provided file path/node ID. This ticket is to track the planning and implementation for the initial logic for determining the s3 path convention from the data payload provided by the client script.
Also within scope for this ticket is defining the input payload schema itself that will determine what is sent by the client.
As a result, a back-end service component is developed for the Data-Upload-Manager. It received the upload request from the client and returns a s3 path (or eventually a presigned S3 URL) where the data should be uploaded by the client.
Current design is to overwrite the data when user upload data via DUM. Propose to add capability to verify if the file modification time and size remain unchanged, thereby allowing the copying to S3 to be skipped. Additionally, providing an optional flag (e.g., --force-overwrite) would enable users to overwrite the file when needed.
Acceptance Criteria
Given When I perform Then I expect
โ๏ธ Engineering Details
Note: Per #91, rclone handles this functionality for us.
The user should also have the ability to force overwrite data that is already out there.
...so that I can add another layer of security to access to S3 buckets
๐ Additional Details
No response
Acceptance Criteria
Given a bucket that I have write access policy to a bucket with data upload manager, and within a set IP subnet When I perform a DUM load Then I expect the data to upload successfully
Given a bucket that I have write access policy to a bucket with data upload manager, and outside the expected IP subnet When I perform a DUM load Then I expect the data to upload successfully
The easiest way to do this would be search the registry either for the file path OR by checksum OR both? We could do this with the LID/LIDVID but I think that will add some significant overhead.
Do we want to figure out some sort of auto-generated UUID for every file we upload to the cloud and add this as metadata? Maybe this is something we could actually store then in the Nucleus database and eventually in the registry. It could link throughout the whole system, agnostic of the LIDVID for the products themselves.
To further secure the data upload process, the ingress service Lambda needs to incorporate generation of pre-signed S3 URL's that the client can use to securely upload files to S3. This will allow all the PDS buckets in Nucleus to be private (so trying to guess an S3 upload URI should not work), while still providing a means for outside users to push to S3 without additional credentials or permissions.
As a workaround, let's plan to comment out the code that is causing this for the time being. Logging into AWS is a lower priority ("should") requirement.