Data Upload Manager (DUM) component for managing the interface for data uploads to the Planetary Data Cloud from Data Providers and PDS Nodes.
Home Page: https://nasa-pds.github.io/data-upload-manager
License: Apache License 2.0
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
Per discussions with team, looking at producing a POC for a few different architectures.
Some options:
Per discussion with the team, going to pursue an approach similar to what @collinss-jpl has proposed:
The approach I had been thinking about utilizes AWS API gateway connected to a Lambda (similar to TEA) to allow a client application on the SBN host to request an upload/sync of a local file or files. The Lambda uses information from the request to determine where in our S3 bucket hierarchy the requested files should get uploaded to (based on product type, PDS submitter node, or whatever other criteria we derive). The S3 URI(s) are then returned back through the API gateway to the client. The SBN client application then uses the returned URI(s) to perform the sync using the CLI or boto library. Eventually we could work in a job queue on the SBN client app so the uploads can be performed asynchronously from the upload requests. We would also be able to use the built-in throttling capability on API Gateway to control how much data or how many requests we’ll allow within a window of time etc…
Detailed work plan TBD
No - I haven't checked
No response
...so that I can know when a directory has completed reading, and we can fully evaluate all the products (XML + data files) in the directory + sub-directories.
dum.lock file with TBD information in itdum.lock fileGiven
When I perform
Then I expect
No response
Develop the necessary documentation, terraform scripts and/or other definitions/scripts needed to deploy the app on a user system and deploying to the cloud.
The authentication token returned from Cognito has a default expiration of 1 hour, which is typically shorter than what is expected for large file transfers. Cognito authentication tokens can be refreshed by providing the "refresh" token that is supplied after initial authentication.
The DUM client script needs to be updated to support an automatic refresh of the authentication token based on when the token is expected to expire. This should allow long running transfers to complete without interruption.
No response
Yes - I've already checked
Node Operator
...so that I can overwrite data that has already been loaded into the PDC
No response
Given a file that has already been loaded into the registry
When I perform a DUM upload with the --overwrite flag enabled
Then I expect the data to overwrite the existing file in the system, and DUM to note that this occurred in the logs
No response
Ticket to develop the logging capabilities of the pds-ingress-client.py script. Added capabilities should include:
To help to support the discipline nodes.
Yes - I've already checked
The current behavior of the pds-ingress-client.py script is to ignore an symbolic links encountered when traversing paths to be uploaded. Going forward, it could be useful to add a command-line option to instruct the client to follow encountered symlinks, rather than ignore them by default.
Would allow pds-ingress-client.py to be used with datasets that are compiled from pre-existing data via symlink to avoid data duplication.
No response
Given
When I perform
Then I expect
No response
Motivation
So that the PDS users have a single login and password for all the PDS services.
The Ingress Lambda function can log messages directly to AWS CloudWatch via the built-in logging library. This will likely be the primary mechanism for tracking incoming requests, so we need to define exactly what we would like to see logged for each request.
Yes - I've already checked
No response
...so that I can [why do you want to do this?]
No response
Given
When I perform
Then I expect
No response
Let's always direct users to setup a virtual environment by default
No response
SBN to sbn (created sbn folder, move gbo... to sbn folder.No response
Doing some rapid prototyping to determine the best approach and develop and architecture. See sub-tasks for more details
Because of bug #75 the centralized logging need to be disabled until the bug is fixed.
No response
Right now data is being pushed to buckets like for SBN, /SBN/my/data/here. the /SBN seems a bit redundant, but we can leave it for now. However, definitely want this lowercase.
No response
Cost model for data upload manager components. Should go hand-in-hand with Design Doc but this will be managed and maintained in a secure location. This Epic will also include consideration of deployment strategies, as needed.
After initial rapid prototyping has completed, develop a design and architecture diagram/document.
Ideally this document would be posted as part of the online documentation for this repository.
No - I haven't checked
Archivist
...so that I can include a checksum with the files being uploaded to ensure data integrity as the files flow through the system
No response
Given a file to be uploaded to S3
When I perform data upload manager execution on that file
Then I expect data upload manager to generate a checksum and add to the object metadata for the S3 object
https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingMetadata.html
Yes - I've already checked
Node Operator
...so that I can upload only data I do not already have in Planetary Data Cloud.
No response
Given
When I perform
Then I expect
No response
The current command-line interface for the Ingress client script only allows a user to provide a single file path, as well as an (arbitrary) node ID. This interface needs to be developed to allow at a minimum:
No - I haven't checked
No response
...so that I can [why do you want to do this?]
No response
Given
When I perform
Then I expect
No response
No - I haven't checked
One potential issue was raised for awareness: currently it seems to take about 5-10 seconds per css product to upload. That's going to need to be improved somewhere in the chain, because at that rate it will take more than 24 hours to upload the number of products that are generated every 24 hours.
...so that I can [why do you want to do this?]
No response
Given
When I perform
Then I expect
Variables we need to take into account:
Ticket to add Sphinx documentation for the entire DUM repository. Topics covered should include:
Ticket to track integration of Cognito authentication with API gateway via Lambda authorizer function.
The following links were provided as a starting point to investigate this feature:
https://github.com/unity-sds/unity-cs/wiki/Getting-Cognito-JWT-Tokens-in-Command-Line
https://github.com/unity-sds/unity-cs-security/tree/main/code_samples
/pds/nucleus/dum/client-log-groupNo response
Currently it is stored on S3 for production.
No response
No - I haven't checked
When testing upload of CSS sample data to DUM, the following warning is generated on every ingest:
`WARNING:root:Unable to submit to CloudWatch Logs, reason: 'LogRecord' object has no attribute 'message'
The LogRecords should still have a message field assigned, and all logs should be uploaded to CloudWatch without issue.
No response
No response
🦄 #xyz
No response
Yes - I've already checked
Node Operator
...so that I can have a summary of how things were uploaded
Given a set of files to be uploaded to S3
When I perform a nominal upload
Then I expect a final report to be output show metrics of files read, files successfully uploaded, files skipped, and files overwritten
No response
Is this useful? Should we consider a refactoring to simply wrap this utility?
No response
Yes - I've already checked
Archivist
...so that I can match the modification datetime from the source system where the data is being copied.
No response
Given
When I perform
Then I expect
No response
The current Ingress client script contains a number of hardcoded constants related to AWS configuration (such as API Gateway ID and region) that should be refactored into an external .ini config (or similar) to allow easy customization without requiring code redeployment.
The current prototype of the Data ingress lambda function contains some dummy logic for determining the product type from the provided file path/node ID. This ticket is to track the planning and implementation for the initial logic for determining the s3 path convention from the data payload provided by the client script.
Also within scope for this ticket is defining the input payload schema itself that will determine what is sent by the client.
As a result, a back-end service component is developed for the Data-Upload-Manager. It received the upload request from the client and returns a s3 path (or eventually a presigned S3 URL) where the data should be uploaded by the client.
No - I haven't checked
Node Operator
...so that I can avoid duplicate copies for data.
Current design is to overwrite the data when user upload data via DUM. Propose to add capability to verify if the file modification time and size remain unchanged, thereby allowing the copying to S3 to be skipped. Additionally, providing an optional flag (e.g., --force-overwrite) would enable users to overwrite the file when needed.
Given
When I perform
Then I expect
Note: Per #91, rclone handles this functionality for us.
The user should also have the ability to force overwrite data that is already out there.
Yes - I've already checked
Cloud Admin / Operator
...so that I can add another layer of security to access to S3 buckets
No response
Given a bucket that I have write access policy to a bucket with data upload manager, and within a set IP subnet
When I perform a DUM load
Then I expect the data to upload successfully
Given a bucket that I have write access policy to a bucket with data upload manager, and outside the expected IP subnet
When I perform a DUM load
Then I expect the data to upload successfully
No response
Yes - I've already checked
Node Operator
...so that I do not try to reload the data
No response
Given
When I perform
Then I expect
The easiest way to do this would be search the registry either for the file path OR by checksum OR both? We could do this with the LID/LIDVID but I think that will add some significant overhead.
Do we want to figure out some sort of auto-generated UUID for every file we upload to the cloud and add this as metadata? Maybe this is something we could actually store then in the Nucleus database and eventually in the registry. It could link throughout the whole system, agnostic of the LIDVID for the products themselves.
Epic to investigate/implement support for client authentication when accessing the API gateway endpoint to the Ingress Lambda service.
To further secure the data upload process, the ingress service Lambda needs to incorporate generation of pre-signed S3 URL's that the client can use to securely upload files to S3. This will allow all the PDS buckets in Nucleus to be private (so trying to guess an S3 upload URI should not work), while still providing a means for outside users to push to S3 without additional credentials or permissions.
No - I haven't checked
When the css data was uploaded to cloud via DUM client, the following error occured:
WARNING:root:Unable to submit to CloudWatch Logs, reason: Failed to create CloudWatch Log Stream pds-ingress-client-sbn-1709312322, reason: 403 Client Error: Forbidden for url: https://yofdsuex7g.execute-api.us-west-2.amazonaws.com/prod/createstream
NO errors. And logs get pushed to the cloud
Run DUM on any data and push data to the cloud.
Linux OS
0.3.0
Any PDS4 data
As a workaround, let's plan to comment out the code that is causing this for the time being. Logging into AWS is a lower priority ("should") requirement.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.