1. Requester Information:
This should include the name and contact information of the person making the request.
Name: Jeff Horsburgh
Institution: Utah State University
Position: Associate Professor
Email: [email protected]
Details: We are requesting that the following individuals be given access to USU's existing subaccount:
Jeff Horsburgh: [email protected] (project Co-PI)
Sierra Young: [email protected] (project PI)
Sajan Neupane: [email protected] (project MS student)
Razin Issa: [email protected] (project MS student)
Safran Khan: [email protected] (project PhD student)
2. Project Information:
Provide a brief description of the project and its goals. This can help the infrastructure team understand the context and purpose of the requested resources. Please highlight how this project will be benefit from and/or provide benefit to other resources on the shared infrastructure.
Project Title: CIROH: Advancing Camera-Based Monitoring for Operational Hydrologic Applications
Project Summary: https://ciroh.ua.edu/research-projects/advancing-camera-based-monitoring-for-operational-hydrologic-applications/
Project Goals: The overall goal of this project is to demonstrate how operational requirements for integrating low-cost cameras and computing infrastructure into existing hydrologic monitoring networks can be met, along with evaluating the benefits of cameras for continuous monitoring and prediction.
Project Benefits: We seek to develop serverless cloud workflows/pipelines for processing imagery collected at stream gaging sites. Information extracted from images will provide realtime, hydrologically relevant information - e.g., stream width, depth, velocity to augment available data from realtime USGS streamflow gages and will make more data and information available at gage locations for use in NextGen modeling efforts.
3. Project Description:
If your project involves developing software or scripts, briefly describe the software you plan to develop.
We plan to develop serverless cloud workflows/pipelines for processing imagery collected at stream gaging sites. Code will generally be written using Python, but will involve a variety of scripts designed to execute different parts of the image processing workflow (e.g., image capture on a datalogger, transfer to cloud storage, image preprocessing, model execution, results generation, and logging hydrologic variables to a data store).
4. Resource Requirements:
Specify the compute, storage, and network resources needed for the project. Be as specific as possible about the number of resources required, and any specific configurations or capabilities needed. This information will help the infrastructure team determine the appropriate resources to allocate.
Options:
- Cloud Provider:
For now we are testing AWS. We will eventually want to compare results across AWS and GCP to examine potential tradeoffs. We can make a separate request for access to GCP when you have that better set up.
- Required Services in the Cloud:
List of AWS Services we anticipate using
- Amazon S3
- Each image we capture is about 2 MB for now based on current image resolution
- May experiment with different image resolutions
- Volume Estimate: 100 MB of image data per day per site. We have to sites installed already and plan to install 1-2 more
- AWS Lambda
- Using AWS Lambda functions to do our image processing
- Invoked to process each image, but could do bursts of images (multiple images processed at a time to reduce invocations)
- Using docker containers to create lambda functions
- AWS RDS
- Used for recording data values extracted from images like water depth and stream width
- Likely using some flavor of PostgreSQL
- AWS ECR (Elastic Container Registry)
- May require up to 3 GB
- Using this because data transfer between ECR and lambda functions is faster
- Using this to host the container images that define lambda functions
- AWS CloudWatch
- This is used for logging of activities
- Will use this a lot in the beginning while testing, but probably used less eventually
- AWS Sagemaker
- This would be used for training deep learning models for image segmentation
5. Timeline:
Indicate the expected timeline for the project and when the resources will be needed. This information can help the infrastructure team plan and allocate resources accordingly.
We have already done quite a bit of prototyping using AWS on a USU account. We are ready to move parts of our workflow to the UA account and will continue our prototyping there. We anticipate using resources through the end of our project, which is a 3-year CIROH project funded starting June 1, 2023.
6. Security and Compliance Requirements:
If there are any specific security or compliance requirements for the project, these should be clearly stated in the request. This will help ensure that the necessary security measures are in place for the project.
We don't currently have any specific security or compliance requirements for our development and testing. The image data we are collecting are not sensitive and have no restrictions on access or release. All of our source code and development is open, so no concerns there.
7. Estimation:
Include any cost estimation or requirements for the project. This will help the infrastructure team select the most cost-effective solutions for the project.
We anticipate that the image storage will be relatively inexpensive in S3, and the rate of image data collected is not overly frequent (e.g., one new image per site every 15-30 minutes). We will incur additional costs with executing lambda functions and storage of the container images that define our lambda functions. We want to try doing image segmentation model training using Sagemaker, but that will be a periodic thing that we won't be doing all of the time. We anticipate that overall costs should be less than $100 - $150 per month (given that my whole AWS bill right now for everything that we are doing at USU across our projects is less than that).
8. Approval:
Indicate the necessary approval processes or sign-offs required for the request.
Please contact me with any questions.
Jeff Horsburgh: [email protected]