AWS Challenge by A Cloud Guru

Cloud Guru Challenge

Automate an ETL processing pipeline for COVID-19 data using Python and AWS Cloud Services

The following image displays the entire architecture developed using THE SERVERLESS FRAMEWORK

The following tech stacks were used:

1. Python
2. AWS
3. The Serverless Framework
4. Github

The following AWS Services are used for this project:

1. S3 Bucket
2. Glue Crawler
3. Glue Database
4. IAM Roles
5. DynamoDB Table
6. Cloudwatch Event Rule
7. Lambda Function
8. Cloudwatch Logs
9. SNS Subscription
10. SNS Topic

Used Python's built-in module, Unittest, to perform some simple tests, to ensure that the code can handle unexpected situations and subsitute invalid data.

Depending on the result of the process, a notification will be sent out to all the subscribers through AWS SNS

Dashboard developed using Quicksight and Data retrived from Athena

https://mohdsaquib.hashnode.dev/event-driven-covid-19-data-etl-process-acloudguruchallenge

Feel free to email me at [email protected]