mwalpole / baywheels-py-demo Goto Github PK

Load the data that has been cleaned into a local Postgres database (preferably containerized) for later querying. Show the code / method used to do this. (A, B, C, D)

Design 1 or more database tables to store the data downloaded in Task 1. Show the DDL, including indexes. (A, B, C, D)

Data cleansing: identify data quality issues in the data and make a plan (or code) to delete the incomplete/inconsistent data (see task 3/4) (A, B, C, D)

Write Python code to fetch the data for the latest month and store it on your local disk and uncompress it. (A, C, D)

Design and implement an ETL process that watches the source website for new data and automatically downloads, cleans and loads the new data into the Postgres database. Show how you handle errors (e.g., connection drops while downloading the data, download is corrupted) and any re-try logic. (A, C)