Spontaneous hackathon for our team :)
https://airflow.apache.org/docs/apache-airflow/stable/start/local.html
AIRFLOW__CORE__DAGS_FOLDER=<dags_folder>
AIRFLOW__CORE__LOAD_EXAMPLES=False
https://airflow.apache.org/docs/apache-airflow/stable/concepts/dags.html
https://airflow.apache.org/docs/apache-airflow/stable/concepts/tasks.html
https://airflow.apache.org/docs/apache-airflow/stable/concepts/operators.html
https://airflow.apache.org/docs/apache-airflow/stable/concepts/sensors.html
https://airflow.apache.org/docs/apache-airflow/stable/concepts/deferring.html
https://airflow.apache.org/docs/apache-airflow/stable/concepts/connections.html
https://airflow.apache.org/docs/apache-airflow/stable/concepts/xcoms.html
https://airflow.apache.org/docs/apache-airflow/stable/concepts/variables.html
https://github.com/airflow-plugins/Example-Airflow-DAGs
In this exercise we will create a basic ETL process
- Create a DAG which schedules every 10 seconds
- Task1: Waits for a new file to arrive in input folder
- Task2: Move all files from input folder to output folder
- Task3: Print success message in Bash using "echo"
In this exercise we will inspect the Dungeons and Dragons REST API.
https://open5e.com/api-docs
https://api.open5e.com/
https://sqliteonline.com/
-
Create a DAG which schedules every 10 seconds
-
Task1: Initialize a new sqlite DB table called Monster that looks like the following example (Without any data):
name strength dexterity constitution intelligence wisdom charisma Aatxe 22 12 20 10 14 14 -
Task2: Waits for D&D API to be available
-
Task3: Request 3 monster details from the D&D API and send it to the next task
-
Task4: Get the monster details from the previous task, then extract only the necessary details about the monsters and save all details into a csv file.
-
Task5: Open the csv file from the previous task and write the monster details into the SQLITE table that you've created in step 2.