In case you choose to host your Airflow cluster in a Kubernetes cluster from a cloud provider, here are some templates and useful resources you can use to install Airflow with its official Helm Chart.
Once you have run the terraform apply
command, which will create your
Kubernetes cluster (among other resources) in the cloud, then you are ready
to install Airflow.
Install these commands according to your OS.
The Airflow Helm Chart may use a custom extended
image of your own. You let the chart know which one with the images.airflow.repository
and the images.airflow.tag
fields. These can be set in the values.yaml
file.
Go to Docker section to see how to configure your image.
First, a brief description of the config files:
nfs-server.yaml
hashelm
configuration to create an NFS volume to store Airflow logsvalues.yaml
hashelm
configuration to install Airflow with the Airflow Helm Chart. It contains a list of environments variables that need to be set in the cluster to configure connections, secrets, and so on.airflow_install
installs Airflow by reading values from terraform. Make sure yaml files are in the same location as this script.airflow_remove
removes Airflow and the NFS volume from the cluster.
To run the instalation script make sure to replace the placeholders for vars TERRAFORM_DIR
dir and CREDENTIALS_FILE
with absolute paths.
# To run the installation script...
bash airflow_install.sh
After you run the installation script, you can access the Airflow web server by forwarding port 8080 from Airflow web server pod to your computer.
kubectl port-forward svc/airflow-webserver 8080:8080 --namespace airflow
# exit with ctrl + c
Now you can go to localhost:8080
from your web browser and access with user admin
and password admin
to view and invoke DAGs.