jghoman / awesome-apache-airflow Goto Github PK

Hi,

I think that the description about Astronomer Open Edition github in Airflow deployment solutions section is wrong.

It's not really open, they provide it under Astronomer Platform Enterprise Edition license..

Hi all,
was wondering whether it would be considered appropriate to add a company that provides operations, support, and development services for (mostly on-premise) Airflow systems in the "Commercial Airflow-as-a-service providers" category? Looking at the project ecosystem page and this awesome list, there seems to be no other category that fits this kind of service. Yet, I think this is information that people are looking for (as I certainly have been in the past for Airflow as well as various other open source projects).

Any thoughts on this? Did I maybe miss a place where this kind of information would fit better? Or just create a PR and see what happens in review?

Best,
Markus

Airflow-dvc provides a plugin to integrate DVC tool into Airflow systems. DVC is a version-control system that allows versioning of ML artifacts, data-science intermediate datasets and other large-volume files. It's the alternative for git lfs that has better support for pipelines and results reproduction.

Link to the PR

I created an example project demonstrating deployment to Elastic Container Service (ECS). I would like to include a link to it in this repository if you find it meets the criteria. It covers various use cases:

• autoscale workers to zero
• route logs to CloudWatch or Kinesis Firehose using fluentbit
• use remote_logging to put/get task logs to S3
• use the AWS provider SecretsManagerBackend to store/consume sensitive configuration options in SecretsManager
• run a single command as standalone ECS task (eg. airflow db init)
• get a shell into a running container via ECS exec
• send Airflow statsd metrics to CloudWatch

https://github.com/hankehly/deploy-airflow-on-ecs-fargate

Very recently docker based airflow container code has been published by me on GitHub repo. Docker airflow container has many features, some of them are listed below -

• Support 1.10.0 and 1.9.0 version of airflow
• Standalone or cluster mode deployment of the container.
• Standalone mode is for exploration and learning purpose based on Sequential scheduler & Sqllite database.
  *Cluster mode is for production & long run use case and includes two modes of deployment 1) As a server & 2) As a worker
• Supports log writes on AWS S3.
• Container also consists support for GCP.
  There are many other silent features. Please include in your curated list.

Link of the Github Repo - https://github.com/abhioncbr/docker-airflow

The apache wiki has a page for meetups https://cwiki.apache.org/confluence/display/AIRFLOW/Meetups

Maybe either link to that or copy the missing meetups?