Topic: airflow Goto Github

👇 Here are 2061 public repositories matching this topic...

abhishek-ch / around-dataengineering
airflow,A Data Engineering & Machine Learning Knowledge Hub
User: abhishek-ch
data-engineering machine-learning airflow datascience spark mlops devops infrastructure

airflow-helm / charts
airflow,The User-Community Airflow Helm Chart is the standard way to deploy Apache Airflow on Kubernetes with Helm. Originally created in 2017, it has since helped thousands of companies create production-ready deployments of Airflow on Kubernetes.
Organization: airflow-helm
Home Page: https://github.com/airflow-helm/charts/tree/main/charts/airflow
helm airflow kubernetes helm-charts helm-chart charts chart k8s

airflow-plugins / example-airflow-dags
airflow,Example DAGs using hooks and operators from Airflow Plugins
Organization: airflow-plugins
apache-airflow dag etl airflow-plugins selenium google-analytics facebook-ads sftp marketo salesforce

ajbosco / dag-factory
airflow,Dynamically generate Apache Airflow DAGs from YAML configuration files
User: ajbosco
airflow apache-airflow python dags

alanchn31 / data-engineering-projects
airflow,Personal Data Engineering Projects
User: alanchn31
data-lake ingest-data data-engineering data-warehouse cassandra aws-redshift mongodb scrapy spark airflow

andreax79 / airflow-code-editor
airflow,A plugin for Apache Airflow that allows you to edit DAGs in browser
User: andreax79
apache-airflow airflow-plugin airflow python

andresionek91 / airflow-autoscaling-ecs
airflow,Airflow Deployment on AWS ECS Fargate Using Cloudformation
User: andresionek91
data-engineering airflow airflow-autoscaling-ecs airflow-deployment airflow-ecs

ankurchavda / streamify
airflow,A data engineering project with Kafka, Spark Streaming, dbt, Docker, Airflow, Terraform, GCP and much more!
User: ankurchavda
airflow kafka data-engineering python dbt spark gcp

apache / airflow
airflow,Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
Organization: apache
Home Page: https://airflow.apache.org/
airflow apache apache-airflow python scheduler workflow automation dag data-engineering data-integration data-orchestrator data-pipelines data-science elt etl machine-learning mlops orchestration workflow-engine workflow-orchestration

apache / airflow-client-python
airflow,Apache Airflow - OpenApi Client for Python
Organization: apache
Home Page: https://airflow.apache.org/
airflow apache apache-airflow apache-airflow-client python

apache / dolphinscheduler
airflow,Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code
Organization: apache
Home Page: https://dolphinscheduler.apache.org/
workflow-schedule azkaban airflow task-scheduler job-scheduler cloud-native data-pipelines orchestration workflow workflow-orchestration

apachecn / airflow-doc-zh
airflow,:book: [译] Airflow 中文文档
Organization: apachecn
Home Page: https://airflow.apachecn.org
airflow

argoproj / argo-workflows
airflow,Workflow Engine for Kubernetes
Organization: argoproj
Home Page: https://argo-workflows.readthedocs.io/
workflow kubernetes argo dag knative airflow machine-learning argo-workflows workflow-engine hacktoberfest

astronomer / airflow-chart
airflow,A Helm chart to install Apache Airflow on Kubernetes
Organization: astronomer
helm-chart airflow apache-airflow kubernetes

astronomer / astro-sdk
airflow,Astro SDK allows rapid and clean development of {Extract, Load, Transform} workflows using Python and SQL, powered by Apache Airflow.
Organization: astronomer
Home Page: https://astro-sdk-python.rtfd.io/
python sql pandas airflow sqlite bigquery postgres snowflake gcs s3

astronomer / astronomer-cosmos
airflow,Run your dbt Core projects as Apache Airflow DAGs and Task Groups with a few lines of code
Organization: astronomer
Home Page: https://astronomer.github.io/astronomer-cosmos/
airflow airflow-operators apache-airflow dbt python workflow

chandulal / airflow-testing
airflow,Airflow Unit Tests and Integration Tests
User: chandulal
airflow airflow-testing testing airflow-dags

coder2j / airflow-docker
airflow,Source code of the Apache Airflow Tutorial for Beginners on YouTube Channel Coder2j (https://www.youtube.com/c/coder2j)
User: coder2j
airflow airflow-docker airflowtutorial docker docker-compose tutorial

cubefs / compass
airflow,Compass is a task diagnosis platform for bigdata
Organization: cubefs
bigdata spark hadoop flink diagnose mapreduce scheduler sql airflow dolphinscheduler

damklis / dataengineeringproject
airflow,Example end to end data engineering project.
User: damklis
big-data scraping mongodb elasticsearch data-engineering kafka kafka-connect debezium django-rest-framework redis

dataplane-app / dataplane
airflow,Dataplane is an Airflow inspired unified data platform with additional data mesh and RPA capability to automate, schedule and design data pipelines and workflows. Dataplane is written in Golang with a React front end.
Organization: dataplane-app
Home Page: https://dataplane.app
golang airflow kubernetes data scheduler workflows workflow-automation data-pipelines data-science data-engineering

elyra-ai / elyra
airflow,Elyra extends JupyterLab with an AI centric approach.
Organization: elyra-ai
Home Page: https://elyra.readthedocs.io/en/stable/
notebooks notebook-jupyter jupyterlab jupyterlab-extensions ai machine-learning pipelines kubeflow-pipelines python elyra

epoch8 / airflow-exporter
airflow,Airflow plugin to export dag and task based metrics to Prometheus.
Organization: epoch8
airflow airflow-exporter prometheus

feluelle / airflow-diagrams
airflow,Auto-generated Diagrams from Airflow DAGs. 🔮 🪄
User: feluelle
airflow diagrams auto-generated python cli

fieldryand / goflow
airflow,Simple but powerful DAG scheduler and dashboard
User: fieldryand
directed-acyclic-graph airflow dashboard go job-scheduler schedule workflow-engine

gocardless / airflow-dbt
airflow,Apache Airflow integration for dbt
Organization: gocardless
Home Page: https://pypi.org/project/airflow-dbt/
dbt airflow-dbt airflow

godatadriven / whirl
airflow,Fast iterative local development and testing of Apache Airflow workflows
Organization: godatadriven
airflow docker local-development testing

googlecloudplatform / airflow-operator
airflow,Kubernetes custom controller and CRDs to managing Airflow
Organization: googlecloudplatform
kubernetes kubernetes-operator apache-airflow airflow crd kubernetes-controller workflow-engine airflow-operator

hgrif / airflow-tutorial
airflow,Airflow basics tutorial
User: hgrif
airflow python tutorial

huseinzol05 / gather-deployment
airflow,Gathers Python deployment, infrastructure and practices.
User: huseinzol05
airflow docker docker-compose kafka pyflink pyspark python tensorflow

iusztinpaul / energy-forecasting
airflow,🌀 𝗧𝗵𝗲 𝗙𝘂𝗹𝗹 𝗦𝘁𝗮𝗰𝗸 𝟳-𝗦𝘁𝗲𝗽𝘀 𝗠𝗟𝗢𝗽𝘀 𝗙𝗿𝗮𝗺𝗲𝘄𝗼𝗿𝗸 | 𝗟𝗲𝗮𝗿𝗻 𝗠𝗟𝗘 & 𝗠𝗟𝗢𝗽𝘀 for free by designing, building and deploying an end-to-end ML batch system ~ 𝘴𝘰𝘶𝘳𝘤𝘦 𝘤𝘰𝘥𝘦 + 2.5 𝘩𝘰𝘶𝘳𝘴 𝘰𝘧 𝘳𝘦𝘢𝘥𝘪𝘯𝘨 & 𝘷𝘪𝘥𝘦𝘰 𝘮𝘢𝘵𝘦𝘳𝘪𝘢𝘭𝘴
User: iusztinpaul
Home Page: https://www.pauliusztin.me/courses/the-full-stack-7-steps-mlops-framework
3-pipeline-design airflow batch-processing cicd data-versioning docker fastapi feature-store gcp github-actions great-expectations hopsworks ml-monitoring mlops model-registry poetry python sktime streamlit weights-and-biases

jghoman / awesome-apache-airflow
airflow,Curated list of resources about Apache Airflow
User: jghoman
airflow apache-airflow workflow-management awesome awesome-list

jgoerner / data-science-stack-cookiecutter
airflow,🐳📊🤓Cookiecutter template to launch an awesome dockerized Data Science toolstack (incl. Jupyster, Superset, Postgres, Minio, AirFlow & API Star)
User: jgoerner
data-science docker python jupyter superset airflow apistar postgres minio docker-image

josephmachado / beginner_de_project
airflow,Beginner data engineering project - batch edition
User: josephmachado
Home Page: https://www.startdataengineering.com/post/data-engineering-project-for-beginners-batch-edition/
airflow database docker emr engineering etl python redshift redshift-cluster spark

obenner / data-engineering-interview-questions
airflow,More than 2000+ Data engineer interview questions.
User: obenner
data-engineering interview-questions interview hadoop hadoop-hdfs spark flink sql kafka hive

orchest / orchest
airflow,Build data pipelines, the easy way 🛠️
Organization: orchest
Home Page: https://orchest.readthedocs.io/en/stable/
data-science machine-learning pipelines ide jupyter cloud self-hosted jupyterlab notebooks docker

pipeline-tools / gusty
airflow,Making DAG construction easier
Organization: pipeline-tools
Home Page: https://pipeline-tools.github.io/gusty-docs/
airflow data-pipeline data-etl

pipelineai / pipeline
airflow,PipelineAI
Organization: pipelineai
Home Page: https://generativeaionaws.com
machine-learning artificial-intelligence tensorflow kubernetes cassandra spark kafka airflow docker redis

puckel / docker-airflow
airflow,Docker Apache Airflow
User: puckel
docker-airflow airflow docker scheduler workflow task management

pytorch / torchx
airflow,TorchX is a universal job launcher for PyTorch applications. TorchX is designed to have fast iteration time for training/research and support for E2E production ML pipelines when you're ready.
Organization: pytorch
Home Page: https://pytorch.org/torchx
pytorch machine-learning kubernetes slurm distributed-training pipelines components deep-learning python aws-batch

raystack / optimus
airflow,Optimus is an easy-to-use, reliable, and performant workflow orchestrator for data transformation, data modeling, pipelines, and data quality management.
Organization: raystack
Home Page: https://raystack.github.io/optimus
airflow etl workflows automation golang bigquery data-warehouse analytics data-modelling analytics-engineering

rjurney / agile_data_code_2
airflow,Code for Agile Data Science 2.0, O'Reilly 2017, Second Edition
User: rjurney
Home Page: http://bit.ly/agile_data_science
data-syndrome data data-science analytics apache-spark apache-kafka kafka spark predictive-analytics machine-learning

san089 / goodreads_etl_pipeline
airflow,An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.
User: san089
etl-pipeline etl-framework spark apache-spark apache-airflow airflow redshift emr-cluster livy s3

san089 / udacity-data-engineering-projects
airflow,Few projects related to Data Engineering including Data Modeling, Infrastructure setup on cloud, Data Warehousing and Data Lake development.
User: san089
data data-engineering data-engineering-pipeline etl-pipeline cassandra-database postgresql-database data-modeling data-warehouse data-lake airflow

teamclairvoyant / airflow-maintenance-dags
airflow,A series of DAGs/Workflows to help maintain the operation of Airflow
Organization: teamclairvoyant
airflow dag workflow airflow-maintenance-dags maintenance cleanup apache-airflow

teamclairvoyant / airflow-rest-api-plugin
airflow,A plugin for Apache Airflow that exposes rest end points for the Command Line Interfaces
Organization: teamclairvoyant
airflow rest-api plugin apache-airflow airflow-webserver airflow-plugin

teamclairvoyant / airflow-scheduler-failover-controller
airflow,A process that runs in unison with Apache Airflow to control the Scheduler process to ensure High Availability
Organization: teamclairvoyant
airflow scheduler-failover-controller apache-airflow airflow-scheduler airflow-cluster high-availability

timkpaine / paperboy
airflow,A web frontend for scheduling Jupyter notebook reports
User: timkpaine
jupyter jupyterlab jupyter-notebooks papermill nteract airflow apache-airflow phosphorjs dask scheduling-notebooks

villasv / aws-airflow-stack
airflow,Turbine: the bare metals that gets you Airflow
User: villasv
Home Page: https://victor.villas/aws-airflow-stack/
airflow airflow-cluster airflow-cookbook aws aws-cloudformation

webankfintech / dataspherestudio
airflow,DataSphereStudio is a one stop data application development& management portal, covering scenarios including data exchange, desensitization/cleansing, analysis/mining, quality measurement, visualization, and task scheduling.
Organization: webankfintech
Home Page: https://github.com/WeBankFinTech/DataSphereStudio-Doc
workflow governance azkaban davinci linkis spark hive hadoop visualis zeppelin