Topic: data-engineering-pipeline Goto Github

👇 Here are 123 public repositories matching this topic...

ahmetfurkandemir / data-engineering-project-with-hdfs-and-kafka
data-engineering-pipeline,Data Engineering Project with Hadoop HDFS and Kafka
User: ahmetfurkandemir
data data-engineer data-engineering data-engineering-pipeline docker docker-compose hadoop hadoop-filesystem hadoop-hdfs hdfs

alanchn31 / movalytics-data-warehouse
data-engineering-pipeline, Data pipeline performing ETL to AWS Redshift using Spark, orchestrated with Apache Airflow
User: alanchn31
docker airflow spark sql python3 redshift data-engineering-pipeline data-engineer-nanodegree pyspark data-engineering

alero-awani / batch-data-engineering-project
data-engineering-pipeline,A batch Data Pipeline that retrieves data from a user purchase table and a movie review table and is transformed to form a user behaviour metric table.
User: alero-awani
airflow aws-s3 data-engineering-pipeline docker pipeline pyspark sql terraform

alphanaksoyoglu / tweeter-etl-pipeline
data-engineering-pipeline,A streaming ETL pipeline for Realtime Tweet Collection, Analysis and Reporting
User: alphanaksoyoglu
data-engineering-pipeline data-engineering tweeter tweeter-stream-api tweepy-library sentiment-analysis slackbot mongodb-database postgresql

anki-code / xontrib-pipeliner
data-engineering-pipeline,Let your pipe lines flow thru the Python code in xonsh.
User: anki-code
xonsh xontrib pipe pipeline pipelines python shell data-engineering data-engineering-pipeline

anna-geller / dataflow-ops
data-engineering-pipeline,Project demonstrating how to automate Prefect 2.0 deployments to AWS ECS Fargate
User: anna-geller
automation aws dataflow orchestration prefect serverless analytics analytics-engineering cicd data

anna-geller / dataflow-ops-aws-eks
data-engineering-pipeline,Project demonstrating how to automate Prefect 2.0 deployments to AWS EKS
User: anna-geller
Home Page: https://discourse.prefect.io/t/how-to-build-a-prefect-2-0-poc-on-aws/1066
automation aws data eks eks-cluster eksctl karpenter kubernetes kubernetes-deployment kubernetes-setup

anna-geller / prefect-aws-lambda
data-engineering-pipeline,Deploy a Prefect flow to serverless AWS Lambda function
User: anna-geller
aws aws-lambda data-engineering data-engineering-infrastructure data-engineering-pipeline dataflow dataflow-ops event-driven event-driven-architecture lambda

anna-geller / prefect-deployment-patterns
data-engineering-pipeline,Code examples showing flow deployment to various types of infrastructure
User: anna-geller
automation aws data data-engineering data-engineering-pipeline data-engineering-team data-science dataflow orchestration pipeline

anna-geller / prefect-getting-started
data-engineering-pipeline,Get started with Prefect by scheduling your Prefect flows with GitHub Actions
User: anna-geller
Home Page: http://prefect.io/
analytics-engineering automation cicd data data-engineering data-engineering-infrastructure data-engineering-pipeline data-pipeline data-science dataflow

antimoz-om / antimoz
data-engineering-pipeline,A data engineering pipeline for digital marketers.
Organization: antimoz-om
data-analytics hadoop digital-marketing data-engineering-pipeline kafka

bayoadejare / lightning-containers
data-engineering-pipeline,Docker powered starter for geospatial analysis of lightning atmospheric data.
User: bayoadejare
Home Page: https://lightning-containers.streamlit.app/
clustering-analysis csv-files data-engineer data-engineering-pipeline data-warehouse databases docker jupyter machine-learning-algorithms noaa-weather

benedekrozemberczki / av_ultimate_student_hunt
data-engineering-pipeline,Solution for the Ultimate Student Hunt Challenge (1st place).
User: benedekrozemberczki
winning-entry machine-learning r analytics-vidhya-competition kaggle driven-data competition xgboost gradient-boosting extreme-gradient-boosting

brunocampos01 / predicting-retail-churn-with-azure-ml-studio
data-engineering-pipeline,Challenge to job: Data Scientist
User: brunocampos01
azure azure-pipelines azure-machine-learning-studio machine-learning-studio softplan challenge python3 pandas machine-learning deploy-machine-learning

charliesergeant / airflow-minio-postgres-fastapi
data-engineering-pipeline,Sample data store project to be hosted on a remote server or cluster. CICD using GitHub actions for SSH Deploy to remote server for docker compose.
User: charliesergeant
airflow data-engineering-pipeline datalake docker-compose fastapi jupyter-notebook minio postgresql python selenium-python

chrisammon3000 / aws-permits-pipeline
data-engineering-pipeline,ETL pipeline for construction permits data in Los Angeles built on AWS S3, Lambda and RDS PostgreSQL.
User: chrisammon3000
aws data-engineering-pipeline postgresql python postgis

datarootsio / notion-dbs-data-quality
data-engineering-pipeline,Using Great Expectations and Notion's API, this repo aims to provide data quality for our databases in Notion.
Organization: datarootsio
data-quality notion notion-api great-expectations data-engineering-pipeline notion-database

delelinus / hfr-data-warehousing
data-engineering-pipeline,End-to-end data engineering processes for the NIGERIA Health Facility Registry (HFR). The project leveraged Selenium, Pandas, PySpark, PostgreSQL and Airflow
User: delelinus
airflow airflow-dags apache-spark data-engineer data-engineering-pipeline data-ingestion data-mining data-science data-warehouse-architecture data-warehousing

dvu4 / udacity-data-engineering
data-engineering-pipeline,Data Engineering Projects including Data Modeling, Data Warehouse, Data Lake Development
User: dvu4
postgresql etl data-modeling etl-pipeline data-engineering data-engineering-pipeline cassandra-database redshift star-schema amazon-sdk

dylanzenner / business_closures_de_pipeline
data-engineering-pipeline,Data Engineering pipeline hosted entirely in the AWS ecosystem utilizing DocumentDB as the database
User: dylanzenner
documentdb data-engineering aws-ecosystem data-engineering-pipeline aws aws-secretsmanager aws-ec2-bastion slack quicksight aws-ssm-document

dylanzenner / greenhouse_gas_emissions_de_pipeline
data-engineering-pipeline,Data Engineering pipeline hosted entirely in the AWS ecosystem utilizing DynamoDB as the database
User: dylanzenner
data-engineering data-engineering-pipeline aws dynamodb aws-ecosystem cloudformation aws-lambda aws-secretsmanager aws-sns aws-sqs

gg-delta / northwind-elt-pipeline
data-engineering-pipeline,Showcasing a high-performance data engineering workflow (ELT) using the classical "northwind" database
User: gg-delta
airbyte aws-rds dbt docker preset snowflake sql aws-s3 data-engineering-pipeline

immu0001 / udacity-data-engineer-nanodegree
data-engineering-pipeline,Classwork projects and home works done through Udacity data engineering nano degree
User: immu0001
spark data-analysis big-data etl data-pipelines classwork airflow-dags data-science data-lake-analytics s3-bucket

inosrahul / f1-data-pipeline
data-engineering-pipeline,F1 Data Pipeline
User: inosrahul
Home Page: https://lookerstudio.google.com/reporting/9fd225dd-a9b8-45d9-87dc-7d7dbae0c841
bigquery data-engineering-pipeline dbt gcs looker prefect python terraform

kaoutaar / velib_v1
data-engineering-pipeline,end to end data engineering project
User: kaoutaar
data-engineering data-engineering-pipeline docker kafka pipeline python spark streamlit webapp spark-streaming

ketgo / marshmallow-pyspark
data-engineering-pipeline,Marshmallow serializer integration with pyspark
User: ketgo
marshmallow pyspark data-pipelines schema spark data-engineering data-engineering-pipeline data-cleaning data-schemas

kishlayjeet / stock-market-real-time-data-pipeline-with-apache-kafka-and-cassandra
data-engineering-pipeline,A end-to-end real-time stock market data pipeline with Python, AWS EC2, Apache Kafka, and Cassandra Data is processed on AWS EC2 with Apache Kafka and stored in a local Cassandra database.
User: kishlayjeet
apache-kafka cassandra kafka pipeline python stock-market stock-market-data-pipeline aws aws-ec2 data-engineering

kishlayjeet / twitter-data-pipeline-using-airflow-and-aws-s3
data-engineering-pipeline,An end-to-end Twitter Data Pipeline that extracts data from Twitter and loads it into AWS S3.
User: kishlayjeet
airflow airflow-dags apache-airflow boto3 data-engineering data-engineering-pipeline data-pipeline etl etl-job etl-pipeline

kkrusere / nhanes-pytool-api
data-engineering-pipeline,The NHANES Data 'API' is a Python tool that simplifies access to the National Health and Nutrition Examination Survey (NHANES) dataset. This project provides an easy-to-use API to retrieve NHANES data, helping researchers, data scientists, health professionals, and other stakeholders access these valuable datasets.
User: kkrusere
Home Page: https://pypi.org/project/nhanes-pytool-api/
data-engineering-pipeline data-mining data-processing health-data health-data-analysis health-data-science nhanes

koksang / social-media-analysis
data-engineering-pipeline,Social Media Analysis, scalable solution, flexible deployment that analyses social media contents
User: koksang
bigquery python ray kafka apache-airflow social-media twitter python3 data-engineering data-engineering-pipeline

longnguyen010203 / youtube-etl-pipeline
data-engineering-pipeline,💜🌈📊 A Data Engineering Project that implements an ETL data pipeline using Dagster, Apache Spark, Streamlit, MinIO, Metabase, Dbt, Polars, Docker. Data from kaggle and youtube-api 🌺
User: longnguyen010203
dagster etl-pipeline minio spark dbt docker docker-compose dockerfile mysql postgresql

markditsworth / tweetanalyzer
data-engineering-pipeline,An environment for analyzing Twitter
User: markditsworth
twitter elasticsearch logstash kibana kafka docker python nlp-machine-learning data-engineering-pipeline

mikeroyal / apache-spark-guide
data-engineering-pipeline,Apache Spark Guide
User: mikeroyal
spark spark-streaming data-engineering pyspark apache-spark machine-learning big-data data-science data-engineering-pipeline awesome

minhky2185 / healthcare_data_pipeline
data-engineering-pipeline,An end-to-end data pipeline for building Data Lake and supporting report using Apache Spark.
User: minhky2185
analytics big-data data data-engineering data-engineering-pipeline data-lake emr-cluster mysql postgresql powerbi

nitindatta8 / realtime-data-streaming
data-engineering-pipeline,End-to-end data engineering pipeline with various technologies to ingest real time data.
User: nitindatta8
apache-airflow apache-kafka apache-spark apache-zookeeper big-data cassandra data-engineering data-engineering-pipeline data-processing docker

pyprogrammerblog / tiny-blocks
data-engineering-pipeline,Tiny Blocks to build large and complex data pipelines!
User: pyprogrammerblog
Home Page: https://tiny-blocks.readthedocs.io/en/latest/
data etl-automation etl-framework pipelines python3 data-engineering data-engineering-pipeline etl generator pandas

salimt / spotify-api-pipeline
data-engineering-pipeline,Spotify API, Airflow, Docker, AWS S3, Snowflake, dbt, localstack, Looker Studio
User: salimt
api aws-s3 data-engineering data-engineering-pipeline dbt docker elt localstack snowflake looker

san089 / goodreads_etl_pipeline
data-engineering-pipeline,An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.
User: san089
etl-pipeline etl-framework spark apache-spark apache-airflow airflow redshift emr-cluster livy s3

san089 / udacity-data-engineering-projects
data-engineering-pipeline,Few projects related to Data Engineering including Data Modeling, Infrastructure setup on cloud, Data Warehousing and Data Lake development.
User: san089
data data-engineering data-engineering-pipeline etl-pipeline cassandra-database postgresql-database data-modeling data-warehouse data-lake airflow

sanjeevai / disaster-response-pipeline
data-engineering-pipeline,ETL pipeline combined with supervised learning and grid search to classify text messages sent during a disaster event
User: sanjeevai
grid-search-hyperparameters etl-pipeline supervised-learning disaster-event sqlite-database data-engineering-pipeline

shihwen / tpe-mrt-traffic-etl
data-engineering-pipeline,A data pipeline from source to data warehouse using Taipei Metro Hourly Traffic data
User: shihwen
airflow python data-engineering data-engineering-pipeline data-warehouse etl-job etl-pipeline redshift s3 taipei-metro

shiv-rna / youtube-data-engineering-pipeline
data-engineering-pipeline,This project repo 📺 offers a robust solution meticulously crafted to efficiently manage, process, and analyze YouTube video data leveraging the power of AWS services. Whether you're diving into structured statistics or exploring the nuances of trending key metrics, this pipeline is engineered to handle it all with finesse.
User: shiv-rna
Home Page: https://gratis-dragonfly-657.notion.site/YouTube-Data-Analysis-Project-10dbf8886e47469f93a248526c953e90?pvs=4
aws aws-athena aws-cli aws-glue aws-glue-data-catalog aws-iam aws-lambda aws-quicksight aws-s3 data-engineering

siddharth271101 / covid-19-and-aviation-industry
data-engineering-pipeline,The goal of this project is to analyse the impact of Covid-19 on the Aviation industry through data engineering processes using technologies such as Apache Airflow, Apache Spark, Tableau and couple of AWS services
User: siddharth271101
analytics athena aws-s3 covid-19 dashboard docker emr opensky tableau airflow

tmaferreira / dataengineeringzoomcampproject
data-engineering-pipeline,Data Engineering ZoomCamp Course Project
User: tmaferreira
bigquery cloud-storage data-engineering data-engineering-pipeline data-lake data-visualization dbt prefect python3 terraform

umairthakur / uber-data-analysis-etl-pipeline-data-analysis_project
data-engineering-pipeline,Uber Data Analysis Project, an End-to-End Data Engineering Project from creating data pipelines to finally creating the dashboard.
User: umairthakur
Home Page: https://lookerstudio.google.com/s/nQI06ax2wMY
bigquery data-engineering-pipeline data-science data-visualization google-cloud-platform looker-studio mage-data-pipeline-tool python3 sql

verazab / nyc-stats
data-engineering-pipeline,Analysis of 311 Service Requests for the City of NYC (from 2010 to 2023) Tech: Prefect cloud, dbt core, BigQuery, Compute Engine, CloudRun, Artifact Registry, Terraform, Docker
User: verazab
Home Page: https://lookerstudio.google.com/reporting/65ee32b0-4626-4a39-8065-5d8c27380a1a
gcp prefect terraform dbt looker-studio cloudrun-jobs compute-engine data-engineering-pipeline ci-cd docker

vmware / versatile-data-kit
data-engineering-pipeline,One framework to develop, deploy and operate data workflows with Python and SQL.
Organization: vmware
data-science data-engineering sql trino data-lineage etl elt data-pipelines data-engineer data-warehouse

waqarg2001 / formula1-insights-de
data-engineering-pipeline,Formula 1 race data engineering project which utilises azure services and databricks to ingest and analyse the data.
User: waqarg2001
azure azure-active-directory azure-data-factory azure-data-lake-gen2 azure-keyvault azure-service-principal data-engineering data-engineering-pipeline data-lake data-modelling

yan-luo-au / data_engineer_project_etl_bi
data-engineering-pipeline,This is an ETL project - extracting data from an ecommerce transactional database on RDS, transforming the data using AWS glue job, and loading it to a Redshift data warehouse, and connected it to Tableau for BI
User: yan-luo-au
data-engineering-pipeline redshift rds-mssql glue datapipeline secret-manager

zarexalvindaria / data-engineering
data-engineering-pipeline,This repo contains the Data Engineering exercises I took in Datacamp.
User: zarexalvindaria
data-engineering data-engineering-pipeline python spark singer airflow airflow-dags