Topic: etl-pipeline Goto Github

Some thing interesting about etl-pipeline

👇 Here are 1299 public repositories matching this topic...

airscholar / e2e-data-engineering

etl-pipeline,An end-to-end data engineering pipeline that orchestrates data ingestion, processing, and storage using Apache Airflow, Python, Apache Kafka, Apache Zookeeper, Apache Spark, and Cassandra. All components are containerized with Docker for easy deployment and scalability.

User: airscholar

Home Page: https://www.youtube.com/watch?v=GqAcTrqKcrY

apache-airflow apache-kafka apache-spark apache-zookeeper big-data cassandra containerization data-engineering data-pipeline data-processing

alexioannides / pyspark-example-project

etl-pipeline,Implementing best practices for PySpark ETL jobs and applications.

User: alexioannides

pyspark etl-job python data-engineering spark data-science etl etl-pipeline

anmol12499 / reddit-analytics-integration-platform

etl-pipeline,Project was based on an interest in Data Engineering, ETL pipeline. It also provided a good opportunity to develop skills and experience in a range of tools. As such, project is more complex than required, utilising dbt, airflow, docker and cloud based storage.

User: anmol12499

airflow aws-redshift aws-s3 dbt docker etl-pipeline google-studio python terraform

apache / incubator-streampark

etl-pipeline,Make stream processing easier! Easy-to-use streaming application development framework and operation platform.

Organization: apache

Home Page: https://streampark.apache.org/

streaming streampark apache development-framework easy-to-use etl-pipeline operation-platform

conductor-sdk / conductor-python

etl-pipeline,Conductor OSS SDK for Python programming language

Organization: conductor-sdk

python conductor workflow data-pipelines durable-computing durable-execution etl-pipeline

cyber-drop / ethereum_analytical_db

etl-pipeline, Ethereum Analytical Database - Ethereum data access solution that can be used for analytics and application development. The solution works on a fast DB - Clickhouse.

Organization: cyber-drop

Home Page: https://cyberdrop.ai

ethereum blockchain etl-pipeline etl clickhouse eth erc20 erc721 erc223 dex

dagworks-inc / hamilton

etl-pipeline,Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage and metadata. Runs and scales everywhere python does.

Organization: dagworks-inc

Home Page: https://hamilton.dagworks.io/en/latest/

data-science python dag data-engineering dataframe etl etl-framework etl-pipeline feature-engineering featurization

danielsaban / data-scraping-sofascore

etl-pipeline,Data Engineering/Scraping Project. Creating a detailed Sports Relational Database for the Top European Soccer Leagues.

User: danielsaban

dataengineering datamining webscraping relational-databases parser etl-pipeline rest-api

dappboard / dappboard-etl

etl-pipeline,ETL pipeline for the Ethereum blockchain

Organization: dappboard

Home Page: http://dappboard.com

ethereum-blockchain blockchain etl etl-pipeline javascript

data-engineering-community / data-engineering-project-template

etl-pipeline,This is a template you can use for your next data engineering portfolio project.

Organization: data-engineering-community

Home Page: https://dataengineering.wiki

data-engineering sql python data data-warehouse etl etl-pipeline

flow-php / flow

etl-pipeline,Flow PHP - data processing framework

Organization: flow-php

Home Page: https://flow-php.com

etl etl-pipeline etl-framework

hifxit / dataligo

etl-pipeline,A library to accelerate ML and ETL pipeline by connecting all data sources

User: hifxit

database datalake datawarehouse etl-pipeline ml-pipeline nosql python

imsanjoykb / data-science-regular-bootcamp

etl-pipeline,Regular practice on Data Science, Machien Learning, Deep Learning, Solving ML Project problem, Analytical Issue. Regular boost up my knowledge. The goal is to help learner with learning resource on Data Science filed.

User: imsanjoykb

Home Page: https://imsanjoykb.github.io/

data-science machine-learning artificial-intelligence deep-learning neural-network data-science-projects data-visualization data-analysis data-science-notebook database-connection

indexical-metrics-measure-advisory / watchmen-matryoshka-doll

etl-pipeline,Watchmen Platform is a low code data platform for data pipeline, meta data management , analysis, and quality management

Organization: indexical-metrics-measure-advisory

Home Page: https://imma-watchmen.com/

charts data-pipeline data-quality-monitoring data-visualization etl-pipeline pipeline visualization

interestinglab / seatunnel-example

etl-pipeline,seatunnel plugin developing examples.

Organization: interestinglab

spark etl-framework etl-pipeline spark-streaming flink sql-engine waterdrop

jitsucom / bulker

etl-pipeline,Service for bulk-loading data to databases with automatic schema management (Redshift, Snowflake, BigQuery, ClickHouse, Postgres, MySQL)

Organization: jitsucom

Home Page: https://github.com/jitsucom/bulker

data-engineering datawarehouse etl etl-pipeline ingestion pipeline

josephmachado / bitcoinmonitor

etl-pipeline,Near real time ETL to populate a dashboard.

User: josephmachado

Home Page: https://www.startdataengineering.com/post/data-engineering-project-to-impress-hiring-managers/

cron docker docker-compose etl-pipeline metabase nearrealtime postgres pytest python3

josephmachado / socialetl

etl-pipeline,Project for "Data pipeline design patterns" blog.

User: josephmachado

Home Page: https://www.startdataengineering.com/post/code-patterns/

dataengineering design-patterns etl-pipeline makefile python reddit social-media-data sqllite3

jvalue / jayvee

etl-pipeline,Jayvee is a domain-specific language and runtime for automated processing of data pipelines

Organization: jvalue

Home Page: https://jvalue.github.io/jayvee/

data-engineering data-pipeline data-science domain-specific-language etl-pipeline typescript

martandsingh / apachespark

etl-pipeline,This repository will help you to learn about databricks concept with the help of examples. It will include all the important topics which we need in our real life experience as a data engineer. We will be using pyspark & sparksql for the development. At the end of the course we also cover few case studies.

User: martandsingh

apachespark data-analysis data-engineering database databricks datalake deltalake etl-pipeline hadoop hive

massstreetanalytics / etl-framework

etl-pipeline,A framework for moving data into a data warehouse.

Organization: massstreetanalytics

Home Page: http://www.datawarehouseetlframework.com/

data-warehouse etl etl-components etl-framework etl-pipeline python sql sqlserver

maxim2266 / csvplus

etl-pipeline,csvplus extends the standard Go encoding/csv package with fluent interface, lazy stream operations, indices and joins.

User: maxim2266

go csv csv-format fluent-interface stream-processing go-csv etl-pipeline etl-framework etl

mmodarre / azuredatafactoryhol

etl-pipeline,Azure Data Factory Hands On Lab - Step by Step - A Comprehensive Azure Data Factory and Mapping Data Flow step by step tutorial

User: mmodarre

azure-data-factory azure adf-pipeline filter-activity lookup-activity foreach-activity metadata-activity mapping-dataflows azure-key-vault etl-pipeline

mobiletelesystems / onetl

etl-pipeline,One ETL tool to rule them all

Organization: mobiletelesystems

Home Page: https://onetl.readthedocs.io/

spark etl etl-components etl-pipeline hwm plugin-system pydantic

mohamedhmini / tweetsolaping

etl-pipeline,implementing an end-to-end tweets ETL/Analysis pipeline.

User: mohamedhmini

analysis api-client cube-analysis datawarehouse datawarehousing etl-pipeline google-api-client multi-dimensional-analysis multithreading powerbi-report ssas-multidimensional ssis tweets tweets-classification tweets-scraper twitter-api web-crawling

mycelial / mycelial

etl-pipeline,Move your data with ease.

Organization: mycelial

Home Page: https://mycelial.com

data-pipelines edge-computing etl etl-pipeline rust

orchest / orchest

etl-pipeline,Build data pipelines, the easy way 🛠️

Organization: orchest

Home Page: https://orchest.readthedocs.io/en/stable/

data-science machine-learning pipelines ide jupyter cloud self-hosted jupyterlab notebooks docker

patterns-app / patterns-devkit

etl-pipeline,Data pipelines from re-usable components

Organization: patterns-app

data-science data-analysis pipelines etl etl-pipeline etl-framework functional-reactive-programming data-engineering sql immutability

razevedo1994 / razv-data-engineering

etl-pipeline,Portfolio of projects and studies conducted in data engineering.

User: razevedo1994

etl-pipeline python3 etl data-warehouse data-lake data-modeling sql spark aws aws-lambda

redis-field-engineering / redis-connect-dist

etl-pipeline,Real-Time Event Streaming & Change Data Capture

Organization: redis-field-engineering

Home Page: https://redis-field-engineering.github.io/redis-connect-dist/

rediscdc cdc replication etl etl-pipeline etl-framework etl-automation csv redisconnect connect

restarone / violet_rails

etl-pipeline,an app engine for your business. Seamlessly implement business logic with a powerful API. Out of the box CMS, blog, forum and email functionality. Developer friendly & easily extendable for your next SaaS/XaaS project. Built with Rails 6, Devise, Sidekiq & PostgreSQL

Organization: restarone

Home Page: https://violet.restarone.solutions

blog cms ember emberjs etl-automation etl-framework etl-pipeline forum multi-tenancy multitenancy rails ruby ruby-on-rails rubyonrails saas saas-boilerplate template violet-rails wordpress-replacement xaas

runprism / prism

etl-pipeline,Prism is the easiest way to develop, orchestrate, and execute data pipelines in Python.

Organization: runprism

Home Page: https://runprism.com

data data-engineering data-science etl etl-pipeline machine-learning orchestration pipeline python data-orc bigquery data-analysis data-integration dbt postgres redshift snowflake trino

san089 / goodreads_etl_pipeline

etl-pipeline,An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.

User: san089

etl-pipeline etl-framework spark apache-spark apache-airflow airflow redshift emr-cluster livy s3

san089 / udacity-data-engineering-projects

etl-pipeline,Few projects related to Data Engineering including Data Modeling, Infrastructure setup on cloud, Data Warehousing and Data Lake development.

User: san089

data data-engineering data-engineering-pipeline etl-pipeline cassandra-database postgresql-database data-modeling data-warehouse data-lake airflow airflow-operators cluster cassandra infrastructure postgres aws aws-ec2 aws-sdk aws-s3 cloudformation

setl-framework / setl

etl-pipeline,A simple Spark-powered ETL framework that just works 🍺

Organization: setl-framework

spark etl framework scala setl pipeline data-transformation data-science data-engineering data-analysis modularization dataset big-data etl-pipeline machine-learning

souvik-databricks / dlt-with-debug

etl-pipeline,A lightweight helper utility which allows developers to do interactive pipeline development by having a unified source code for both DLT run and Non-DLT interactive notebook run.

User: souvik-databricks

Home Page: https://pypi.org/project/dlt-with-debug/

big-data big-data-processing databricks delta-live-tables dlt etl etl-pipeline python3 spark

stellar / stellar-etl

etl-pipeline,Stellar ETL will enable real-time analytics on the Stellar network

Organization: stellar

stellar stellar-network stellar-lumens etl-pipeline data-analysis etl-framework ethereum bitcoin blockchain

stellar / stellar-etl-airflow

etl-pipeline,Airflow DAGs for the Stellar ETL project

Organization: stellar

blockchain stellar-lumens etl-pipeline python stellar airflow stellar-network etl-framework data-analysis

stitchfix / hamilton

etl-pipeline,A scalable general purpose micro-framework for defining dataflows. THIS REPOSITORY HAS BEEN MOVED TO www.github.com/dagworks-inc/hamilton

Organization: stitchfix

Home Page: https://www.github.com/dagworks-inc/hamilton

python pandas dag data-science data-engineering numpy software-engineering etl-framework etl-pipeline etl feature-engineering featurization dataframe stitch-fix data-platform hamilton hamiltonian machine-learning

sundios / seo-dashboard

etl-pipeline,SEO dashboard from Search console Data using the Google Search API, Mysql database , NodeJS RESTAPI( ExpressJS) and reactJs Dashboard

User: sundios

Home Page: https://www.kwrds.ai

seo seotools etl-pipeline dashboard react seo-monitor rest-api google-search-console-python google-search-console mysql

techascent / tech.ml.dataset

etl-pipeline,A Clojure high performance data processing system

Organization: techascent

clojure dataframe csv xlsx datascience machine-learning dataset etl-pipeline java

tharwaninitin / etlflow

etl-pipeline,EtlFlow is an ecosystem of functional libraries in Scala based on ZIO for running complex Auditable workflows which can interact with Google Cloud Platform, AWS, Kubernetes, Databases, SFTP servers, On-Prem Systems and more.

User: tharwaninitin

scala spark etl bigquery etl-framework zio etl-pipeline aws gcp redis

toddbirchard / jira-database-etl

etl-pipeline,:mens: 💾 Script to import issues from a JIRA instance into a database.

User: toddbirchard

Home Page: https://hackersandslackers.com/jira-to-sql-etl/

jira-rest-api python3 pandas flask flask-sqlalchemy etl etl-pipeline

usc-isi-i2 / dig-etl-engine

etl-pipeline,Download DIG to run on your laptop or server.

Organization: usc-isi-i2

Home Page: http://usc-isi-i2.github.io/dig/

crawling information-extraction search-engine etl-framework etl-pipeline information-visualization

vim89 / datapipelines-essentials-python

etl-pipeline,Simplified ETL process in Hadoop using Apache Spark. Has complete ETL pipeline for datalake. SparkSession extensions, DataFrame validation, Column extensions, SQL functions, and DataFrame transformations

User: vim89

apache-spark spark spark-sql python python3 pyspark etl etl-pipeline etl-framework etl-components xml xml-parsing datalake big-data hadoop hadoop-mapreduce hadoop-hdfs data-pipeline

visiologyofficial / vixtract

etl-pipeline,

Organization: visiologyofficial

Home Page: https://www.vixtract.ru

etl etl-pipeline etl-framework etl-automation etl-components etl-job

wathon / data_engineering_with_python-track-datacamp

etl-pipeline,Data Engineer with Python lecture notes from #datacamp.

User: wathon

data-engineering python datacamp-track shell sql scala aws-boto pyspark spark-sql mongodb

wittline / uber-expenses-tracking

etl-pipeline,The goal of this project is to track the expenses of Uber Rides and Uber Eats through data Engineering processes using technologies such as Apache Airflow, AWS Redshift and Power BI.

User: wittline

Home Page: https://wittline.github.io/uber-expenses-tracking/

data-engineering python apache-airflow etl-pipeline aws-redshift power-bi data-modeling uber-eats uber-data expenses-dashboard