analytical_dp_with_sql

Code for my "Efficient Data Processing in SQL" book.

beginner_de_project

Beginner data engineering project - batch edition

beginner_de_project_stream

Simple stream processing pipeline

bitcoinmonitor

Near real time ETL to populate a dashboard.

change_data_capture

Repo for CDC with debezium blog post

data

open data for blog content at https://www.startdataengineering.com/

data_engineering_best_practices

Sample project to demonstrate data engineering best practices

data_engineering_best_practices_log

Code to demonstrate data engineering metadata & logging best practices

data_engineering_project_template

A template repository to create a data project with IAC, CI/CD, Data migrations, & testing

data_test_ci

Repository showing how to automate data testing as part of CI

dbt_development

Repo to explain development, CI/CD cycle in dbt

docker-trino-cluster

Multiple node presto cluster on docker container

docker_for_data_engineers

Code for blog at: https://www.startdataengineering.com/post/docker-for-de/

e2e_datapipeline_test

Example repo to create end to end tests for data pipeline.

efficient_data_processing_spark

Code for "Efficient Data Processing in Spark" Course

files

public file hosting

hive-metastore

iceberg-features

idempotent-data-pipeline

Making data pipelines idempotent

josephmachado

Profile readme

local_dev

Local development environment for python data projects, with Docker

online_store

End to end data engineering project

sde_superset_demo

Apache Superset Demp

simple_dbt_project

Code for dbt tutorial

simple_polars_etl

socialetl

Project for "Data pipeline design patterns" blog.

soho

Minimalist Hugo theme based on Hyde

spark_submit_airflow

Simple repo to demonstrate how to submit a spark job to EMR from Airflow

spark_submit_airflow-

Simple repo to demonstrate how to submit a spark job

trigger_spark_with_lambda

Simple example showing how to trigger a spark job with AWS Lambda