avin-xt Goto Github PK

adf-docs

adfdataflowdocs

Azure Data Factory Data Flow Documentation

config-driven-data-pipeline

databricks-lakehouse

This repo contains live examples to build Databricks' Lakehouse and recommended best practices from the field.

databricks-nutter-repos-demo

Demo of using the Nutter for testing of Databricks notebooks in the CI/CD pipeline

dbsql_sme

DBSQL SME Repo contains demos, tutorials, blog code, advanced production helper functions and more!

delta-live-tables-notebooks

dlt-pii-firewall

Demo of using Delta Live Tables to identify potential PII data

ease-with-apache-spark

Series follows learning from Apache Spark (PySpark) with quick tips and workaround for daily problems in hand

lakehouse-tacklebox

This repo is a collection of tools to deploy, manage and operate a Databricks based Lakehouse.

langchain-tutorials

Overview and tutorial of the LangChain Library

mack_type2_scd

Delta Lake helper methods in PySpark

modern-data-eng-dbt-databricks-azure

In this project, we setup and end to end data engineering using Apache Spark, Azure Databricks, Data Build Tool (DBT) using Azure as our cloud provider.

pyspark-cicd-template

PySpark data-pipeline testing and CICD

pyspark-delta-scd2

This project utilizes `faker-pyspark` to generate random schema and dataframes to mimic data table snapshots. Using these snapshots to process and apply SCD2 pattern into delta table as the destination.

pyspark_schema_generator_code

This Python Script generates a PySpark StructType schema from a PostgreSQL table schema.

python-programming-exercises

100+ Python challenging programming exercises

pytorch-transformer

Attention is all you need implementation

spark-structured-streaming

sparkinternals

Notes talking about the design and implementation of Apache Spark