hadiezatpanah Goto Github PK

hadiezatpanah

pyspark-aws-postgres-etl

An End to End solution to read TSV files from AWS S3 and process and import them into postgres relational database.

spark-scala-data-pipeline

An End to End solution to read XML data from FTP server and process and import them into postgres relational database

spark_java_mostvaluablecustomers

This Spark Java project serves as a demonstration of Gradle Spark configuration, specifically focusing on utilizing the MemoryStream class as the streaming source.

spark_java_stateful_processing

This project presents a distributable solution based on Spark Java, aiming to connect start and end session events together in a stateful manner. The project utilizes `flatMapGroupWithState`functionality which is a powerful feature for stateful stream processing in Spark. It enables you to maintain and update the state across batches.

spark_structured_streaming_java

In this solution, the issue of creating a table with case-sensitive columns (in the scenario where the table doesn't exist or when writing the table in overwrite mode) in Oracle has been addressed by developing a custom Oracle dialect and registering it.

trending_topic_spark_streaming_scala

This is an End to End solution to read data from streaming source (kafka), extract different topic from data in each time window, calculating Hot Topics using a modified Z-Score Algorithm and storing Final Trend Topics in Postgres SQL Database