Light

Dogukan Ulu photo

dogukannulu Goto Github PK

followers: 253.0 following: 4.0 repos: 24.0 gists: 0.0

Name: Dogukan Ulu

Type: User

Bio: Data Engineer

Blog: https://dogukanulu.dev

Hi! I'm Dogukan

Data Engineer

About Me :

I am working as a Data Engineer. I develop many data engineering projects using the below tools and frameworks. I always start working on the projects and jobs with a can-do approach. I always learn the best practices to increase efficiency and apply them while handling the tasks. I am a strong believer of sustainability and we, as data engineers, should complete the tasks as sustainable as possible. I also believe that theoretical part of the data is quite important. That's why I always learn and keep myself up-to-date about the principles of distributed computing, big data applications, how to handle the streaming data, scheduling/orchestrating the pipelines, working principles of cloud environments and many other topics. While doing those, I also develop myself in coding especially at Python and SQL. I create many complicated Python scripts as well as many SQL queries. You can find lots of data engineering projects here, welcome to my GitHub :)

Connect with me:

Tech Stack:

Dogukan Ulu's Projects

airflow_kafka_cassandra_mongodb

Produce Kafka messages, consume them and upload into Cassandra, MongoDB.

amazon_msk_kafka_streaming

Create Kafka topic, stream the data to producer and consume on the console using Amazon MSK

aws_end_to_end_streaming_pipeline

An AWS Data Engineering End-to-End Project (Glue, Lambda, Kinesis, Redshift, QuickSight, Athena, EC2, S3)

crypto_api_kafka_airflow_streaming

Get Crypto data from API, stream it to Kafka with Airflow. Write data to MySQL and visualize with Metabase

csv_extract_airflow_docker

Writes the CSV file to Postgres, read table and modify it. Write more tables to Postgres with Airflow.

csv_to_kinesis_streams

This repo will write a CSV file to the Amazon Kinesis Data Streams

data-generator

This repo is for generating data from existing dataset to a file or producing dataset rows as message to kafka in a streaming manner.

datasets

This repo contains datasets used in trainings.

docker-airflow

Docker Apache Airflow

docker-hadoop

dogukannulu

My personal repo

glue_etl_job_data_catalog_s3

Glue ETL job or EMR Spark that gets from data catalog, modifies and uploads to S3 and Data Catalog

ibm-data-science-capstone-project

This repository is created for IBM Data Science Professional Certificate Capstone Project

kafka_spark_structured_streaming

Get data from API, run a scheduled script with Airflow, send data to Kafka and consume with Spark, then write to Cassandra

kaggle_projects

In this repository, I created ML algorithms for various Kaggle Competitions

parquet_gcs_bucket_to_bigquery_table

Parquet files will be obtained regularly from a public GCS bucket. They will be written to BQ table

prefect-example-flows

Create sample Prefect flows, deploy them as Docker containers and store within GitHub

read_from_s3_upload_to_rds

Upload the remote data into Amazon S3, read the data and upload to Amazon RDS MySQL

s3_trigger_lambda_to_rds

Send a dataframe to S3 automatically, trigger Lambda and modify dataframe, upload to RDS

send_data_to_aws_services

This repo automates the processes when we want to send remote data to AWS services such as Kinesis, S3, etc.

snowpipe-aws-stream-processing

Get the streaming data from the S3 bucket with SQS queue. Load into Snowflake with Snowpipe and modify the data with Snowflake task

streaming_data_processing

Create a streaming data, transfer it to Kafka, modify it with PySpark, take it to ElasticSearch and MinIO

super_lig_streamlit

twitter_etl_s3

Get data via Twitter API, orchestrate with Airflow and store in S3 bucket

1

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.