The current repository was created to develop a data pipeline using the Scala language. It is important to highlight that this is just a personal project to understand how powerful this language is. I do not intend to build an ideal project, as there will likely be many points for improvement.

That said, the current project proposes to take raw data from CSV and JSON files, and using Scala with Spark, read that data, perform some transformations, and then write it to another directory. As I believe in continuous improvement, I want to perform a Scala data pipeline on the AWS Cloud. So, soon I will create the cloud infrastructure using Terraform and then develop the respective scripts.

I would be very grateful if you could send me some feedback! I am eager to learn more and more! So, the main purpose of the repository is to learn about different technologies used in the ETL process.