Home | Internet | AWS | Serverless | Optional - Big Data History | Spark Overview | Spark DataFrame API, SQL and Internals |

Help/Resources.

Welcome to the course.

• For "official" communication we are using Moodle.
• For informal and in-class communication, we are using Slack.
• On this page you will find additional courseware, R and Python scripts, which accompany the course.
• In case you need help, you can find the instructor's and the TA's contact details at the bottom of this page.
• Before you'd reshare/reuse any of these materials keep in mind that Datapao owns the copyright of the Spark notebooks (distributed separately).

Data Engineering 3: Big Data and Cloud Computing

Prerequisites:

R programming

SQL Knowledge

Linux command-line knowledge

The contents of this site have moved to https://bigdata-cloud.github.io/. Please update your bookmarks.

Course description This is a technology-focused course on cloud and distributed data analytics systems. Current Data Analytics Architectures often work with an amount of data that cannot be fit on a single computer. Even companies that work with reasonably small datasets are expecting rapid growth, so they prefer to use data analytics solutions that are easy to distribute and scale when needed. In this course you will get an overview and hands-on experience with modern distributed data-analytics (a.k.a. Big Data) systems and Serverless Solutions in the cloud. You will see how cloud computing can help you quickly iterate and scale your data analytics infrastructure and how it can help you reduce operational costs.

Learning outcomes At the end of this course you will have an overview of Cloud and Big Data technologies applied in modern businesses. You will have a general understanding on how these technologies work and you will be able to reason about when to use or not to use them. You will be hands-on with Amazon Web Services and Apache Spark. Once you completed the assignments for this course, you will be hands-on with the following technologies: - Public Key Encryption - Basic Cloud Computing concepts: Storage, Virtual Machines - Serverless Services for image and text recognition - Spark Architecture and internal operations - Spark SQL and DataFrames in Spark 2

Reading List - Matei Zaharia et al.: Spark, the Definitive Guide (sections)

Course schedule and materials for each session

1. Basics of Internet. TCP/IP, Servers, Ports, Firewalls
2. Basics of Cloud Computing using Amazon Web Services (AWS): Storage and Virtual Machines
3. Serverless solutions in the cloud
4. Using AWS programatically
5. Apache Spark Overview
6. The Spark DataFrame and SQL API
7. Spark internals 8. Advanced Optimizations in Spark

Many of the course materials were created by Miklos Petridisz