Repo Status

Tooling

See the documentation for information on using this repo. The Avengercon VIII conference homepage has details about the event.

This is a hands-on Python programming workshop. At least one year of recent Python experience, some experience with Docker, and a computer you administer is strongly recommended.

Python can be challenging to use in production when "real-world" workloads involving megabits per second (Mbps) of streaming data or terabytes of stored data are involved. The Global Interpreter Lock (GIL) means that a Python interpreter is effectively single-threaded and can't take advantage of modern processors' capacity for parallel computation. Laterally scaling workloads across many Python interpreters is one of the most viable workarounds to the shortcomings of the GIL. This workshop will introduce you to leading frameworks for doing this: Celery, Dask, and Apache Beam. This workshop will walk through establishing an Extract, Transform, Load (ETL) pipeline in each framework which reads and writes from Redis and a MinIO locally hosted S3 bucket.